ROI Understanding

8/14/2019 ROI Understanding

1/52

Return on Investment

In Information Technology:A Guide for Managers

Anthony M. Cresswel

Center for Technology in GovernmentUniversity at Albany, SUNY

187 Wolf Road, Suite 301Albany, NY 12205

Phone: (518) 442-3892Fax: (518) 442-3886

E-mail: [email protected]

August 2004

2004 Center for Technology in GovernmentThe Center grants permission to reprint this document provided this cover page is included.


2/52

CENTER FOR TECHNOLOGY IN GOVERNMENTRETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS


3/52


Acknowledgments

This report was developed on a foundation of previous work of the Center for Technologyin Government and valuable assistance from several staff members. Sharon Dawes,Theresa Pardo, Donna Canestraro, and Meghan Cook made important contributions to earlyversions. Natalie Helbig, Dubravka Juraga provided excellent suggestions and edits on thefinal version. Mark LaVigne helped with editing as well, and also worked with Paula Hauseron the design of the printed document. The research on costs and returns of investmentin Web site conversion, reported here as a case study, is based on the technical work ofDerek Werthmuller and James Costello, and the assistance of Carrie Schneider andYi-Jung Wu.


4/52



5/52


Executive Summary 1

Chapter One 5 ROI and the Need for Smart IT Investment Decisions

6 An approach to understanding and using ROI analysis

7 Understanding strategic objectives in ROI design

8 Defining and measuring the costs and returns from IT investments

9 Understanding the enterprise: technology in the business context

12 Understanding ROI decisions in their political and policy context

12 Political risk factors

13 Organizational risk factors

13 Business process risks13 Technology risk facto rs

Chapter Two 14 Methods of ROI Analysis for IT

14 Issues of t ime and scale

14 Importance of business process analysis

15 Benefits of process modeling

16 Methods for understanding the business process

16 Descriptive models

18 Use of analytical or formal models

18 Types of for mal modeling approaches

22 Measuring costs and returns22 Financial metrics and government accounting

23 Measuring costs and cost-effectiveness

24 Efficiency measures

24 Impact measures

25 How time perspectives change the measurements

25 The problem of choosing the project or investment life cycle

25 Risk analysis in the public sector

Appendix A: Case 1 28 Reducing the Cost of Web Site Development and Maintenance

29 Advantages of a d ynamic Web s ite

29 Data sources

30 Cost estimation

31 Benefits

33 Summary of costs and returns

Table of Contents


6/52


Appendix B: Case 2 35 ROI for Data Integration in Health and Human Services

35 Project goals and the context of State-Level welfare reform

35 Project rationale

35 Investment in improved welfare administration

36 Project methods

36 ROI framework

37 ROI results for the Welfare Refor m Related Technology Fund

38 Potential risks in benefit estimates

Appendix C : Case 3 40 Socia l ROI

40 The investment philanthropy approach

40 The social return on investment method

43 Measuring value and returns

43 Limitations in the SROI approach

Appendix D 45 Additional Resources

45 Enterprise architecture

45 Stakeholder analysis

45 Business process analysis and modeling

45 Formal modeling

45 Agent base d model ing

46 UML modeling

46 Workflow modeling

46 Operations research

Table of Contents


7/52


Executive Summary

1

Today more than ever, governmentdecision makers must make the mostof scarce resources and at the same timerespond to ever-increasing demands forimproved performance and new technology.These competing demands generate closescrutiny of proposals for new informationtechnology (IT) investments. Whats more,high profile IT system failures have raisedconcerns about why these investmentsso often fail to live up to expectations. Asa result, many IT investment planningprocesses now require some analysis ofthe costs and returns expected from that

proposed investment. Unfortunately, publicsector managers often lack models that canguide them through such analyses. ThisGuide is offered to help fill that gap.

The Guide provides that help by presentinga practical approach to understandingwhat ROI analysis can and cannot do.

A meaningful return on investment (ROI)analysis in information technology is a littlelike saying you want to live a healthierlifestyle. Like lifestyle changes, ROIanalysis is not just a single component.Instead, it is a collection of methods, skills,

tools, activities, and ideas. They can becombined and used in many different waysto assess the relative value of an investmentover time. Applying this collection in aparticular situation requires making manychoices among the ideas and methodsavailable and conducting an analysisappropriate to the decision at hand.

Different choices will produce differentresults. Therefore the Guide presents aframework of the key questions that shouldlead to an appropriate ROI analysis. It thenpresents a review of the methods and

resources needed along with examplesof different approaches in detailed casestudies.

How extensive should yourROI analysis be?

Once the decision has been made to conductan ROI analysis, what should it look like?

The choice of how to conduct the analysisshould be based on four critical principlespertaining to:

the strategic objective(s) of the ROIanalysis,

the place (and importance) of the IT

investment in the overall enterprisearchitecture,

the type of analysis that should beconducted (i.e., what data and methodsof analysis are best suited to thoseobjectives), and

how the ROI analysis fits in the overalldecision context for IT investments.

Understanding the strategic

objectives of your ROI analysis

Understanding the strategic objectives ofan ROI analysis will determine how theanalysis is ultimately done and used. Ahandful of questions like these can helpmanagers decide what the objectives ofthe analysis should be.

Is the proposed project critical to thebusiness objectives of your agencies orgovernment?

What are the risk factors associated with theinvestment?

Who will be impactedpositively ornegativelyby the proposed project?

Is an ROI necessary for approval andsupport of the proposed project?

Is the proposed project worth theinvestment of an ROI analysis? And ifso, how detailed should it be?


8/52

CENTER FOR TECHNOLOGY IN GOVERNMENTRETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS2

Answering these questions will help identifythe resources needed to conduct an ROIanalysis, which in some cases can itself bea substantial investment. Extending its levelof detail beyond what is needed for effectivedecision making is a waste of resources.Focusing on strategic objectives keepsattention on the full range of benefits to beexpected from the investment, and how tomeasure them.

Understanding the context

of your IT investment

Any IT investment project is embedded in anorganizations technology infrastructure(enterprise architecture), relevant businessprocesses, organizational environment, andexternal relationships.

Technology infrastructure.There aredirect costs associated with the technologyand services in which you invest, andthere will also be costs in terms of itsimpact on other technology systemsalready in place. The benefits range frommore efficient automation and workflow to

improved collection, storage, and accessto information.

Business processes.An ROI analysismust not only account for the improve-ments to relevant business processes,but also for the costs associated withtraining staff involved in using theproposed technology system.

Organizational environment.Othercosts and returns will be linked to theorganization, for example through alteredresource flows, performance changes,changes in work flows and internalrelationships.

External relationships.Linkages with theexternal environment may be significantas well. Resources may be committedfrom this environment to support theproject and additional costs maybe imposed on external persons ororganizations by changes in the wayservices are delivered or other businessis conducted.

Choosing the right type of

analysis

Choosing and using the various methods ofROI analysis requires sound knowledge and

judgment: knowledge about the methodsand judgment about how best to applythem. The methods chosen should fit theparticular questions asked of an ROIanalysis. Different questions requiredifferent measurement approaches to fitthem. In general, there are four types ofquestions that prompt or drive an ROI

analysis: financial, effectiveness, efficiency,and impact.

Financial: Can we afford this? Will it payfor itself?

An ROI analysis that answers thesequestions is based on expected savings andrevenue increases compared to the dollarcost of all expenditures on the new system.The measures are set by generally acceptedor legally mandated accounting standardsand practices that apply to the par ticulargovernment organization. The costs andsavings or revenue might be projected overa multi-year time span to show a paybackperiod or to estimate the present value offuture returns.

Effectiveness: How much bang for thebuck will we get out of this project?

The ROI analysis that will answer thesetypes of questions considers how much theinvestment contributes to achieving programgoals and producing the desired results. Itconsiders direct, indirect, and opportunitycosts. The indirect costs include such thingsas training and administration over time. An

opportunity cost could be the loss of returnor revenue you would have received hadyou chosen a different alternative. Themeasurement of returns will be expandedbeyond cost savings to include levels ofperformance relative to program or projectgoals.


9/52


When deciding how to prepare and presentan ROI analysis, therefore, it is best to takeinto account all the potential risks thatinfluence the decision process. Undertakingan ROI analysis should include attention tothe risk factors identified below.

Risk factors that can impact

investment decision process

Politics and policy factors

Public exposure to failure

Divided authority over decisions Multiple stakeholders Year-to-year budget cycles Highly regulated procurement

processes

Organizational factors

Complex program networks Misalignment of (or conflicting)

internal goals Lack of leadership support

Business process factors

Impact on existing process

Fear of changing work assignmentsTechnology factors

Rapid changes in technology Interacting with parallel systems Scale and complexity

Most of the risk assessment issues listedhere involve problems related to thinkingbeyond the boundaries of the project,measuring factors, or determiningprobabilities. Simply recognizing whereuncertainty and potential damage may lieis half the battle. Careful risk analysis,

based on the best available data andestimates, will surely assist in ROI analysisand improve planning, even if the amountor quality of data is less than ideal.

Considering these various risk factorscan help shape the style, emphasis orpresentation strategies employed tointroduce the analysis into the decision

3

Efficiency: Is this the most we can get forthis much investment?

The ROI that tackles this question requiresinformation about the greatest possiblevalue relative to its costs. Efficiency cannotbe separated from effectiveness. It is usuallyexpressed in terms of optimizing thevalue of a return for a given cost or input.Establishing that some particular result isthe best of all possible results requireseither examining many alternatives orsimulating performance in some way thatprovides a valid picture of what is possible.

This can be done for some (but not all) kindsof systems with sufficient resources anddata. However to do so can substantiallyincrease the cost and complexity of theanalysis.

Impact: Will the benefits to society (ourstate, our city, etc.) justify the overallinvestment in this project?

The analysis that answers impact questionswill be concerned with the larger social andeconomic benefits and costs of a project. Todefine and measure variables that representsocial costs or benefits requires morethan the typical economic or accountingframeworks. These measures are based oneither the specific program results desiredby an agency or on general social benefitsand improved quality of life. Though notimpossible, the breadth and complexity ofthis kind of ROI analysis is rarely found inIT investment planning.

How does the ROI analysisfit into the overall decisioncontext for IT investments?

Investment decisions in the public sector,whether they involve IT or not, necessarilytake place in a context of political andpolicy influences. No matter how solidor technically sophisticated an ROIanalysis may be, it will not likely be thesole determinant of an investment decision.


10/52


making process. Such considerations asthose listed here may also help in recruitingsupport for the conclusions of your ROIanalysis and guiding how the analysisprocess is positioned when seeking thatsupport.

Conclusion

There is no single right way to conduct areturn on investment analysis. Nor is there aConsumer Reports for ROI products and

services. In determining how to conduct youranalysis, the best advice is to focus on thestrategic objectives of the analysis alongwith the goals and business processes ofthe proposed project. This focus will helpguide decisions about the resources andmethods to use to conduct a sound andvaluable ROI analysis.


11/52


Chapter One: ROI and the Need

for Smart IT Investment Decisions

5

It [ROI] will let us begin to make assessments and decisions about funding aproject or developing a new service based upon some true data. That moves you

from having emotional debates about projects to having factual discussions.Gerry Wethington, Missouri CIO1

Government decision makers must makethe most of scarce resources and atthe same time respond to ever-increasingdemands for improved performance andnew technology. Thus the need for wiseinvestment in information technologycontinues to grow. Growing demand in theface of scarce resources generates hardquestions and close scrutiny of proposalsfor new investments. Whats more, thedismal failure record of many governmentIT investments raises legitimate concernsabout the value of these investmentsand why they so often fail to live up toexpectations, or even to work at all. As aresult, IT planning processes often include,or even require, a rigorous business caseto justify new IT investments. These includeways of assessing the costs and returns to

be expected from that investment, that is,return on investment (ROI) analysis. ThisGuide is designed to help governmentexecutives who need to design, direct,conduct, or work with the results of suchan analysis to make the most of theirinvestments in information technology.

Growing interest in assessing returns on ITinvestments has spawned wide interest inmethods of return on investment analysis.However, there is little agreement aboutbest practices or specific methods forassessing ROI. Professional publications

and consultant white papers present quitea variety of possible approaches. As aresult, government executives and decisionmakers have difficulty choosing or designinga return on investment analysis that is bothfeasible and appropriate to their needs. Tohelp administrators and decision makers

with these choices this guide presents anoverview of the purposes and concepts ofROI along with an introduction to basicmethods and example cases. It alsoincludes links to other resources for thosewho wish to explore some subjects ingreater depth.

This guide treats ROI analysis as par t ofthe overall decision making process for ITinvestment. The planners and designers ofan IT project can use ROI analysis to helppersuade decision makers to support theproject. Decision makers can use an ROIanalysis, indeed may even require one,as part of an IT investment proposal to aidthem in evaluating it. In either case, an ROIanalysis will be shaped by the situation inwhich it is designed and carried out.

Decisions about what sort of return oninvestment analysis to do, or whether todo one at all, will usually depend on avariety of factors. ROI analysis may or maynot fit the larger context of investmentdecision making. Decision situationsdriven by very short deadlines or highlyspecific policy directives may rule outextensive analysis. Scale matters as well.

An elaborate ROI analysis would hardly bejusti fied for a small-scale, low-risk projectthat requires a fast decision. By contrast,large complex projects are typically high-

risk propositions for which the added timeand cost of an extensive ROI analysis wouldbe fully justified. Even though justified, insome environments ROI analysis is notused at all in favor of best practice reviewsor benchmarking to evaluate investmentpossibilities. Current practices vary

1Running the Numbers, Government Technology, June 2002, p. 23.


12/52


considerably. In Iowa, for example, thestate government has a standard frameworkfor all agencies submitting IT projectproposals, including business case andROI analysis requirements.2 The US Officeof Management and Budget has imposedsimilar requirements on federal agencies.

Agencies and local governments in otherstates may develop their own internalbusiness case and ROI requirements.

Given the diversity of practice in the ITinvestment world, this guide takes aneclectic, non-prescriptive approach. It

treats the subject of ROI analysis from thepoint of view of a curious but uncertaindecision maker. The key issues facing thisdecision maker are whether to do an ROIanalysis and, if so, how. The guide doesnot advocate engaging in ROI analysisunder all circumstances, nor does it favorany particular technique. Instead it presentsan approach to understanding a range ofpurposes and methods for ROI analysisthat can assist that decision maker tomove forward with wise and effective ITinvestment choices.

An approach tounderstanding andusing ROI analysis

Decisions about how to use ROI analysisdepend on understanding the nature ofthe methods themselves and how theyrelate to the business setting. ROI analysis

in general is a rather diverse collection ofmethods, skills, tools, activities, and ideas.They all may be useful for assessing therelative value over time of some investment.These methods are not, however, a singleformula or predetermined calculation thatwill yield a simple yes-or-no answer tothe question of how to invest. ROI is nota silver bullet. Actually designing andcarrying out any kind of ROI analysisrequires making many choices amongthe ideas and methods available andconducting an analysis appropriate to thedecision situation. Different choices will

produce different results.

Consequently, a meaningful analysisof returns on investment in informationtechnology is far easier said than done.Choices about how to conduct an ROIanalysis should be based on criticalunderstandings about:

the strategic objective(s) of theanalysis,

the place of the proposed ITinvestment in the overall enterprise3,

exactly how the analysis should bedone (i.e., what data and methodsof analysis are best suited to thoseobjectives), and

how the ROI analysis fits in the overalldecision context for IT investments.

This guide will introduce you to thesefour basic understandings and provideresources for deeper investigation of each.

2The Iowa ROI approach can be found at http://www2.info.state.ia.us/roi/index.html. The Federal business case requirements are found in OMB

Circular No. A-11.3The enterprise can be a single agency or unit, or something as broad as t he education or justice enterprise. The U.S. Chief Information Officers

Council defines an enterprise in terms of enterprise architectures, or blueprints for systematically and completely defining organizations current

(baseline) or desired (target) environments. SeeA Practical Guide to Federal Enterprise Architecture, Version 1.1, Chief Information OfficersCouncil, February 2001, p. 2. More specifically, the National Association of State Chief Information Officers defines enterprise architecture as anoverall plan for designing, implementing and maintaining the infrastructure to support the enterprises business functions and underlying networks

and systems. See Enterprise Archi tecture Development Tool-Kit, Version 2.0, National Association of State Chief Information Officers, July 2002,p. 243.


13/52

CENTER FOR TECHNOLOGY IN GOVERNMENTRETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS 7

Understanding strategicobjectives in ROI design

Your understanding of the strategicobjectives of an ROI analysis will determinehow the analysis is ultimately done andused. The matter of strategic objectives hastwo related parts. One deals with theobjectives and context of the proposedinvestment. The second deals with theobjectives and context of the ROI work itself.

An adequate understanding of the first par tof the strategic objective must includeanswers to these key questions detailedbelow.

What are the programmatic and businessgoals of the proposed investment?

The value of an investment is directlyrelated to the programmatic goals andbusiness process employing the newtechnology. For example, the strategic goalof investing in a new financial managementsystem would be to improve the qualityof financial decision making and control ofresource flows, not simply to produce faster

or more complex accounting reports.Attention to the strategic objectives keepsattention focused broadly on the kinds ofbenefits sought and how to measure them,rather than narrowly on the technology.Focus on the strategic and businessobjectives will also lead to a clearerunderstanding of the full range of benefitsto be expected from the investment.

What are the needs or interests of theprimary customers and other keystakeholders with respect to productsor services involved and impacts on

business processes?

Stakeholders are critical players in thesuccess of new projects. Ignoring orunderestimating the importance of theirrole puts the success of an investmentat great risk and can lead to substantialunanticipated costs or missed benefits.One large organization recently developedan extensive new customer help deskapplication without the participation of the

staff who handled customer phone calls.Those staff members learned of the newsystem the day before it went live. The resultwas poor performance and a serious blow tostaff moralelower returns and highercosts.

How extensive is the analysis to be? Whatrange of costs and returns are expected?What resources and frameworks areavailable or required for the ROI analysisitself?

An ROI analysis can itsel f be a substant ial

investment. Extending its scope, levelof detail, or complexity beyond what isneeded for effective decision making isa waste of resources. Attempting an ROIanalysis that is beyond the resources orcapacity of the organization will also wasteresources. Choosing an appropriate scopefor the analysis is therefore a critical par tof the process. These choices determinethe details of the ROI analysis itself: kindsof data used, whether hard or soft estimateswill suffice, the kinds of projections,quantitative or qualitative data that areneeded; which financial or non-financial

outcomes (customer/user satisfaction,social or political outcomes, improvedequity) are all important. The answers tothese questions then influence the kindsof personnel, tools, and other resourcesneeded.

What are the risk factors and howmight they affect the projects costsand results?

Consideration of risk should always bea part of the ROI analysis. Risk factorsarise from the nature of the project

itself (complexity, scale, novelty), itsorganizational setting (conflict, resourceconstraints, top-level support, timepressures), and the larger environment(political turbulence, crises, and policyshifts). Risk analysis, discussed in somedetail later in this chapter, examines thelikelihood of risk factors affecting theproject and what elements of the projector its results are likely to be affected.


14/52


4Tom Pisello, in Infoworld , June 10, 2002, p. 47.

Defining and measuringthe costs and returns fromIT investments

If an ROI analysis were just one simplething, then there would be one simple wayto measure the costs, returns, and benefits.In practice, however, there can be manydifferent questions asked of an ROIanalysis requiring different measurementapproaches to fit those questions. Howto identify these measures and apply them

to analysis are the next parts of the puzzle.

Solving that puzzle requires understandingthe differences among the questions askedof an ROI analysis. There are four differentbut related types: financial, effectiveness,efficiency, and impactquestions.

Can we afford this? and Will it pay foritself?

Answering these questions requiresinformation about costs and returns in termsof the monetary value of the resources used(inputs), as measured and recorded bystandard financial factors. In its simplestform, an ROI analysis based on this kindof question would calculate the return interms of the expected savings and revenueincreases (if any) compared to the dollarcost of all expenditures on the new system.The costs, savings, and revenues might be

projected over a multi-year time span inorder to show a payback period or toestimate the present value of future returns.

8

IT OPERATING EFFICIENCYHow will the project improve IT, suchas simplifying management, reducingsupport costs, boosting security, orincreasing IT productivity?

RISKWhat are the potential risks associatedwith the project? How likely will risksimpact the implementation schedule,proposed spending, or derived target

benefits?

COMPETITIVE IMPACTHow does the proposed project comparewith competitors spending plans?

ACCOUNTABILITYHow will we know when the project isa success? How will the success bemeasured (metrics and time frames)?

REQUIRED INVESTMENTHow much investmentincluding capitalexpense, planning and deployment,application development, and ongoingmanagement and supportwill the projectrequire?

FINANCIAL BENEFITSWhat are the expected financial benefitsof the project, measured according toestablished financial metrics, including

ROI, savings, and payback period?

STRATEGIC ADVANTAGEWhat are the projects specific businessbenefits, such as operational savings,increased availability, increased revenue, or achievement of specific goals?

Table 1. IT ROI Questions from the CFO to the CTO4


15/52


How much bang for the buck will we getout of this project?

Answering effectiveness questions requiresinformation about the costs of the project inrelation to how much it contributes toachieving program goals and producingthe desired results. The metrics will bemore complex, involving unit cost or activitycost calculations. The measurement ofreturns will be expanded beyond costsavings or revenue increases to includelevels of performance relative to programor project goals.

Is this the most I can get for this muchinvestment?

Answering efficiency questions requiresinformation about whether the project willproduce the greatest possible value relativeto its costs. Efficiency questions poseserious analysis problems. Establishingthat a particular result is the best ofall possible results requires eitherexamining many alternatives or simulatingperformance in some way that gives a

valid picture of what is possible. This canbe done for some kinds of systems withsufficient resources and data, but cansubstantially increase the cost andcomplexity of the analysis.

Will the benefits to society (our state orour city) justify the overall investment inthis project?

Answering impact questions requiresinformation about the larger social andeconomic benefits and costs of a project.These questions pose two tough problems.Measuring the broad social and economiccosts of an investment requires data farbeyond what typical financial systemsprovide. Measuring, or even identifyingthe full range of social and economic

benefits from some government IT projectcan be even more daunting. The idea offiguring a cost/benefit ratio is appealing,but seldom feasible. Though not impossible,the breadth and complexity of this kind ofanalysis means it is rarely found in ITinvestment planning.

The four types of measurement questionsand approaches differ in several ways.The choice of relevant metrics is one criticaldifference. Some approaches are based onstrictly financial metrics (costs or returns indollar terms), others include production

output metrics, such as quality of goodsor services, client or customer satisfaction.Metrics may extend to organizational factorssuch as morale or to social and politicaloutcomes. These can include impacts onquality of life, social equity, social or humancapital, or political support. Thesedifferences are summarized in Table 2(page 11).

Understanding theenterprise: technology in

the business context

This guide views any IT investment projectas embedded in its organizationsenterprise architecture and in a contextwith three major elements: relevantbusiness processes, the organizationalsetting, and the external environment.5

These are illustrated in Figure 1 on page 10.The immediate context of any IT project isfound in the current business process(es).Some of the costs and returns of the projectwill be directly linked to business processes,such as training costs for staff involved or

the improved efficiency of the overallprocess resulting from project implementa-tion. Other costs and returns will be linked tothe organization, for example through

9

5While a more detailed discussion of enterprise architecture is beyond the scope of this guide, both the mapping and understanding of an

organizations blueprint are critical steps that any organization should accomplish prior to conducting effective return on investment analysis.

Please see the Additional Resources section in Appendix D for several links to enterprise architecture development tools available to federal,

state, and local government decision makers and ongoing enterprise architecture initiatives at the state level.


16/52


changes in resource flows, performancechanges and changes in workflow andinternal relationships. Linkages with theexternal environment may be significantas well. Resources may be committed fromthat environment to support the project,and additional costs may be imposed onexternal persons or organizations bychanges in the way services are deliveredor other business is conducted.

For example, an agency implementing EDI(electronic data interchange) to supportpurchasing transactions may be imposing

costs on vendors who wish to do businesswith the agency and have to invest indeveloping their own EDI capabilities. Theway Figure 1 below represents investmentcosts and returns as part of the samecontext is an important way of looking atROI. Neither the costs nor the benefits of anIT project begin and end at the projectsboundaries. Financial allocations to a newproject mean fewer resources somewhereelse in the organization or its environment.

10

Figure 1. The Context of IT Investment Projects

Changes in one part of a business processmay impose costs on other par ts that haveto adjust activity, retrain staff, or modify othersystems. Increased efficiencies in onebusiness process can make resourcesavailable elsewhere in the organization,but may also result in changes in otherlinked processes within the enterprise.

Whether you see something as a cost orbenefit of an IT project may depend onwhose perspective you take. The expendi-ture on a new personnel managementapplication is a cost to the agency that pays

for it, but it is a benefit to the vendor, who isan external stakeholder. These links make itclear that an analysis of costs and benefitsof an investment can require attentionextending well outside the project itselfinto the organization and its environment.Business processes are the criticalconnection between the project and the restof the organization. So attention to businessprocess linkages with the project is animportant part of the overall ROI analysis.


17/52


MeasurementQuestion

Can we afford thisand will it pay foritself?

How much bang forthe buck will we getout of this project?

Is this the most I canget for this muchinvestment?

Will the benefits tosociety (our state,our city, etc.) justifythe overall invest-ment in this project?

MeasuringCosts

Financial metrics; defined bypolicy and accepted accountingprinciples; reporting andcontrol-oriented; standards-based or consistent; not linkedto business process; ignoresimportant cost factors; shorttime frame; data routinelycollected/reported

Financial and outcome/qualitymetrics; operations andmanagement oriented; definedby program and businessprocess; may or may not bestandardized; often requiresnew data collection; mayinclude organizational andmanagerial factors

Financial and organizationalmetrics; management andpolicy oriented; non-standard-

ized; requires new datacollection and simulation oranalytical model; can revealhidden costs

Financial, organizational,social, individual metrics;individual and managementoriented; nonstandard; requiresnew data collection andexpanded methods; revealshidden costs; potentially longtime-frame

MeasuringReturns/Benefits

Savings as measured inaccounting categories; narrowin focus and impact; increasedrevenues, reduced total costs,acceptable payback period

Possible efficiency increases;increased output; enhancedservice/product quality;enhanced access and equity;increased customer/clientsatisfaction; increasedorganizational capability;spillovers to other programsor processes

Efficiency increases;spillovers; enhancedcapabilities; avoidance of

wasteful or suboptimalstrategies

Enhanced capabilities andopportunities; avoidingunintended consequences;enhanced equity; improvedquality of life; enhancedequity; enhanced politicalsupport

Table 2. Approaches to Cost and Return Measurement


18/52


Understanding ROIdecisions in their politicaland policy context

Investment decisions in the public sector,whether they involve IT or any otherexpenditure, take place in a context ofpolitical and policy influences. No matterhow solid or technically sophisticated anROI analysis may be, it will not likely be thesole determinant of an investment decision.It is useful in deciding how to prepare and

present an ROI analysis, therefore, totake into account possible political andorganizational factors. Such a considerationof external factors may help shape the style,emphasis, or presentation strategiesemployed to introduce ROI analysis intodecision processes. Such considerationsare discussed below and may also help inrecruiting support for the conclusions of theanalysis and guiding how the analysisprocess is described or defended.6 All ofthese considerations can be classified asdifferent types of risks.

Political risk factors

Public exposure of failure or error.Governments business ispublic business.Most new ideas are implemented in fullpublic view. Any investment-gone-wrongrisks not only dollars, but the credibilityof an agency and its leadership withlegislators, executive officials, and thepublic. So government tends to reduce risksby relying on the tried and true. Failurerisk can be mitigated by taking care not toover promise the benefits of new projectsand to ensure that there is adequate

strategic planning to reduce the probabilityof failure. Undue caution can also riska different kind of failure: missedopportunities for successful projects.

6The content of this section is adapted from Sharon S. Dawes, Theresa A. Pardo, Stephanie Simon, Anthony M. Cresswell, Mark F. LaVigne,

David F. Andersen, and Peter A. Bloniarz. Making Smart IT Choices: Understanding Value and Risk in Government IT Investments. Albany, NY:Center for Technology in Government, 2003. http://www.ctg.albany.edu/publications/guides/smartit2

Divided authority.Public executivesseldom have a clear line of authority overagency operations. Their decisions arecircumscribed by existing laws, budget andfinancial controls, civil service systems,political constraints, and a variety ofregulations imposed by both legislaturesand the cour ts. These restrictions impedemanaging the complexities of multi-milliondollar IT projects in a rapidly changingtechnical environment.

Multiple stakeholders.Stakeholderstypically have competing goals. Customers,

constituents, vendors, service providers,elected officials, professional staff, andothers all have some stake in IT projects.Understanding how different choices mayaffect and be affected by each stakeholdergroup helps to prevent unexpectedproblems.

Annual budgets. Government budget cyclesincrease the uncertainty about the sizeand availability of future resources. Thisdiminishes government agencies abilitiesto adopt or sustain new IT innovationssuccessfully, especially those that have long

development periods.

Highly regulated procurement. Regulationsin the typical competitive bidding processare ill suited for the experimentation andlearning that is often essential for successfulIT investments. While promoting integrityand fairness, procurement regulations areoften a source of problems and delays.These are especially troublesome whenagencies write requests for proposals(RFPs) that depend on the limitedinformation they have been able to gainfrom inadequate experience and research.


19/52


Organizational risk factors

Complex program networks.Governmentprograms are connected in many complexways to other programs in the same or otheragencies, or to non-governmental entities.Sometimes the connections are explicitand formal. Often they are informal orunintended. Changing programs can haveunintended consequences for severalothers, producing additional costs andproblems for the investment project.

Misalignment of goals. Some parts ofan organization may see the goal of an ITproject in narrower, possibly conflictingterms. For example, an IT unit may becomeenamored with a database or officeautomation project because of its technicalelegance. The end users of the system,however, may want capabilities that are notcompatible with the new technology. Withoutsome goal alignment a project is on the pathto failure.

Lack of leadership support andorganizational acceptance. Topmanagement support for a technologyinitiative is critically important. Similarlysupport and acceptance throughout theorganization, especially among the peoplewho will use the technology or itsproducts,are equally important, and often moredifficult to achieve. Understanding andenhancing support reduces or limits risks.

Business process risks

Reality of the process.IT inventions areembedded in business processes. Failureto understand and account for this reality

in project design and implementationintroduces major risks. Systems are oftencreated that do not serve business needs,are too expensive for the small productivitygains they provide, or are not flexibleenough to meet changing demands.

Technology risk factors

Rapid change.Obsolescence is a risk assoon as a project chooses its technology.The pace of technological change makesit difficult for planners to keep up with thedetails of new developments and tounderstand comprehensively how eachnew technical tool works, or may besuperceded. Technology choices guidedby long term strategies and stronglinkages to business goals can mitigatemany of the risks produced by rapidly

changing technology.

Technology interactions.A basic conceptof system analysis is you cant do just onething. New technologies interact with oldtechnologies and work processes. Theinteractions may enhance the value of theolder and newer technology, or interfere withboth. A careful analysis of these interactionswill identify risky situations and provideinsights for avoiding problems and errors.

Scale and complexity. Risks increasedirectly with the scale and complexity of ITprojects. Planning an incremental processof development, with careful contingencyplans, can mitigate some of these risksand will avoid problems that cannot beanticipated on the path from small to large,complex systems.


20/52


Chapter Two: Methods of ROI

Analysis for IT

Choosing and using the various methods of ROI analysis requires both goodknowledge and good judgment: knowledgeabout the methods and judgment abouthow best to apply them to a particular ITproject and its setting. This section presentsan introduction to the methods. It is intendedto impart enough knowledge so that thereader can exercise effective judgmentabout whether to use a par ticular method,how and to what extent it may be applied,and where to go for more completeinformation and resources.

We begin with a discussion of the issuesassociated with project time frames andscale, then present various approaches toprocess analysis and measurement. Weconclude with a brief overview of riskanalysis.

Issues of time and scale

The selection of method will depend tosome degree on the time frame andthe scale of the project and the ROIanalysis. Time frame refers to both thetime perspective for the analysis and theoverall time period of the project to beincluded in the analysis framework.

Anticipating the future. ROI analysis iscommonly used prospectively. The resultsof the analysis are intended to inform adecision about a future IT investment.For a prospective analysis, estimates ofanticipated cost and performance are basedon assumptions about the future that may

involve considerable uncertainty. Theresults of the analysis will depend, to alarge degree, on how accurate thoseassumptions turn out to be.

Learning from the past. A retrospective ROIanalysis will show actual performance dataabout the IT projects costs and returns. Thelonger timeline and complexity of largerprojects can lead to substantially different

results. The analyst must take thesepossibilities into account when evaluatingthe results of pilot projects. These concernsbecome part of the risk analysis discussedin a later section.

Importance of scale. Even when there areactual data from such a pilot project, theproblem of the accuracy of assumptionsremains. There is no guarantee that costsor returns from a small pilot project willaccurately predict what happens in alarger effort. The scale of the larger effortcan itself lead to different results.

In the case study described in Appendix A(page 28), the analysis combines the twoperspectives. The developers created apilot project, which was a small part of alarger project they planned to undertake.They then used the data gathered from aretrospective analysis of the pilot projectto estimate the costs and returns that wouldresult from the full development.

Importance of businessprocess analysis

IT projects can have large and wide-rangingimpacts on business processes. Theseimpacts can occur directly in the businessprocesses involved in the project and inother related processes. Methods for theanalysis of business processes are toolarge a subject to cover in detail in thisguide, so a brief introduction to thesetechniques is included in the section on

methods below. The reasons for includingattention to these methods in ROI work isrelated to the overall strategic objective ofthe project and how it fits into the rest of theorganization.

The business process perspective concen-trates on where an IT project fits within thelarger picture of the organizations missionor core purpose. For example, a statedepartment of transportation typically


21/52


has responsibility for ensuring the safetyof the transportation infrastructure. Thiswould usually include responsibility forinspecting bridges on state highwaysystems and contracting for repairs whereneeded. The main elements of that businessprocess includes many activities, fromdecision making about inspection policiesand schedules through contracting forrepair and construction work. The activitiescould include scheduling and performinginspections, reporting and analyzingresults, then moving into the RFI, RFP

development, bidding, contracting, andproject supervision. Any change in onepart of the IT system in an interconnectedbusiness process like this one, would likelyhave important impacts on other businessprocesses. An IT project that placeselectronic sensors on bridges that sendstress data to the central office will affectmore than the inspection process. It willhave ripple effects all the way from theemployment and training of inspectors to thecosts of new construction and the publicsperception of bridge safety. These latterconcerns are all potential elements in an

ROI analysis.

The interconnectedness of IT projects andthe overall business process is nicelyillustrated in one recent experience witha Web project in New York City. The Citygovernment added a section on its Website reporting the results of its HealthDepartment inspections of restaurants.The IT staff did not anticipate the popularityand high demand for such information. Nordid they understand the effects that demandwould have on other linked elements of thebusiness process, which includes both

producing and disseminating inspectionresults. The Web server was initiallyoverwhelmed by hits on the inspectioninformation soon after it was posted; wordof mouth communication rapidly spread thenews. That problem could be easily fixedwith additional server capacity. However,the high-level of public demand for thisinformation and its significant impacts on

restaurateurs put new pressures on theHealth Department. They were pressedto increase the timelines of inspectionsand to use more plain language to ensurebetter understanding of the results. Thesewere substantial cost factors directly relatedto the Web site project. A narrow focus onthe IT component of the project led theplanners to miss the effects on linkedelements of the business process. Theresults were both unanticipated costsresulting from down-stream impacts onthe business process and unanticipated

benefits due to increased service levels tothe citizens.

The linkages in this example extend fromthe specific details of an IT projectnewdata on a Web siteall the way to therelationship of the organization to itspolitical environment. These linkagesand their importance are obvious inretrospect, as are many implications forboth costs and returns. The goal of goodbusiness process and ROI analyses is toprovide the same insights and informationfor planning and decision making inadvance. How that can be done is outlinedin the description of methods below.

Benefits of process modeling

Business process models make the implicitassumptions and mental models ofindividual managers and stakeholdersmore explicit and open for discussion.They also:

inhibit premature jumping to a solutionbecause of the way they structure

thinking about a problem; create an externalized definition of the

problem that can serve as a focus ofdiscussion and can help to align thinkingabout what are root causes of observedproblems;

force managers and analysts to come togrips with the precise logic or causalforces that are causing a problem;


22/52


require attention to decide which keyvariables to measure;

allow analysts and managers tocommunicate their reasoning effectivelyand efficiently to external audiences;

can provide a simulation of how the fullsystem will operate within a full contextof organizational and human factors;

push managers to see the implicationsof a limited prototype when it isexpanded to full scale operations,requiring careful attention to technical,organizational, and policy issues; and

can include financial elements that allowexplicit exploration of costs and benefitsof proposed solutions.

Sensitivity analyses of models provideanswers to what if questions about varioustypes of system functionalities and possibleorganizational and human effects. Thishelps planners anticipate issues andproblems before they are encounteredin a real world system implementation.

Methods for understandingthe business process

IT investments of any sort must be viewedas part of the business processes in theirorganizations. Both the costs and returnsof the investment are tied to how the newtechnology fits with and how it altersbusiness activities. The notion of whatconstitutes IT must be expanded to includenot only the chips, wires, and software, butalso the activities and interactions thatgenerate the costs and value that resultsfrom using the IT (i.e., the enterprise

architecture). Failing to map andunderstand the enterprise invites a badlyflawed understanding of how the ITinvestment will work, and can be a short cutto failure. For example, the U.S. Departmentof Education recently invested millions ofdollars in a Web site for college students toapply for financial aid. The site generatedvery little use and was closed down. Itfailed because it ignored the important

role of college financial aid officers in theoverall process of students seeking andreceiving aid. The designers did notunderstand or ignored the overall businessprocess of financial aid administration.

Business process analysis tools andmethods range from simple flow andGANTT charts to sophisticated formaland mathematical modeling. Simple flowcharts are sufficient to identify the basicelements of most business processes andsome of the more obvious dependencies.How far business process modeling needs

to go beyond simple diagrams dependson the questions to be answered andthe resources available for the work.

The basic components of a businessprocess are activities, flows, controls,and dependencies. Analyzing thebusiness process therefore meansidentifying the various kinds of activities,what kinds of flows they generate andsupport, and how the activities and flowsare controlled and linked to produce someparticular collection of goods or services.Flows can consist of information, persons,

resources, control inputs, and workproducts moving from one activity toanother. The analysis consists of gatheringinformation about the activities and flowsin as much detail as necessary. Dependingon the kinds of modeling to be used, thisinformation can include descriptions ofactivities and flows in strictly qualitativeterms or detailed measurements ofresources used, flow metrics, and outcomemeasures.

Descriptive models

Approaches to process modeling can begrouped roughly into three levels ortypes: descriptive, analytical, anddynamic. Descriptive models are themost fundamental. They describe theelements of a business process and therelationships among them. A flow chartthat describes how travel expensereimbursements are processed, for example,might show how the request originates, how

16


23/52


it moves from one work process to another,what actions or decisions take place at eachpoint, and the elapsed time for each step.Descriptive models may involve somemeasurements, such as personnel time,expenditures, other resources used, andoutputs. The flow chart in Figure 2 belowillustrates the actual billing information flowsfor a nonprofit provider of services tomentally retarded and developmentallydisabled clients. It shows how informationabout services delivered is converted intobills submitted to the various governmentsources of funding for these client

7For example, Medicaid billing goes to the Medicaid Management Information System (MMIS), education related billing goes to the State Education

Department, and so forth.

services.7 This flow chart was producedto help understand how a change in thereporting and processing technology forone of the funding agencys would affectoperations among the providers who dealwith multiple agencies. The IT projectwas planned by one state agency, butthe planners recognized that the businessprocess involved a range of other agenciesand local practices. Therefore their analysisof the costs and potential benefits of theproject had to include attention to theoverall business process, not just whathappened within one state agency.

17

Figure 2. Example of Process Flow Chart


24/52


The level of detail used for such a model isa matter of judgment, in this case dictating amodel at an intermediate level of detail. Forsome purposes, a more highly aggregatedmodel may be sufficient, such as thefollowing.

It might be necessary to examine amuch finer level of detail, such as how

administrators review service recordsbefore approving billing in Figure 2 onpage 17. The level of detail depends onthe questions that guide the overall effort.

A descr iptive model influences the choiceof what is to be measured and whethermore formal modeling will be needed.

Use of analytical or

formal models

Analytical or formal models go beyonddescription to represent the performance

of a business process or project in somequantitative or mathematical form. Thiskind of model has many advantages foranalysts and decision makers, along withsome substantial problems and limitations.There are at least four kinds of advantages.

Formal models require making theimplicit explicit, and to do so in preciseand systematic ways.

Many formal modeling methods allowexploring what if? questions and workwith alternative scenarios by simulatingthe results of using the model of the new

system. The planner can thus test ideasand alternatives without risk to realsystems, operations, or data.

The analysis yielded by the modelingcan reveal insights into the behaviorof persons or organizations that wouldotherwise be difficult or impossible todetect.

The modeling process, if done in acollaborative way, produces a sharedunderstanding of the work amongmanagers and stakeholders that canlead to improved decisions about theproject and its ultimate implementation.

Because of these advantages, the useof these techniques for process improve-ment and project planning has grownsubstantially and spawned an industryproducing software applications, consulting,and training in one or more of thesemethods (see resources in Appendix D).

These methods are not without problemsor limitations, the most important of whichis that they raise the costs of analysis inseveral ways. These models typicallyrequire more detailed and extensiveinformation, particularly performanceand outcome measures, than simplerdescriptive methods. In addition, theconstruction and use of these modelingmethods requires considerable time andexpertise. Organizations that do not havestaff prepared to do the modeling workthemselves would have to invest either in

extensive training or employing consultants.It is important to review carefully whetherthe complexity of the project justifies theinvestment in formal modeling beforetaking that course of action. The reviewof alternative methods below is intendedto help in that regard.

Types of formal

modeling approaches

Agent-based modeling. The basic idea ofagent-based modeling is that complex

behaviors of collectives (social groups,members of an organization, population)can be modeled by simple computationalrules. The rules treat the members of thecollective as autonomous agents that followsimple rules of behavior, rules that can bemodeled on a computer. What happens to


25/52


the collective is the result of the individualagents interacting with each other accordingto the rules. Changes in the rules changethe overall results, sometimes in predictableand sometimes unpredictable ways.

One early example of agent-basedmodeling in the social arena was a modelof residential segregation developed inthe late 1960s.8 The model starts witha hypothetical city with a mixed racepopulation randomly distributed overthe residential area. Each householdeither stays or moves according to two

simple rules: a preference for living nearpersons of the same race, and decidingto move if the population in surroundingareas has too low a proportion of suchhouseholds. There is no rule involvingdislike for households of different raceor desire to avoid them. Nonetheless,the action of these rules over timeproduces a city of racially segregatedneighborhoods.

In agent-based modeling, the rules arethe model or theory. Thus they makeexplicit the assumptions and motives of

the agents and what governs their actions.The value of the model depends on thedegree to which the rules produce resultsthat match the actions and outcomes forthe actual collective of interest. If rulesare developed that reproduce the resultsof interest, they can provide a powerfultool for understanding how those resultsoccurred and for testing ideas about howto change or improve them. This kind ofmodeling may be particularly appropriatefor information technologies that willinvolve the activities of large numbersof persons engaged in the same type of

activity, such as large numbers of citizensseeking the same kinds of information ona Web site. A valid model of how peoplewill use a system can be a valuable tool forexploring what costs and benefits to expect.

Agent-based modeling is not for thefaint of heart, however. It is mathematical,often requiring advanced knowledge ofalgorithms and computing to implement.There are special modeling languagesfor developing the computer programsto pursue agent-based work, butthey require experienced users or aconsiderable investment in training. If therules are sufficiently complex or abstractthey may be difficult to communicate toparticipants in the planning or to keystakeholders.

Operations research and statisticalmodeling.If the factors that influence abusiness process can be identified andmeasured, modeling that process byoperations research and statisticalmethods may be feasible. These modelingapproaches use mathematical equations torepresent the business process. Thisrequires measurements of the businessprocess itself and of the factors that arethought to influence how the process works.Then the modeling equations are chosen tofit the way the significant elements of thebusiness process are conceptualized.

If the process is seen as a series ofactivities that consume resourcesand occur in particular interdependentsequences, project description toolssuch as PERT charts or critical pathanalysis may be used.

If the sequence of events in thebusiness process is uncertain, orif multiple paths or outcomes arepossible, probabilistic methods such asMarkov chain models may effectivelyrepresent the process.

If the sequence or path of influence

is not known, but some measureof their general effects is available,linear modeling and regression methodsmay be useful to represent the overallprocess or predict the results of changes.

8 Thomas Schelling, (1971) Dynamic Models of Segregation, Journal of Mathematical Sociology1, 143-186.


26/52


If particular combinations of influencesare thought to work together to affectthe flow and/or performance of thebusiness process, then scenarioanalysis may be useful.

These modeling methods are similar toagent-based methods in that they requiremeasurements. They differ in that theyare based less on specific theories orassumptions about the activities thatmake up the business process. Rather,they use simplifying assumptionsabout the underlying cause and effect

relationships that drive the process. Thatmakes the models easier to use in somerespects. Little original theory building isneeded. In addition, the methods aresupported by commonly availableapplications, and training in many of theparticular methods is typically part ofmanagement education programs.Choosing among these alternatives is nota straightforward task and a reasonablyhigh level of expertise is needed. Thesame applies to interpreting the resultsand understanding the limiting assumptionsthat apply to each.

System dynamics modeling. Manybusiness processes involve feedback,that is, chains of effects in which whathappens at one point in the process hasan impact elsewhere that in turn influences(increases or decreases) the original effect.For example, a new database applicationmay produce a new kind of report that usersfind particularly valuable. Seeing onepossible new outcome prompts the usersto request additional kinds of reports thatproduce more valuable results, and morenew requests, and so forth. Any kind of

process that can be represented as stocksor supplies that flow from one place toanother with either reinforcing or negativefeedback can be represented using themethods of system dynamics modeling.These models can provide both a concep-tual and mathematical representation of theprocesses. The conceptual or graphicalrepresentation is usually referred to as a

causal loop model. These models can bebuilt by analysts working with the staffinvolved in the process, making use of staffknowledge and insights. The image createdis a potentially powerful tool to evokeknowledge about the process and tomove the participants to a sharedunderstanding. An example of such adiagram is shown in Figure 3 (page 21).This particular model represents a workingtheory about how work progresses. Taskscan exist in one of the four following states.

Work to do

Undiscovered rework

Known rework

Work really done

At the start of work, all tasks are in thestock of work to do. As par ticipants performwork, correctly done tasks move to thestock of work really done, with a probabilityof 1- error rate (a fraction). The error rateis based on the idea that it is impossibleto perform all tasks correctly on the firsttry. Tasks performed incorrectly requirerework, so as work proceeds some propor-

tion of the tasks enter the accumulationundiscovered rework, according to theprobability of an error occurring, errorrate. Similarly, problematic tasks canbe redone correctly (moving from theaccumulation of known reworkto workreally done) or incorrectly (returning toundiscovered rework), based on the errorfraction. The concreteness of instructionsand transformability of the tasks themselvesare shown as affecting the ability to dowork right in the first place or to correcterrors. With appropriate assumptionsand values for the variables and

parameters in such a model, a mathematicalrepresentation can be built that reproducesthe behavior of the process itself. Analysts,who should have appropriate expertise insystem dynamics modeling, can changeassumptions or values and explore theconsequences, yielding new insights intothe possible costs and performance of aproject.

20


27/52


Figure 3. Causal Loop Diagram of Project Management Processes

Unified Modeling Language (UML) and UseCase Modeling. UML is a programming

and modeling language that provides asystem for representing and documentingobject-oriented software development andthe business process in which it resides.9

It is par ticularly tailored to analyze theapplication development process andfor use by development teams in thatenvironment. The overall conceptualscheme for use of the UML includes whatis referred to as Use Case Modelingas acomponent. A Use Case is a guide forsoftware design based on a behavioral casestudy of how an application is used in anactual business process. This differs from

traditional approaches that specify softwarerequirements in terms of technical needsand goals. For IT projects that are primarilyapplication development, UML can be avery useful and powerful tool for modeling

the project itself for management purposes,as well as for its linkages to the business

process. However, the UML is highlytechnical and requires considerable studyand programming experience to useeffectively.

Investment in the use of UML may bejusti fied, especial ly for large appl icat iondevelopment projects that involve multipleteams. As the UML documentation states:

In the face of increasingly complexsystems, visualization and modelingbecome essential. The UML is awell-defined and widely accepted

response to that need. It is the visualmodeling language of choice for buildingobjectoriented and component-basedsystems.10

9The Unified Modeling Language (UML) provides system architects working on object analysis and design with one consistent language for

specifying, visualizing, constructing, and documenting the artifacts of software systems, as well as for business modeling. Object Management

Group, p. xi.10Object Management Group. OMG Unified Modeling Language Specifica tion, Version 1.3. Framingham, MA: Object Management GroupHeadquarters, 1999, p. 1-4. 13 Office of the New York State Comptroller.Accounting and Reporting Manual. Albany, NY (no date), p. 17.


28/52


Workflow modeling.The workflowmodeling approach is similar to UML butinvolves a different focus and approach.Instead of using a programming languageto model work processes, workflow methodsuse generic models of work processesand the relationships among them. Thecomponents of these generic modelsrepresent business activities, resources,dependencies, and controls. The designof the models is aimed particularly atenhancing coordination by identifyingdependencies, failure modes, andproblems of handling exceptions. For

complex business processes, a flow chartwill not reveal the consequences of errorsat critical points, unanticipated resourceshortages, or missing factors during somepoint in the process. Workflow models allowthese situations to be explored and provideanalysis of where processes are particularlyvulnerable to failure or waste.

Workflow modeling technologies andcommercial products may be useful toanalyze an existing business processor re-engineer one. But they are designedprimarily to develop automated work

processes. These depend on computer-based controls, typically where the flow ofinformation or materials involves networkedlinks among the activities. However, thebasic concepts of dependencies, exceptionhandling, and error modes may be appliedregardless of the level of automationin a workflow. So the conceptual toolsof workflow analysis may contribute to aprocess model concerned with costs andreturns to a new system.

Measuring costs andreturns

Financial metrics and

government accounting

What is measurable within a governmentaccounting frame is set by the generallyaccepted and/or legally mandatedaccounting standards and practices thatapply to the particular governmentorganization. Definitions and structuresfor financial data are standardized anddesigned to serve specific governmentpurposes. The Office of the New York StateComptroller describes the state frameworkin these terms:

The purpose of classifying accounts is toprovide a standard format for recordingand reporting financial transactionswhich allows comparisons to be madewith others municipalities or otherfinancial periods.

The classification system serves as a

basis for budgeting, accounting, andreporting as well as for administrativecontrol purposes, accountability to theOffice of the State Comptroller and thegeneral public, cost accounting, and thecompilation of financial statistical dataon the state level.11

This framework organizes the accountingdata about costs/expenditures and revenuesaccording to general organizationalprograms, functions or divisions, and byobjects of expenditure (e.g., salaries,equipment, etc.). This is sometimes called

a function-object accounting or budgetingsystem. A typical function-object accountingsystem for a state agency has a highlydetailed and structured way of definingwhat kind of financial information is to berecorded and how it is to be organized.

22

11Office of the New York State Comptroller. Accounting and Reporting Manual. Albany, NY (no date), p. 17.


29/52


The strengths of a government accountingframework for analysis of costs andbenefits are also its weaknesses, namelystandardization and rigid structures andrules. Standards and rules make such aframe-work useful for control of finances,generating standardized reports, andoverall fiscal accountability. And if theexisting categories of accounts happento provide the particular cost and returndata needed for ROI analysis, theframe-work would be readily available anduseful. However this is seldom the case,particularly for IT investments. The full value

of an IT investment is typically a mixture ofmany types of expenditures and othercosts, the accurate accounting of which isnot possible in the traditional accountingsystem. These systems will usually recordthe costs of equipment and personnelattached directly and full-time to IT activities.But they seldom record the changes inpersonnel costs for those who use thetechnology or the investments in trainingand learning to operate new systems.Maintenance and supplies may appear ina range of separate accounts, and manymanagerial and analytical costs are very

difficult to estimate or prorate accurately.

Measuring costs and

cost-effectiveness

In its most basic economic sense, a costis whatever you have to give up to getsomething else. And a return is anythinggood that you get as a result, whether thatgood is measured in financial terms orotherwise. This is, of course, a muchbroader view of costs and returns thanis found in most accounting systems.

For example, an accounting system wouldnot likely include as part of the cost of anew computer system the amount of timedevoted to the purchase decision. Norwould the accounting system record thebenefits of increased staff morale from amore functional or reliable system.

A full consideration of costs requiresattention both to opportunities and toindirect costs. An opportunity cost

framework takes into account whatalternative actions or returns would bemissed or foregone as a result of aparticular investment decision. The path ofany IT project necessarily excludes otherpaths not taken. If opportunities along thoseforegone paths can be identified andvalued, they can become part of the costcalculation. If I choose to implement a newprocurement processing system, forexample, the staff time spent training forimplementing the new system is timeunavailable for training in some other areaof work skills. If a value can be assigned,

this lost opportunity may be consideredpart of the overall cost calculation. Indirectcosts (sometime referred to as imputedcosts) are those that are not incurred ormeasured directly, but are calculated orestimated from other measures. Forexample, the cost of using existing networkcabling for a project could be calculatedfrom the amount of new traffic generated orfrom some other prorating factor. Similarly,apartment renters do not pay real propertytaxes directly, but the renters portion ofthat tax can be calculated as an indirectcomponent of rent.

These costs are often off the books inthe sense that typical public accountingand financial management systems donot provide this kind of cost analysis.There are no standard accountingpractices to guide the measurement ofthese costs. An agency may have anexisting standard indirect cost factor thatis used in budgeting for Federal grants orcontracts (e.g., some fixed percentageof personnel expenditures). But thesestandard factors are based on averagesover many projects and are not particularly

useful measures for any one particularproject. Opportunity costs are evenmore problematic and are often notincluded, leading to possibly significantunderestimation of overall costs.Underestimating true costs can makean investments returns look more attractivethan they may really be.

Assessing effectiveness requiresidentifying the outputs of the project and


30/52


its implementation in business processes orprogram terms. That means identifyingmeaningful units of output that can berelated to the cost estimates. For example,the effectiveness of a new system to processbusiness permit applications over the Webcould be measured in terms of increasednumbers of permits processed during agiven time. Lower costs per processedpermit would be a useful cost/effectivenessindicator. Producing such a cost/effectiveness measure requires detaileddata about permit processing and a way ofassigning costs on a per permit basis.

Without some enhancement, most govern-ment accounting systems do not provide thebasic data necessary for such a calculation.It may be necessary to do detailed datacollection on a sample of permit transac-tions, for example, to establish a baselineunit cost figure. A business process modelcould provide the necessary data if sodesigned. Even if such a unit cost measureis available, it may have problems, such asthe distorted assumption that all permitsare equally important or costly.

Efficiency measures

Efficiency is a way of describing theeffectiveness of a project or system inrelationship to costs (or other inputs).Efficiency cannot be separated fromeffectiveness, since using resources andfailing to achieve a desired outcome can belittle more than waste; you dont save moneyby building half a bridge. Efficiency isusually expressed in terms of optimizing thevalue of a return for a given cost or input, oralternatively minimizing the cost for a givenvalue of result. It is possible, of course, toimprove efficiency without necessarily

achieving an optimum. As long as it ispossible to compare cost/effectiveness orreturn ratios for alternative systems ormethods it is possible to make judgmentsabout efficiency. To demonstrate an optimumresult or projection, it is necessary to havea simulation or optimization calculation toprovide the data. Some operations researchand simulations, such as linear program-ming or workflow simulations, can generate

this type of calculation if the necessarymodels and data are available. Howevermost IT projects do not have the data oranalytical resources to include thesemethods in an ROI analysis.

Impact measures

The identification of variables that make upa social cost or benefit calculation alongwith their definitions is broader than boththe economic and accounting frames. Theyare based either on the specific program

results desired by an agency or on generalsocial benefits and improved quality of life.These impact measures can come fromseveral sources.

Operational data.Some useful measures ofsocial or broader economic outcomes maybe available in the data collected duringordinary program operations. Providers ofservices for homeless persons, for example,routinely collect data about programs andactivities of clients that could be indicatorsof impact on their quality of life or progresstoward independence. Similarly, lawenforcement agencies routinely collectcrime statistics that can be useful indicatorsof neighborhood climate or quality of life.

Social and economic statistics.Government agencies collect statistics onsocial and economic indicators that arerelevant to the overall social and economicstatus of their jurisdiction. These range fromthe enormous resources of the US CensusBureau to state and regional planningagencies, and include market researchstatistics available from private sources.

Special studies and evaluations.For large,

high cost projects separate data collectionactivities, such as surveys or field research,may be used to collect and analyze dataabout social and economic impacts of aproject. These efforts may be expensive andtime-consuming, but may also be the onlyway to obtain data about particularoutcomes. The case example in Appendix Con Social Return on Investment providessome examples of how this may be done.


31/52


How time perspectives

change the measurements

Time has important effects on themeasurements and calculations. Both thecosts and returns of an IT investmentextend over time. Moreover, the costs maybe incurred long before the returns arerealized. So when estimating the moneyvalue of costs and returns, it may benecessary to take into account the effectof the time perspective on the value ofmoney. That is, a dollar in hand today is

worth more than the promise of a dollarin hand a year from now. How much lessthat future dollar is worth, known as thediscount, depends on the current timevalue of money, which is usually calledthe interest or discount rate. Thus if thediscount rate is 5 percent per year, thenthe promise of a dollar a year from now isworth only 95 cents today. The farther outin the future, or the larger the discount rate,the less the present value of the promiseof that future dollar. Calculations based onthe discounted value of future returns areuseful to estimate what size of return is

needed to justify the costs of an investment.The typical methods used in this kind offinancial analysis are called Net PresentValue and Internal Rate of Returncalculations. Both methods use an initialcost and some assumed figures for a futurestream of returns, along with a reasonableassumption for the interest (or discount)rate. The Net Present Value (NPV)calculation estimates the value today ofreturns coming in the future, less the cost ofthe investment. If the NPV result is less thanthe cost, then the investment does notappear to yield a net gain. The Internal Rate

of Return (IRR) calculation yields the samesort of result by a different path. Usingthe same basic information as the NPVcalculation the IRR calculation yields therate of return for the investment assuming abreak even, or NPV of zero. If the IRR is less

than the current interest rate, this indicatesthat putting the investment in an interestbearing bank account would yield a biggerfinancial payoff than investing in the project.

An example of NPV and IRR calculationsare shown in Table 3 (page 26). Thisexample is based on a hypotheticalinvestment analyzed over a six year period.It shows that when the time value of futurereturns is taken into account, the totalreturns must exceed the total cost by asubstantial margin to generate a positivenet present value or internal rate of return.12

The problem of choosing the

project or investment life cycle

Notice that any calculation of this sortrequires specifying a time-frame withinwhich the value of the project orinvestment will be assessed. This timeframe is sometimes referred to as the lifecycle of the project. Any IT investment isassumed to have a limited useful life,beyond which the returns are notconsidered, or when the expected returnsfrom some new technology will lead to

replacement of the previous investment.For IT investments, the rapid change intechnology makes the choice of areasonable life cycle or planning frameworka difficult but necessary part of the analysis.

Risk analysisin the public sector

Risk analysis consists of assessing theimportance of threats and how to mitigate oreliminate them. We usually evaluate a threat

in terms of how likely it is to materialize andhow much damage or cost would result if itdid materialize. We know, for example, thata large asteroid could hit the Earth anddestroy a continentan enormous threat interms of consequences. But astronomers

12The discount rate shown in these calculations was c hosen for illustration only.


32/52


13Calculates the net present value of an investment by using a discount rate (rate) and a series of future payments and income (values) in eachperiod (i) for nperiods.

tell us that the likelihood of that happeningis very, very remotetherefore, a smallthreat in terms of probability. Because ofthe low probability, the threat is seen as

small enough that few of us (asidefrom some science fiction writers andastronomers) take it into account inday-to-day life. The same logic applies torisks in IT investment. Consequently riskanalysis can be thought of as having fourbasic steps:

identifying the source of threats,

assessing the extent of potentialdamage or cost to the project,

assessing the likelihood of the threatmaterializing, and

devising ways to reduce or eliminatethe threat (i.e., mitigate the risk).

Table 3. Net Present Value Calculation13

Year

1

2

3

4

5

6

Total

Costs

$ 100,000

$ 5,000

$ 4,000

$ 4,000

$ 4,000

$ 3,000

$ 120,000

Returns

$ 15,000

$ 20,000

$ 25,000

$ 25,000

$ 30,000

$ 35,000

$ 150,000

3.0%

$ 13,297

6.0%

Discount Rate

Net Present Value

Internal Rate of Return

Threats can be reduced by taking stepsto lessen the damage or cost that wouldoccur if the threat materializes and alsoby reducing the probability that the

threatening event or action will occur. Forexample, a project could call for using themost reliable platform for a critical databaseapplication, reducing the probability thatthe system will experience a failure. Theproject could also implement a backupsystem capable of taking over processingif the primary system does fail, reducingthe potential damage of a failure.

The overall subject of risk analysis is toolarge to treat in detail here. However, mostof the risk assessment issues describedabove involve problems of thinking beyond

the boundaries of the project, measuringfactors, or determining probabilities. Thisshould not discourage risk analysis.


33/52


The experience of those involved in ITprojects can be a rich source of intelligenceand experiential data on which to basereasonable estimates of risk potentialand problem sources. The literature onIT investment is another rich source ofanalyses of successes and failures thatprovide additional insights into risks andmitigation strategies. Simply recognizingwhere uncertainty and potential damagelie is half the battle. Careful risk analysis,based on the best available data andestimates, will surely assist in ROI analysisand improve planning, even if the amount

or quality of data is less than ideal.


34/52


In order to better understand the problems and issues of ROI analysis, the Center forTechnology in Government decided to usea form of ROI analysis for one of its own ITinvestment projects. The Centers missionto build and disseminate knowledge aboutIT uses in government usually leads to

projects with other organizations. But thiscase provided an excellent opportunity forlearning and knowledge building as partof the Centers own internal work. TheCenters Web site is a critically importanttool in disseminating information andmaintaining contact with colleaguesand customers.

During the time covered by this case, theWeb site consisted of approximately 3500pages and experienced over 1000 visitorsper day. The Web site was supported bya full time Webmaster and par t time

maintenance and development staff of twoprofessionals and two graduate studentswho are part of the Centers TechnologyUnit. Th

Documents

ROI Understanding