Factors Affecting Return-On-Investment (ROI) in Automation

<RoI in Automation>

Factors Affecting Return-On-Investment (ROI) in Automation

Author:

Viveka Nand Bharti

<RoI in Automation>

Background / Objective:

The objective of this paper is to provide the guidelines for quantifying the hard benefits

and soft competitive benefits that are important in developing a complete picture of the

total Return-On-Investment (ROI). This paper helps the decision makers in evaluating the

financial viability by comparing the magnitude and timing of the expected gains to the

investment costs. This document primarily takes test automation as classic example for its

ROI analysis.

<RoI in Automation>

TABLE OF CONTENTS

1 Introduction..................................................................................................................42 Benefits.........................................................................................................................53 ROI Computations.......................................................................................................104 Case Study..................................................................................................................15

4.1..................................................................................................................................15Automation of Smoke Test examples:............................................................................15

5 Conclusion..................................................................................................................176 References..................................................................................................................187 Author’s Profile...........................................................................................................19

<RoI in Automation>

1 Introduction

Any project team or an organization, which wants to go for test automation, will first think

in terms of COST benefit and TIME/Effort gained. Return-On-Investment (ROI) is the first

step in the way of test automation, but it is very important to note that this is not the only

thing to keep in mind before going ahead with implementing test automation. There are

many more compelling benefits for and some of these benefits are soft and difficult to

measure. Releasing poor quality software can bear negative financial and brand name

consequences to the departments deemed responsible within the organization and to the

organization as a whole.

A benchmark comparison conducted in 1995 by the Quality Assurance Institute (QAI)

analyzed the specific difference in effort, measured in man-hours to perform testing using

manual methods as compared with using automated test tools. The testing involved 1,750-

test cases that uncovered 700-application defects. The results reflect an overall 75%

reduction in the software testing lifecycle.

This paper provides guidelines for quantifying the hard benefits and soft competitive

benefits that are important in developing a complete picture of the total ROI. In this way

this paper not only takes the financial aspect of ROI but also the tangible aspect of it.

<RoI in Automation>

2 Benefits

Tangible Benefits:

A. Accelerated Time to Market

In today’s dynamic economy, delivering products to market quickly has become an

absolute mandate. Every day a product is not available in the market, revenue is reduced

by sales being lost to competitors or technology obsolesces. Of course, this situation is not

completely avoidable — there is no such thing as immediate delivery of a new product to

market. But minimizing time to market is a goal that just about every organization strives

for.

Test Automation offers the organization a way to reduce time to market by compressing

the final stage that the product must complete before being delivered to market. Although

testing does not usually comprise a major component of time to market, it is a component

that can be easily and measurably reduced.

What is this reduction worth to the organization? Again, this varies considerably, but as a

rule of thumb:

Value of reduced time to market = time savings (in weeks) x the unit-production per week

x the margin per unit

This value reflects net income recaptured by delivering the product to market faster.

This can be an amazing figure. This formula multiplies 3 measures, any one of which can

be substantial, and when multiplied can produce very large pay-offs very quickly.

B. Reduced Cost of Testing

Probably the most obvious payoff comes in the form of reduced resource required to

conduct the testing process itself. Testing can’t be conducted by just anybody; it usually

requires an engineer that who is familiar with the design of the product, the characteristics

of likely or known problem areas, the sequence of events required to recreate a particular

state, and so on. These people are expensive to the organization, both in terms of the

amount they are paid and in terms of the opportunity cost of dedicating them to the

mundane and highly repetitive task of testing. Reducing these costs is one major

advantage of test automation.

How much will automation reduce the cost of testing? This of course varies, but consider

these relatively reliable factors:

<RoI in Automation>

Automated testing is faster. Automated tests take less elapsed time

than manual tests simply because automation executes test steps faster

than humans do. Further, automated tests do not suffer from the cumulative

effect of the minor distractions and delays that are inevitable with human

testers: they don’t take breaks, don’t need to eat, don’t take calls from their

family, and so forth.

Automation can execute tests 24 hours a day, 7 days a week. With

manual tests, the tester is involved from beginning to end — nothing

happens unless the tester is physically pushing the buttons. With an

automated test, the tester only needs to start the script and evaluate the

results. This means that organizations that currently use a single 8-hour

shift for their test cycles will immediately realize a 67% reduction in elapsed

time by going to 24-hour/day testing. Factor in weekends and holidays, and

the benefits get even larger. To get a feel for the impact of this

improvement, consider that a human tester works about 2,000 hours per

year out of a potential of approximately 8,760 hours. If the organization can

fully exploit 24x7 testing, then the compression of the test cycle will be over

75%.

Incremented testing time due to automation can be used to cover

more test scenarios, thus by decreasing the field errors possible.

Automated testing also yields the other benefits like person

independency, faster transition across test engineers, and less

training and easy product maintenance.

As a rule of thumb, once an organization has fully automated a given test procedure, the

test cycle can be easily compressed by 50%, more frequently by 75%, and sometimes by

up to 90%.

To be fair, getting to a point where the organization can realize these gains does involve a

price: the organization needs to become familiar with the test automation product, and

needs to create the scripts that automate the test process. The net effect of all these

considerations is that, as with any sophisticated enterprise product, there are some start-

up costs that must be charged against the productivity gain offered by automation. Once

these costs have been covered, the savings offered by automation rapidly accumulate to

deliver a substantial benefit for the organization.

C. Reduced Cost of Quality

<RoI in Automation>

Although "quality" can be an abstract concept, the "cost of quality" is highly tangible, and

in most cases represents a trade-off that every organization makes, consciously or

otherwise. Organizations that skimp on testing do save money on their test expenses, but

these are at least partially offset by the need to fix defects once the product is released to

market.

In this discussion, we will use a simple cost of quality metric developed by Rex Black,

author of Managing the Testing Process and a well-known consultant specializing in test

practice and theory. In his presentation, "Tips for Successful investing... In Software

Testing," Mr. Black shows that a key benefit of improved testing is in identifying defects

earlier: defects that are found before the product is released to the market are less

expensive to fix than are defects found after the product reaches the market. Using this

approach, Mr. Black formulates a "cost of quality" metric that varies with the share of

defects found in each stage of the product life cycle:

Cost of Quality = (number of defects found prior to release x cost of fixing defects prior to

release)

To put a value on the quality improvement offered by test automation, compare the cost

of quality using manual test methods to the cost of quality using automated methods. If

automated testing provides more rigorous and disciplined testing, it should uncover a

greater number of defects earlier in the cycle, and therefore reduce the "cost of quality"

by reducing the cost the organization must bear to fix the defects.

The value of this savings will depend on two factors: the number of defects identified

before vs. after release (automated testing uncovers a greater number of defects earlier in

the cycle), and the cost of fixing defects before vs. after release (fixing defects before

release is less expensive than fixing them after release). From this line of reasoning, it

becomes clear that the cost-of-quality payoff will be most substantial for situations in

which:

Defects are subtle or difficult to identify using manual techniques,

The cost of fixing a defect after the product has reached the market

is substantially more than fixing the same defect before the product

reaches the market.

D. Better ROI than Manual over the period of time

I did a study on the relationship between the code coverage and time for both – manual

and automated testing and find the behavior as follows:

<RoI in Automation>

Manual Testing:

Automated Testing:

100% 100%

TimeTime

Test Test CoverageCoverage

Test CyclesTest Cycles

TargetTargetReleaseRelease

Adjusted Adjusted ReleaseRelease

Actual ReleaseActual Release

<RoI in Automation>

As we can analyze both the grpah above and we can conclude that the initial time taken in

case of the automated testing is much higher than that of the manual testing but over the

period of time this time is getting averaged out. And at the end, the investment done in

automated testing is justifiable.

100%100%

Implement Implement CycleCycle

Test Test CyclesCycles

Target ReleaseTarget Release TimeTime

Test Test CoverageCoverage

<RoI in Automation>

3 ROI Computations

In this section we will discuss about the way we can calcualate the ROI. I have used the

formulae given by “Douglas Hoffman”, a famous speaker on quality methods. I have

applied these formulae to many of my projects and have got the wonderful result. I have

introduce these to you in the following paragraph.

As mentioned above there are many factors besides financial costs and benefits, the

measurable financial values are the ones used to compute returns on investments in the

four equations shown below. The first two examples provide a ratio of efficiency of

automated tests over manual tests. The second two provide a ratio of the amount of return

on each dollar invested on automation.

(1) En = Aa/Am = (Va + n*Da) / (Vm + n*Dm)

(2) En¢ = Aa/Am = (Va + n1*Da) / (Vm + n2*Dm)

(3) ROI (automation) (in time t) = (Benefits from automation) / (Costs of automation) =Ba /

Ca

(4) ROI (automation) (in time t) = D (Benefits from automation over manual) / D (Costs of

automation over manual) = DBa / DCa

Where:

En: the ratio of automated costs over manual costs for the same number of developed

tests runs the same number of times.

En¢: the ratio of automated costs over manual costs for different numbers of test runs.

Subscript a stands for automated, m stands for manual

Va: Expenditure for test specification and implementation

Vm: Expenditure for test specification

Da: Expenditure for test interpretation after automated testing

Dm: Expenditure for single, manual test execution

n: Number of automated (and manual) test executions

n1: Number of automated only test executions

n2: Number of manual test executions

N: Average number of runs for automated tests before maintenance is needed

Ba: the benefits from automated testing.

Ca: the costs of automated testing.

DBa: the incremental benefits from automated over manual testing.

<RoI in Automation>

DBa (in time t) = S (improvement in fixed costs of automated testing times (t/Useful

Life)) + S (variable costs of running manual tests n2 times during time t) – S

(Variable costs of running automated tests n1 times during time t)

DCa: the incremental costs of automated over manual testing.

DCa (in time t) = S (increased fixed costs of automated testing times (t/Useful Life)) +

S (variable costs of creating automated tests) – S (variable costs of creating manual tests)

+ S (variable costs of maintaining automated tests) times (n1/N)

Three of the four equations compute values based on comparisons between manual and

automated testing. In most situations the return from automation should be computed as

incremental return on incremental investment. The manual test numbers are the baseline

costs for the test organization – these amounts will be invested whether testing is

automated or not. Automation is additional cost with some expected additional returns,

which is where the real value of automating shows up. The third equation expresses the

ROI conceptually in terms of costs and benefits, but benefits are not computable in the

absolute sense, only in relation to some alternative.

Equation (1) compares costs of creating and running a manual test with the same number

of runs for an automated test. It computes the ratio of the cost of creating and running

automated test n times over the cost of creating and running an equivalent manual test n

time. A value less than one is advantageous for automation as it shows the cost of the

automated test as a fraction of the manual test cost. The equation can be used to compare

costs for a single test implemented both manually and automatically, but only applies to

groups of tests if there is exact correspondence between the manual and automated tests

and all tests are run the same number of times. However, it does not account for overhead

or maintenance costs associated with automation. The fixed costs described earlier that

are not associated with individual tests, are some of the costs associated with automation

that are overlooked by equation (1). Maintenance costs for the automated tests or

automation environment are also overlooked by the equation. In order to understand the

real cost and value of automation, both of these cost areas need to be considered.

The values used in equation (1) and the equation itself are based upon several

assumptions that aren’t true for most organizations, and therefore isn’t applicable for

them. Three of the assumptions are particularly important. Firstly, most often, manual and

automated tests are not run the same number of times, and every run of a test isn’t

equally valuable (since the first time is much more likely to find problems than any

subsequent time). Secondly, most organizations don’t have resources to create as many

automated tests as manual ones, so there are usually fewer new automated tests. Thirdly,

many automated tests require maintenance from time to time to keep them working at all,

so there are likely to be additional costs incurred before an automated test is run n times.

<RoI in Automation>

Equation (2) accounts for the different numbers of times for running manual test and

automated tests. It still expresses the cost of automation, as a fraction of the manual

costs, so value less than one shows automation is advantageous. This equation more

reasonably accounts for how the tests are run, but still doesn’t factor in the potential

differences in the number of tests, fixed costs, or maintenance. To represent the ratio for

sets of tests the equation assumes that all automated tests are run n1 times and all

manual tests are run n2 times.

Equation (3) shows how the ROI can be computed in a general form. It is the ratio of the

benefits from automation over their costs. A value greater than one indicates that the

benefits are greater than the costs. It shows that for every dollar invested in automation

we get ROI dollars worth of benefits. The costs to develop, run, and maintain automated

tests can be computed. However, it is very difficult to see how to compute the benefits of

test automation in an absolute sense.

Equation (4) shows a relative ROI for comparing the added benefits from automation with

the added costs from automating. Although the equation is very general, it more

reasonably represents the value of automation in relation to manual testing. It allows

selection of relevant parameters and tailoring of the components to specific organizational

situations and it includes allocation of fixed costs and benefits as well as variable costs

and benefits. The equation can be applied easily to projects and groups of tests, and with

some care it can be used to show the benefits for individual tests. This is the equation that

will be explained and used in the examples.

To use equation (4) we first need to identify all relevant costs and benefits. Values need to

be determined for the items for both automated and manual testing, since we are

comparing automated and manual testing. Some items may only apply to one or the other,

but in some of these situations we have shifted costs by changing our activities. For

example, an investment in hardware and tools may be required for automation with no

corresponding expenditures when testing manually. In another case, a tool we buy for

automating testing may provide a background load that requires additional people to be

involved in the equivalent manual test, thus shifting a variable cost in manual testing to a

fixed cost in automated testing. (The savings from manual testing are included in the

second factor in the equation for DBa, while the costs in automation are included in the

first factor in the equation for DCa.)

As mentioned earlier in the paper, selecting the cost factors is not easy, nor is assigning

values to them. The closer one looks at the numbers, the more difficulty one has deciding

what part of the costs or benefits to apply.

The intangible factors tend to confound the computations if we try to measure costs or

benefits too closely.

<RoI in Automation>

Usually we are trying to answer the question of whether we gain or loose by investing in

automation, and we don’t need tremendous granularity in the component values to figure

that out. After determining the applicable factors we can estimate approximate values

based on broad organizational views such as people-months of effort. To compute these

values we may substitute alternate computations in parts of the equations to achieve the

same results:


Life)) + S (variable costs of running manual tests during time t) –

S (variable costs of running automated tests during time t)



+ S (variable costs of maintaining automated tests during time t)

(Read S as sigma)

Generalizing or aggregating the costs is not only faster and easier, but it often leads to

more reasonable and usable results. These computations depend upon general amounts of

time and costs rather than specific measures of individual test contributions. We can more

easily identify the number of people involved in manual testing (and then allocate their

time between test design and test execution) than we can catalog all of our tests and then

measure and sum up the time spent designing and running each of them. This approach

also has the benefit of including much of the real world overhead and inefficiencies that

are overlooked when more focused measurements take place. The resulting numbers are

much more conservative and defensible, and better represent what happens.

Two simple examples are provided as case study to show the kind of return we might see

from automating some testing. The first example is for automating build tests, which is

one area of testing where there is usually a relatively quick payoff from automation. These

are run frequently and the tests are simple and relatively stable.

The second example is based on automation of GUI tests, which tends to be at the other

extreme, with more delayed and smaller payoff. (GUI testing is often where there is a

financial loss on the automation investment because maintenance costs may be much

higher and more frequent than in the example.)

<RoI in Automation>

4 Case Study

In this section, we will discuss about the live examples. I have presented the following 2 case studies which I have prepared when I was working on Device Driver testing for HP (Hewlett Packard) high end devices mainly printers.

4.1

Automation of Smoke Test examples:

It talks automation of smoke testing. When I have done the analysis, I found that this is the right case for test automation. I would like to list down the thougths which make the smoke test for the right candidate for test automation:

We used to get the nightly builds and we need to the smoke testing on each build. We do the smoke testing on the next day morning, and if something fails then

development team has to fix it, which waste our half of the day or sometime the entire day itself.

Our thougth process was to get these automated smoke test get fired whenever the development team completes the build process.

Assumptions and computational values:

· Daily builds and test runs (5 times a week)

· Manual tests take 5 days to design, 1 hour to run

· Only half the manual tests would be run on any given day (1 hour) with the other half run

the following day

· Automated tests take 15 days to design and implement, automatically run (zero cost)

· Automation is done with batch scripts and integrated into the build process, requiring

$1,000 in added hardware, with a useful life of 3 years

· Automated tests need to be maintained every 25 runs, one day of work required

· Periods of time (t) selected: 6 months (125 days) and 18 months (375 days)

· People cost $100,000 per year = $400 per day = $50 per hour


Life)) + S (variable costs of running manual tests n2 times during time t) –

S (variable costs of running automated tests n1 times during time t)

Where:

DBa: the incremental benefits from automated over manual testing

DCa: the incremental costs of automated over manual testing

DBa (in 6 months) = 0 + (1 hours * 125) – (0 * 125) hrs = ($50 * 125) = $6,250

DBa (in 18 months) = 0 + (1 hours * 375) – (0 * 375) = ($50 * 375) = $18,750


<RoI in Automation>


+ S (variable costs of maintaining automated tests) times (n1/N)

DCa (in 6 months) = ($1,000 * (6/36)) + (15 * $400) – (5 * $400) + ($400 * (125/25))

= $167 + $6,000 – $2,000 + $2,000 = $6,167

DCa (in 18 months) = ($1,000 * (18/36)) + (15 * $400) – (5 * $400) + ($400 * (375/25)) =

$500 + $6,000 – $2,000 + $6,000 = $10,500

ROIautomation (in 6 months) = $6,250 / $6,167 = 1.013 [about break even]

ROIautomation (in 18 months) = $18,750 / $10,500 = 1.786 [80% return]

The return for 6 months is just above the investment cost and there is substantial return

over 18 months. When the automated tests are run less frequently or require more

maintenance there is much lower return. The return is also helped substantially by the low

cost of creating and maintaining these tests.

<RoI in Automation>

5 Conclusion

ROI analysis must be used in context of a broader evaluation framework because it is just

one of several financial measurement tools that can be used to support an investment

decision. You may want to complement your ROI financial measures with other methods

that address the key limitations of ROI metrics. For example, ROI does not factor in risk

and does a poor job accounting for intangible rewards. Other financial methods include net

present value, scenario planning, and options theory. If used properly within the context of

a "balanced scorecard" evaluation framework that factors in non-financial decision criteria,

ROI can be an invaluable tool in your investment decisions.

ROI is particularly effective at:

Facilitating investment prioritization by making hard number comparisons between

investment options, allowing decision-makers to focus on the intangible benefits

separately.

Imposing some discipline on the part of vendors and decision-makers to support

business impact claims by taking a more methodical and quantifiable approach to

business justification.

Enforcing an understanding of the top/bottom line business impact of the

investment since it is impossible to complete an ROI analysis without

understanding the potential impact on cost and revenue generation.

<RoI in Automation>

6 References

1. Managing the Testing Process by Rex Black.

2. Software Test Automation by Mark Fewster & Dorothy Graham

3. Presentations/Write-ups by Douglas Hoffman

<RoI in Automation>

7 Author’s Profile

I am a man of few words and the most dominating word in my life is “Excellence”. In the

past, I was associated with companies like Wipro Technologies and Trianz Consulting Pvt.

Ltd. I have presented 2 white papers naming “Excellence using 6 Sigma” and “Test Project

Management” during my stay in Wipro Technologies. Other than this we developed a tool

to doing day to day activities of WHQL (Windows Hardware Quality Laboratory) testing. I

have submitted papers in international conferences organized by IIST. The initial version of

my paper “Return on Investment Issues in Test Automation” is selected for presentation in

IIST testing conference Oct-2004 held in Minneapolis.

Documents

Factors Affecting Return-On-Investment (ROI) in Automation