Testing and Cost / Benefit

Tor Stålhane

Why cost / benefit – 1

For most “real” software systems, the number of possible inputs is large.

Thus, we can use a large amount of resources on testing and still expect to get some extra benefit from the next test.

At some point, however, we will reach a point where the cost of the next test will be larger – maybe even much lager – that the benefit that we can expect to receive.

Why cost / benefit – 2

The reason for cost / benefit analysis is the need to answer the question:

“When should we stop testing?”

In economic terms, this can be answered by

• Comparing costs and benefits for the next test

• Stop when the cost is greater than the expected benefits

Why cost / benefit – 3

The result of a cost / benefit analysis will depend strongly on which costs and benefits we choose to include.

Thus, a cost / benefit analysis can never be completely objective.

We should also remember that we compare a real cost against a estimated benefit

Which costs should be included

As a minimum we need to include costs to• Develop new tests• Run new tests• Correct newly discovered defectsIn addition we may include costs incurred by• Not being able to use the personnel for

other, more profitable activities• Dissatisfied customers – bad-will, bad PR

Which benefits should be included

As a minimum we need to include benefits from

• Finding the defects before release – lower correction costs

In addition we may include benefits from

• Better reputation in the marketplace

• More satisfied customers

• Alternative use of free personnel resources

Cost / benefit

When the costs and benefits are identified, the decision can be made based on the value of the difference:

Benefits – costs

The main challenge is how to identify the important costs and benefits.

Several alternatives

One of the important benefits of stopping testing is alternative uses of freed personnel resources – e.g. develop a new product or improve an existing product.

Thus, there can be several possible costs and benefits. They are best compared using the concept of leverage.

Leverage = (Benefit – Cost) / Cost

Hard costs and “soft” benefits

The main problem for a cost / benefits analysis of testing is that

• Many of the costs are “now” and easy to identify and enumerate

• Many of the benefits are “later” and difficult to identify and enumerate.

How to assign value to “soft” benefits

Many of the ”hard” benefits are related to saving money. However,

The company’s main goal cannot be to save money.

The main goals are about increasing e.g.

• Profit or shareholder value

• Market shares

• Market reputation

Creation of value

Improving market share or reputation, building a strong brand etc. is all about value creation. This is achieved through creativity and for this we need people.

Thus, moving people from testing and error correction to development is not about saving money but about creating new value.

“Soft” benefitsTo assign value to soft benefits we need to: • Identify important company goals and

factors (means) contributing to these goals• Map the factors onto events related to

testing• Ask the company what they would be

willing to pay to– Get an increase of a certain factor – e.g.

market share– Avoid a increase of a certain factor – e.g.

customer complaints

“Soft” benefits – example (1) Goal and means identification

• Important company goal: Increase sales

• Testing benefit: Better reputation in the market place

The important questions are:

• How much will a product with fewer defects contribute to the company’s reputation?

• How much will this increase in reputation increase sales?

“Soft” benefits – example (2)The answers to the to latest questions will

have to be answered by management.

Usually they will not be able to give you a number but they will, in most cases, be able to give you an interval.

Thus, the answer to the question “How much will this increase in reputation increase sales?” can be something like 10% to 30%, which then can be used for a benefit assessment.

Simple cost / benefit

Assess

• Cost = assessed total costs

• Benefits(low) = hard benefits + minimum soft benefits

This is a good idea if Cost < Benefits(low)

Testing and information – 1

Cost / benefits are used to decide when to stop testing.

For this decision, we need to consider two parameters:

• P(wrong) – the probability of making the wrong decisions. This will depend on the available information.

• Cost(wrong) – the cost of making the wrong decision.

Testing and information – 2Before running a test we need to consider

what more information we will have if the test case (1) fails or (2) runs OK.

Wrong approach: Make a test, run it and see what we can learn.

Right approach: What info do we need? Design and run test designed to get the needed info.

Testing and information – 3

Based on cost and probability, we can compute the risk of the decision:

Risk = P(wrong) * Cost(wrong)

The risk should be added to the costs in the cost / benefit analysis, e.g. in the leverage expression.

The value of information

Without any information, the probability of making the wrong decision will be 0.5.

We can decrease this probability by collecting more information. In our case this means running more tests.

It is, however, important to remember that running more tests also will increase our costs. Thus, the two factors risk and cost need to be considered together.

Regret As the name implies, regret is the assessed

value of something we regret we did not do. In cost / benefit analysis, it is an opportunity that we did not grab.

The question used to assess the value of the regret is:

“If you do not grab this opportunity, how much would you be willing to pay to

have it some time later?” Remember – the more we regret it, the

higher is the value

Leverage, risk and regret

We can easily include the assessed risk and regret in the leverage computation:

Total benefit = Regret + Benefit

Total cost = Risk + Cost

L = (Total benefit – Total cost) / Total cost

Advanced cost / benefitTo reduce the complexity of the problem, we

will assume that

• The cost of a wrong decision is constant

• The benefits are constant until a time T. After T, the benefits will drop to 0, e.g. because a “window of opportunity” has been closed.

• P(wrong) decreases exponentially with the money invested in information collection.

Example - 1

-250,00

-200,00

-150,00

-100,00

-50,00

0,00

50,00

100,00

150,00

200,00

250,00

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Risk

Money spent

Benefit

Total

Example – 2

-250,00

-200,00

-150,00

-100,00

-50,00

0,00

50,00

100,00

150,00

200,00

250,00

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Risk

Money spent

Benefit

Total

We see that everything done after day 7 is a waste of time and resources.After this, we spend money getting rid of a steadily smallerrisk.

Summary

• Understand the company’s goals. Saving money is probably not the most important one.

• Include all benefits. Use available company knowledge and experience.

• Use the “What would you be willing to pay” strategy to get values for soft benefits.

• More knowledge has both a benefit and a cost.