What Is the Value of Testing and How Can We Increase It? Paul Gerrard Technical Director

©2003 Systeme Evolutif Ltd

Slide 1we’ll change the way you think about testing

What Is the Value of Testingand How Can We Increase

It?

Paul GerrardTechnical DirectorSysteme Evolutif

Systeme Evolutif Limited9 Cavendish PlaceLondon W1G 0QD, UKTel: +44 (0)20 7636 6060email: [email protected]://www.evolutif.co.uk/



Themes

“[A cynic is] A man who knows the price of everything but the value of nothing”

Oscar Wilde (1854-1900), Lady Windermere’s Fan.

“To win without risk is to triumph without glory”

Pierre Corneille (1606-84), The Cid.



Agenda Let’s play Marketing Quantity, Quality, Process or Product –

Where is the Value? The Most Valuable Product of Testing is

Project Intelligence Influences on the Value of Testing Increasing the Value of Testing Process “Improvements” and the Value of

Testing Summary.



Let’s Play Marketing We need to consider:

– Who are the customers of testing?– What exactly does the customer value?– How can we increase the value of our testing?– How can we increase their appreciation of our

testing?– How do we select an “improvement” to make?– How do we judge the value of an

“improvement”?



Quantity, Quality, Process or Product – Where is the

Value?



Mine clearing According to captured records, there

are 100 mines in that minefield We have identified and removed

seven mines I crossed it once but don’t remember

how

Is it safe?



Test activity and information

I ran some tests– But I can’t remember what happened or how I could

repeat those tests– Was this testing valuable?

I ran a hundred tests and the system failed 23 times– I can’t remember how I did it or where the system

failed– Was this testing valuable?

The ACTIVITY of testing only has VALUE if it generates INFORMATION (this is the deliverable).



How much detail? Which is best?

– “We ran lots of tests”– “We found lots of faults”– “10% of our tests caused the system to fail”– “This test case caused a failure”– “This set of test steps caused the system to fail at

precisely this point, in precisely this way…” Test records should be detailed enough to

plan, design, implement, execute tests and interpret test results and reproduce failure– AND NO MORE.



Quantity or quality? Which would you pay more for?

– 100 page test plan or 200 page test plan?– 100 incident reports or 200 incident reports?– 100 requirements faults or 100 coding faults?– 100 random tests or 100 ‘designed’ tests?

Quantity is one thing, but the nature and quality of the information and the way it was prepared makes a huge difference to its value.



Testing generates “project intelligence”

If you don’t know where you are, a map won’t help If you don’t know the status of the deliverables,

the best plan won’t help you manage your project A project is like driving off-road in the dark

– Testing provides the headlights for the journey Testing provides data on the status of deliverables

and generates “PROJECT INTELLIGENCE” Stakeholders and management need intelligence

to make decisions, command and control the project.



Does testing ‘change’ anything?

Product TestingProduct

(unchanged)

ValuableStuff

Testing doesn’t change the product in any way. We do it to get the ‘valuable stuff’.



What is the valuable stuff? Test deliverables (infrastructure,

test plans, specifications, scripts, data, results etc…)

TEST RECORDS – test documentation (but do only testers read it?)

PROJECT INTELLIGENCE – rarely documented as such though.



Military intelligence?



The Most Valuable Product of Testing is Project

Intelligence



Documentation PRODUCTS

– Test Strategy– Test plans, designs, specifications– Procedures/scripts, expected results– Actual results– Incident reports– Log– End of Phase Report– Etc.

TANGIBLE, BUT LESS VALUABLE.



Test infrastructure PRODUCTS

– Test process– Configured test hardware– Test databases– Tools– Manual and automated tests– Skilled resources– Etc…

TANGIBLE, USUALLY AN OVERHEAD– (unless these are required deliverables).



TEST RECORDS DATA

– How the product SHOULD work (test specs)– How the product ACTUALLY works (test results)– Where the product is STABLE (incident trends)– Where the product is UNSTABLE (incident trends)– What they have COVERED in the product (test log)– What they have NOT COVERED (test plan – test log)– How (and where) the product FAILS (incidents)

TANGIBLE AND VALUABLE, BUT…– (valuable to testers, developers, NOT management).



PROJECT INTELLIGENCE KNOWLEDGE OF PROJECT RISKS/BENEFITS

– RISKS» Requirements are poor and/or unstable» Less of the product is ready, product is not as good as

planned» They should have been less ambitious, done it differently» The project will slip

– BENEFITS» Some features/benefits are available» Other features/benefits are blocked by risk uncertainty

INTANGIBLE AND VALUABLE– Crucial to management and stakeholders– Intelligence often arrives TOO LATE TO BE OF USE.



Where the value of testing lies

LOW VALUE– Documentation– Test infrastructure** UNLESS

these are specific objectives or deliverables of project

HIGH VALUE– Test Records– Project Intelligence

We can increase the value of testing MOST by:• Improved INTELLIGENCE• Providing INTELLIGENCE EARLIER.



Influences on the Value of Testing



Intangible value, tangible cost

Testers think testing is good and has undeniable value, but we are not independent!

Not everyone thinks that (software suppliers, in-house developers?)

If you are not a tester, the value seems intangible– But the cost IS tangible– Exercise, flossing, house maintenance, life insurance,

testing can all be put off till another day– If you don’t do it now you store up problems for later

A bit like a lifestyle choice – it’s human nature to put things off.



Key factors in the value of testing

Adding value depends on how well we achieve and communicate:1. Coverage2. Incidents3. Risk-based testing4. Benefits-based testing

Time to deliver that value depends on the test process (more later)

Willingness of the organisation to accept and act upon project intelligence is key (more later).



Coverage Breadth of coverage

– Are all aspects of the product tested? Depth of coverage

– Are critical aspects of the product tested more thoroughly?

Can the measure be applied systematically? Is the measure relevant to the faults to be

detected or the risks of concern? Can we communicate coverage in a

meaningful, understandable way?



Incidents Are the failures of most concern

being found and logged as incidents? Can these failures be reproduced

easily? Are the incidents being addressed in

the right order? Are the incidents of most concern

being resolved quickly?



Risk-based testing Are product risks identified early in the project? Are stakeholders involved in the decision to rank

these risks? Is the decision to include risks in scope for

testing done by consensus? Is the decision to de-scope risks done by

consensus? Is test design driven by the need to address risk? Is test progress reporting based on risks

addressed?



Benefit-based testing Does the project have cardinal

objectives (benefits) defined? Are all risks related to the benefits? Are tests related to the risks that

block the benefits? Is test progress reporting based on

the benefits that are threatened or available?



Increasing the Value of Testing



Contribution of the test process

Does testing generating valuable intelligence? Does testing generate intelligence early

enough? Is testing flexible enough to react to new risks? Are the test communication channels clear? Improvement types:

– Effectiveness– Efficiency– Perceived value of testing.



Effectiveness improvements Improvements to “intellectual” test

activities enhance the quality of intelligence Implementing these activities earlier make

intelligence available earlier Examples:

– Formal test design improves coverage, and improves the quality of test records

– Improvements that promote earlier test activities tend to make the value available earlier (e.g. more reviews).



Efficiency improvements These reduce the cost of doing testing Improvements to the mechanical or clerical

aspects of our test process make us more efficient (but don't affect the value of testing)

For example– Using a tool to automate tests might make it

cheaper to do a task, but does it add value?– It might save time, allowing us to do other useful

activities Ultimately, effectiveness improvements are

more valuable.



Organisational culture Willingness to accept and act on the

intelligence that testers generate is key

Primarily a cultural issue If the intelligence isn’t valued by the

culture - it doesn't matter what value it has

We need to EDUCATE our customers.



Increasing the perceived value of testing

No such thing as “absolute value” of testing We must work on the PERCEIVED VALUE to the

customers of our testing:1. Improve the value in line with their

expectations– Provide richer, deeper, focused, relevant information– “Create more complex, subtle, interesting wine”

2. Increase their appreciation of what we do– Educate them to understand testing products– “Train their palate to detect the complex flavours”

Improvements must address both issues.



Process “Improvements” and the Value of Testing



Test process or test product assessment?

Assessments of the “Test Process” may miss the point

“Process healthchecks” are useless because they don’t focus on the VALUE of test deliverables

Assessment of the perceived value of Test Products must be a key driver

We must focus on process changes that:– add value to the test products– enhance the perceived value of test products– make test products available earlier.



The challenges of testing improvement

Barriers are organisational, personal, not technical:– changing management perceptions so they support

testing and improvement– overcoming management and practitioner

resistance to change– design and implementation of workable processes

and management controls People, not tools, implement test strategies Change is easier if people see VALUE being

added.



Problems or symptoms? Need to separate problems from symptoms:

– “Management doesn't understand the value of testing”

– “The cost of testing is high but difficult to pin down”– “Developers, testers, users never trained”– “The quality of the product delivered into testing is

poor, so takes longer to system test”– “The perceived value of testing is low”

Need to understand symptoms to identify the source of problem and fix the problem

Improvements often tackle the symptoms only.



Right improvement mix is key

A mix of improvements is most likely to be required:– management awareness– tester training– developer training– improved definition of the test stages and

their objectives– measurement of the quality of the product at

each stage– etc. etc.



Constraints and expectations

Not all improvements a good idea straight away– some improvements are expensive– some save time, but bring dramatic change– some improve the quality of the testing, but take

longer to implement– some tidy up ‘loose ends’ but have negligible value

Very few improvements save time, improve quality, cause minimal change and pay back after two weeks

Need to set/agree clear management expectations.



Do tools increase the value of testing?

With tool vendors present, I’d better be careful!

Need to distinguish between– Quantity, quality and the information provided

A tool might allow you do to testing faster, but it probably won’t increase the value of tests

It might make a task interesting, so a valuable task gets done

It might give you more time to do other valuable tasks.



To Summarise…



How to increase the value of testing

The value of testing lies in the project intelligence it generates

Produce documentation with purpose:– To allow testers to do their job (THE MEANS)– To communicate intelligence (THE END)

Process ‘improvements’ selected to deliver better intelligence, earlier, faster and cheaper

You need to train your customers to appreciate the project intelligence that testers produce.



Themes revisited“A poor tester knows the challenge and cost of testing but not its value”

– Don’t be cynical – testing isn’t just a costly activity to be gotten through, testers should aim to deliver value

“To win without risk is to triumph without glory”– Risks are inevitable - the testers’ role isn’t to avoid

them, but inform management and reduce their uncertainty

– Satisfaction (and recognition) comes from overcoming risks, not doing the mundane tasks of development

– Make your management aware of software product risks and report progress towards their goal, not just incidents.

Testers deal in success, not just failure.



www.evolutif.co.ukwww.riskbasedtesting.

com

What Is the Value of Testing

and How Can We Increase It?

Paul Gerrard

Documents

What Is the Value of Testing and How Can We Increase It? Paul Gerrard Technical Director