Getting Business Value out of (Test) Metrics

Richard Terry (UK)richard.terry@sogeti.com

Rob Baarda (NL)

What we will do / Structure

• Subjects:>Test Metric goals / stakeholders>Which metrics; some demo>Process around metrics>How to apply metrics / pittfalls>Wrap-up

• For each subject: intro, discussion• Discussion

>hand raise>hand raise with comments>open

Metric Goals / Stakeholders

• Hand raise only1. Who is using project metrics?2. Who is using test metrics?3. Who would like to use test metrics?

• Some opinions please!• Goals / Stakeholders for test

metrics? OPEN!• Your votes please

Which test metrics?

Test basis

Test object

Test Execution Defects

Repair

Production

Specifying test

cases / scripts Test cases/

scripts

Test Process

Size test basis

Size test object # defects in

test object# defects in

production

For each process:# hours effortlead time

# test cases

# = number of

# defects in test basis

# repair rounds

Deductible metrics

• Effort = Size /# hours• Productivity = # test cases / # hours• Efficiency =

# defects / (# hours or # test cases)> Specification> Test execution> Retest of repaired defects

• DDP Defect Detection Percentage (Europe)DRE Defect Removal Efficiency (USA)

• Defect injection rate for rework• Damage prevented?• More metrics:

> Availability of test environments> OPEN

Dutch test metrics experiences

• Dutch initiative to gather test metrics

• Parties involved>NESMA

Netherlands Software Metrics Association

>Testnet, including SogetiDutch Testing community

>LaQuSOLaboratory for Software QualityUniversities Eindhoven & Nijmegen

Some slides with data from the database will follow in the presentation, not in the hand-out, due to a lack of statistical evidence

Processes around metrics

• Collection in a project> Embedded in daily work> Weekly summarisation> Sanity checks> Cost: about 2% project budget

• Distribution• For a benchmark on the level of:

> Project releases> Organisation> Country> International: ISBSG

International Software Benchmarking Standards Group

To what extent?

• OPEN Discussion about>Accuracy of definitions>Number of types of defects>Is a batch test case the same as an

online test case?>Only testing of functionality or also

security, performance, usability>How to include regression testing?>Predictive value

average (mean), median, standard deviation, correlations with?Prediction model needed?

To apply test metrics

• Project>Estimating Testing New development>Estimating Testing Maintenance release>Controlling the test project

• Benchmark>Measure the effectiveness of the testing,

Improve inside / outside the test process?

• PITTFALLS?

Make a judgement based with metrics?

• Of person production/ quality of work>OPEN Discussion>Some Dutch effects

• Of Service Level Agreement>What to measure and how to judge

Wrap-up

• Business value out of metricsHand raise> Who would like to use test metrics?

• Changed opinions> Comments please (- + ,+ -)

• Reading: TMap Next> Chapter 13 Goal-Question-Metrics for test> Chapter 11 for data

• End of 2008: www.isbsg.orgParticipate? Leave your business card