Test Process Improvement; Measurement is Critical to - UKSMA

Listen | Challenge | Understand | Interpret | Create

Private amp Confidential Experimentus Ltd 17a Dorset Square London NW1 6QB T +44 (0)207 871 2300 wwwexperimentuscom

Test Process Improvement Measurement is Critical toSuccess

Presented byBrian Wells

UKSMA 22nd Conference London27-28 October 2010

This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license

Copyright copy 2010 Experimentus Ltd2

Need to Improve


Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd

Need to focus on quality

3

sect Projects need to balance delivery of quality on time and to budgetsect Historical delivery focuses on cost and timescale alone - often results in

increased costs and produces lower quality outputndash Projects invariably being over budget and behind schedulendash Evidence of increased costs in production support - the major hidden cost of

software development

sect Quality requires focus on the ldquoDo it once and do it right principlesect Focus on quality to achieve three project objectives will achieve controlled

costs controlled timescales and higher product quality

This is progress



The test process is critical

4

sect Testing demonstrates levels of quality and by finding and removing defectswill improve quality

sect Any activity that identifies issues is ldquotestingrdquo in its fullest sensendash All types of testing activities consumes approximately 50+ of the IT budget

sect If you are able to improve the quality of the test process and the product theprocess deliversndash You will reduce delivery and support costsndash You will deliver on or before timendash More chance to deliver what the client wants



Evaluate your capability

5

sect Start byndash Identifying where you are today ndash strengths and opportunitiesndash Identify where you need to go (and why)

sect TMMi model isndash The emerging global standard test process reference modelndash The ldquobest of breedrdquo taken from existing models standards and practitionersndash A model that includes the test industry recognised good practicesndash Generic ndashapplicable to all development and delivery models including industry

sectorsndash Specific ndashcovers all required activities and deliverables expected of a good quality

test process



TMMi Output

6

sect A TMMi Assessment will producendash A structured identification of strengths within the test processndash A structured definition of process improvement requirements

sect Any process improvement requirements indicatendash Where there is a poor quality process elementndash A potential for poor product quality being delivered

sect The (optional) ability to calculate potential Return on Investment of processimprovements to the business to demonstrate the value that is achievable



Approaching ProcessImprovement



What is test process improvement

8

Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle

The ISTQB glossary seems to have got its focus wrong

when it defines test process improvement as

A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at

system testing and acceptance testing





The process improvement process

9

PrioritisePlan

InitiateMeasure

DefineRedefineOperateValidateEvolve

M

RP

O

EV

Confirm the objective choose appropriate model

List the possible process improvements

Undertake the assessment

Createamend the processes

Process improvements are deployed

Benefits are realised and success criteria met

Stopstart the next stage of improvement

I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve

Methodology - IMPROVE

10

Initiate

Organisation

Process Technology

bull Action Planningbull Target Setting

bull New processdefinitionrefinement

bull Trainingbull Pilotbull Full deployment

bull Monitoring of resultsagainst targets

bull Evolution of programmeapproach amp targets

bull Assess Processmonitoragainst plan

bull Obtain agreementto progress



The business case

11

sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate

ndash Intangiblendash Tangible

sect Understand the costs involvedsect Timescalessect Quick wins



Intangible benefits

12

sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff



Financial benefits

1313

PIPID

PIP Summary EstimatedPersonDay

EstimatedCost

PotentialBenefit

Net PotentialBenefit

1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290

2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097

3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170

4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065

5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000

117000 1110622 99362295 105509150 175500

879591

TOTAL NET POTENTIAL BENEFITSConfidence Factor

Implementation FactorADJUSTED NET POTENTIAL BENEFITS

Potential Benefits and Implementation Estimates Summary



Define-redefine

14

sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring

them up to date



Operate

15

sect This is the rollout phasesect Will be at least 80 of the improvement project

sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process

sect Provide trainingsect Provide mentoring



Validate

16

sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the

start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project



Assess and developImprovements

Implementation

17

ImplementationBAU

review

Initiate ndash Measure ndashPrioritise Plan and

DefineIMP Redefine

OperateRO ValidateEvolveVE

Ent

husi

asm

MaximumHype

Actual UseDisillusionment

Practical Benefit

RIP



To Measure



How to approach measuring

19

sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach

ndash Add scales and thresholds

sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements

ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered

ndash Consistent interpretation



Working in Practice



The Organisation

21

sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to

test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels

ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between

organisation and clients

sect The strategic and organisational objective was to achieve TMMi Level 3certification



The Road Map

22

sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3

sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate

commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint

activities

sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to

be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues



Oh to measure

23

sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so

sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality

sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well

ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before



Measurement as implemented

24

sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case

sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects



Andhellip

25

sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011

ndash Test case specificationsect Efficiency had increased 8 between January and June

sect Effectiveness had increased 11 between January and June

ndash Test case executionsect Efficiency had increased 5 between January and June


ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier



And the message is

26

sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following

implemented change

It is good to measure (but make it easy to do so)


Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27

Thank you

Brian Wellsinfoexperimentuscom

+44(0)207 871 2300

wwwexperimentuscom

wwwtmmifoundationorg



Who owns TMMireg

sect The TMMi Foundation

ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights

ndash Creation and management of an independent central data repository and provisionof industry and other analysis services

ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources

ndash Provision of a public forum of interested parties

sect For more information visit wwwtmmifoundationorg

28



TMMireg Foundation Contacts

29

sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg

sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg

sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg



Need to Improve




3






This is progress




4







5




test process



TMMi Output

6










8











9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







3






This is progress




4







5




test process



TMMi Output

6










8











9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







4







5




test process



TMMi Output

6










8











9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







5




test process



TMMi Output

6










8











9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






TMMi Output

6










8











9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29










8











9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







8











9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







9

PrioritisePlan

InitiateMeasure


M

RP

O

EV








I



Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Measure

Prioritiseamp Plan

Define Re-define

Operate

Validate

Evolve


10

Initiate

Organisation

Process Technology










The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






The business case

11






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Intangible benefits

12




Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Financial benefits

1313

PIPID


EstimatedCost

PotentialBenefit







117000 1110622 99362295 105509150 175500

879591






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Define-redefine

14


them up to date



Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Operate

15






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Validate

16






Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







Implementation

17

ImplementationBAU

review


DefineIMP Redefine


Ent

husi

asm

MaximumHype


Practical Benefit

RIP



To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






To Measure




19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







19








Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Working in Practice



The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






The Organisation

21








The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






The Road Map

22




activities





Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Oh to measure

23








24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29







24





Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Andhellip

25









And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






And the message is

26


implemented change




Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Thank you


+44(0)207 871 2300

wwwexperimentuscom




Who owns TMMireg







28




29






Who owns TMMireg







28




29







29





Documents

Test Process Improvement; Measurement is Critical to - UKSMA