Why Do We Bother With Test Strategies

8/9/2019 Why Do We Bother With Test Strategies

1/34

Listen | Challenge | Understand | Interpret | Create

Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com

Why do we bother with Test Strategies?

Presented by:

Geoff ThompsonSIGiST9th December 2008


2/34

Agenda

Drivers for change?

What had to change?

What we did first

The problem

How we made it work

Experimentus Ltd 2008

2


3/34

Drivers for change


3


4/34

4


5/34

5


The Business was demanding a higher

standard to remain competitive

The Business wanted clearer

identification of Risk to quality and

Risk Mitigation

The IT department was under pressure

to do more for less

There was a continual drive to

increase in productivity

- Cost Reduction

- Risk Management

- Improving Quality

- Revenue Enhancement


6/34

6 Experimentus Ltd 2008

So they were looking for change to avoid the

wrong solution going live late!


7/34

They wanted software that just worked!



8/34


9/34

How did we find out?

We reviewed many assessment methods We looked at TPI and TMM

Selected TMM because:

It was a research project developed by Illinois Inst. of Technology

It describes the processes, which ensure a well planned and controlled test

projectHas very clear 5 levels of test maturity

Is based on the evolutionary testing model

Has been empirically validated

Inspired by the CMM (complementary)

9



10/34

TMM


The 5 levels of TMMi


11/34

11

How does it work?


Maturitylevel

Indicates Testing Processcapability

ContainsKey process

areas

Achieves (Sub) Goals ContainsKey practices

Describe Implementation


12/34

For more information see

www.tmmifoundation.org

12



13/34

The recommended improvements

Needed a Test Policy Needed a Testing Strategy/Framework

Needed a test process definition

Needed a defined Risk Based Approach to Testing

Needed a career path for testers

Needed to complete the roll out of TestDirector

Needed to implement reviews and static analysis

13

So what happened next?



14/34

What we did first



15/34

TESTING POLICY TESTING DEFINITIONRisk Based Testing is the process by which we explore and understand the status of the business benefits and the business risks associated with a release of software or of new or upgraded hardware, by:-1. Making a prioritised list of risks,2. Performing tests that explore each risk,3. As risks evaporate and new ones emerge, adjust the test effort to stay focused on the current crop.

TESTING SYSTEMDevelopment and execution of a test plan in accordance with The Legal & General Test Strategy:-

i. Testing will be organised in line with the Testing V model,ii. The different levels (phases) of testing will be clearly defined and understood,iii. Testing will be an integral part of the Project Lifecycle,iv. Formal test techniques will be used at all levels (phases) of testing, v. Appropriate training in testing will be provided,vi. Defined roles and responsibilities and Career paths will be available, vii. Appropriate Entry and Exit criteria will be defined, communicated and enforced for all test levels (phases), viii. The Test environment requirements,ix. Test tools will be used where appropriate,x. All testware will be maintained under Configuration Management, to enable audit and reuse,xi. Metrics and measures will be recorded and used to enable test improvement and better estimating for test activity, xi As well as the process for Incident Management , the definition of fault priority and severity rating will be defined.

MEASUREMENT OF TESTING EFFECTIVENESSTest Effectiveness will be measured by comparing the volume of Priority 1, 2 and 3 faults found in the first 6 months of live runningversus the volume of priority 1, 2 and 3 faults found during testing - DEFAULT DETECTION RATE (DDR).MEASUREMENT OF TEST EFFICIENCYTest Efficiency will be measured by determining the total cost of testing (both Business and BIS time both planning and executionbut excluding fix costs) as a ratio of the total design and build costs (including Project Management) TEST EFFICIENCY RATE(TER).

TESTING STANDARDSDDR of less than XXXXTER of XXXX or less TEST PROCESS IMPROVEMENTQuality reviews will take place at key stages in the test projects lifecycle. Specific testing focused Post Project reviews will be held after every release of software or of new or upgraded hardware.

------------------------------------------- ---- -------------------------------------------We agree to the general principals outlined above, and changes or additions made will be subject to change control.

Manager )________________

Dated 1st


15


16/34

Test Tools

Reviews

GenericTest Process

Measures andMetrics

HR

Implementationand

Communications

Risk BasedTesting

CorporateTest Strategy


16


17/34

CorporateTest

Strategy

GenericTest

Process

Test

Tools HRReviews

Implementationsteps

Implementationsteps

Implementationsteps

Implementationsteps

Pros- Quicker delivery- Early buy-in

- Earlier benefit realisation

Cons- Rework- Harder to manage Strategic vision- Delivery costs are larger

- Loss of confidence due to rework- Support costs are higher

Delivery choices - Vertical

Implementationsteps


17


18/34

CorporateTestStrategy

GenericTestProcess

TestTools HR

Reviews

Implementation steps



Cons- Slower delivery- Needs commitment and trust

Pros- Minimal rework- Clear strategic view- Clear understanding of impacts between

deliveries- Greater acceptance as full picture understood- Quick wins giving early buy-in

Delivery choices - Horizontal


18


19/34

The problem



20/34

Up to this point we had tremendous support Particularly from the Project and Programme Management Community

BUT

The process and templates just didn't get used to provide the greatest benefit sowere in danger of being bypassed by everyone

WHY?



21/34

There were two problems in theimplementation of the process improvements:

Testers

Project and Programme Managers



22/34

The Testers

Couldnt see past their noses!

22



23/34

They couldn't plan!

Common issues

Requirements

Estimating

Activity plans

Resource requests

Budgeting

Planning was traditionally successful but not effective, when undertakentwo weeks before test execution was due to start, when everything wasknown!



24/34

Lessons learnt

Test resources were traditionally put in place long after the project has started,

and often just before test execution starts when...All requirements are defined and almost built

Budget has been given rather than asked for

Resources are set

Timescales are set, and probably squeezed

Basically everything is in place to enable accurate plans to be drawn up!

Development Managers provided mentoring to Test Managers using theirexperience of planning when all that was available was a high level requirement

Test Managers were taught

High level estimating

Early Project Planning



25/34

The Project and Programme Managers

Couldn't understand the deliverables from testing

Test Strategy?

Test Plans?

They knew they needed a Test Strategy but didn't know why we always have

one

They were not clear when each deliverable should be produced

They still didn't invite testing in at the start of a project



26/34

Lessons learnt

Align Test deliverables with Project Deliverables

We reintroduced the section for an overview of testing that had been removedfrom the Prince2 template Quality Plan

Test Policy remained Test Strategy abolished and replaced with a documentedTest Process, and a Master Test Plan

Where appropriate rename Test Deliverables to match Project Deliverable

So lets see what we did to achieve this



27/34

Early Test Planning the inputs and outputs

Customers Quality

Expectations

AcceptanceCriteria

Quality Approachto Project

Deliverables

Quality PlanSummary

ApprovedTesting Budget

Project Plan

Project Controls

Project RisksInputstoTestPlanning

Testing Budget

Test Stages forProject Plan

Project Controls

Testing Risks

Original idea

and estimateproduced

High LevelTerms ofReference

Project

Initiation

Document

Quality Plan

ApprovedBusinessCase

Business

Requirements

Testing Estimate

InputfromT

esting

Test Policy

Test Managers

Roles andResponsibilities

EstimatingMetrics

27



28/34

Master Test Planning - the inputs and outputs

MTP or PID

PID

Master TestPlan

AcceptanceTest Work

Package

Team Plan

OperationalAcceptanceTest WorkPackage

SystemIntegrationTest WorkPackage

System TestWork Package

ComponentIntegrationTestWork Package

Component

Test Work

Package

Team Plan

Team Plan

Team Plan

Team Plan

Team Plan

Risk BasedTesting

Assessment

Customers Quality

Expectations

AcceptanceCriteria

Quality Approachto Project

Deliverables

Quality PlanSummary

Approved

Testing Budget

Project Plan

Project Controls

Project RisksInputstoTestPlan

ning

28


Test Framework


29/34

Testing Deliverable Breakdown


Master Test Plan

Component Test /Component

Integration Test

Workpackage

Plan

System

Integration Test

System Test /Business

Acceptance Test

Operational

Acceptance Test

Co-Existence

Test

Production

Acceptance Test

Test

Specification

Tested/Integrated

Components

Actual Results

Workpackage

Plan

Test

Specification

Integrated

Systems

Actual Results

Test Closure

Report

Test Incidents

Workpackage

Plan

Test

Specification

Tested Solution

Actual Results

Test Closure

Report

Daily Incident

Report

Daily Progress

Report

Test Incidents

Team Plan

Test

Specification

Operationally

Tested Solution

Actual Results

Test Closure

Report

Daily Incident

Report

Daily Progress

Report

Test Incidents

Team Plan

Test

Specification

Co-Existence

Tested Solution

Actual Results

Test Closure

Report

Daily Incident

Report

Daily Progress

Report

Test Incidents

Team Plan

Test

Specification

Tested

Implementation

Actual Results

Test Closure

Report

Test Incidents

Handover Report

Weekly Test

Specification

Status Report


30/34

Learning styles

Having a process isn't the full answer though!

Humans will initially learn the process verbatim, and will expect significant helpto be able to adapt the processes to their environment

We first have to learn, before we can be agile with our use of the process

This takes time, which is why the process improvement experts suggest an 18month timeline before process and culture come together to be at their most

efficient

30



31/34

Reflective

ObservationUnderstanding

ConcreteExperience

Following the rules

Active

ExperimentationDoing

AbstractConceptualisation

Thinking

Learning styles

This is where no realevaluation of whether the

process is right or notoccurs

At this point theprocess usage is

second nature, withall users being ableto evaluate what is

right and wrong.

31



32/34

No matter how adaptable to change people are, there will always be cultural

issues which if not managed will inhibit the benefits of the change being realised

Remember...


That to make any process change work, agility in its delivery is the keyits no good enforcing change that no one can understand or is able to use

En

thusiasm Maximum

Hype

InitialInvention

ActualUse

Dis -illusion-

ment

PracticalBenefit

RIP

32


33/34

While computers are not likely to make mistakes, people are. And people control

the computers.

Fred Joseph Gruenberger, 1975

Computers programmers are among the great innovators of our time. Unhappily,

amongst their most enduring accomplishments are several new techniques forwasting time.

Robert Jourdain, 1986

To the future but remember....


33


34/34

Thank You

Any questions?


34

Documents

Why Do We Bother With Test Strategies