Building on SuccessBeyond the obvious; Practices for “Agile Testing”

1

Erik van Veenendaal

www.erikvanveendaal.nl

Erik van VeenendaalErik van Veenendaal

� Founder and major shareholder ImproveQS� In testing since 1989 working for many different

clients and in many different roles� Author “TMap”, “The Little TMMi” and

many other books and papers

www. erikvanveenendaal.nl

Improve Quality Services BV 2

many other books and papers� Vice-President International Software Testing

Qualifications Board (ISTQB) 2005 - 2008� Vice-Chair TMMi Foundation� Keynote speaker, e.g., EuroSTAR, STAReast� Winner of the European Testing Excellence

Award (2007)

ImproveImprove QualityQuality Services BVServices BV

�Service organisation in the area ofTesting, Requirements Engineering and Quality Management

�Consultancy, Subcontracting and Training

www. improveqs.nl

Improve Quality Services BV 3

�Consultancy, Subcontracting and Training

SW Process ImprovementQuality AssuranceIT-AuditingRequirements Engineering& management (IREB)

Testing (TMap, TMMi)Test Process ImprovementCertification (ISTQB)- incl. Advanced !!Inspections / Reviews

CommonCommon TestingTesting PracticesPractices

At least 50% of the organizations:� No Test Policy / Strategy� Less usefull Test Plans� No Test Design techniques� Few apply Formal Reviews

The Answers“We do Exploratory Testing …….”� Few apply Formal Reviews

� Test Automation not successful� Re-usable testware hard to find� Testers not trained for the job�Most of us are still at (TMMi) level 1

Improve Quality Services B.V. 4

4

“We do Exploratory Testing …….”“We do Agile Testing ………..”“We do SCRUM ……………..”

“We do Lean and Mean SW development”

The ParadoxThe Paradox

� We still release our systems (most of the times) .....− ‘A little bit too late’− ‘A few too many defects’− ‘Budgets are slightly overrun’

For many yearsFor many years only only 2525% of the IT projects % of the IT projects

are are successfullsuccessfull

− ‘Budgets are slightly overrun’

� How to be ‘good enough’ and ‘survive’

� A tool box for agility in testing?!

Improve Quality Services B.V. 5

What is agile testing?What is agile testing?

� Testing in an immature (test) organzation− The great excuse !!

� Testing in an agile development project� Testing in an agile development project− Re-shaping traditional practices

� Testing following the statutes of the agile manifesto

− A different mind set to start from

Improve Quality Services B.V. 6

AgileAgile ManifestoManifesto

� Individuals & Interactions over processes and tools

� Working Software over comprehensivedocumentation

�� Customer Collaboration over contract negotiation� Responding to Change over following a plan

� Agile Values (Kent Beck)− Communication, Feedback, − Simplicity, Courage

Improve Quality Services B.V. 7

Core Practice # 1: Risk BasedCore Practice # 1: Risk Based

Setting Clear Test Priorities

CommunicationCommunicationCommunicationCommunicationCommunicationCommunicationCommunicationCommunication, , , , , , , , Customer CollaborationCustomer CollaborationCustomer CollaborationCustomer CollaborationCustomer CollaborationCustomer CollaborationCustomer CollaborationCustomer Collaboration

Improve Quality Services B.V. 8

� Risk identification and analysis� Determine a risk-based test approach� Risk based reporting (stakeholders’ language)

Testing is Risk Based - Hetzel

Working software & Responding to changeWorking software & Responding to changeWorking software & Responding to changeWorking software & Responding to changeWorking software & Responding to changeWorking software & Responding to changeWorking software & Responding to changeWorking software & Responding to change

Setting Priorities: Risk AssessmentSetting Priorities: Risk Assessment

�Risk identification− Requirements based (e.g. PRISMA)− Brainstorm workshops (e.g. FMEA)− Faliure history− Faliure history

�Risk analysis− Risk = impact x likelihood

�What is the impact for the business ?●damage, usage intensity, external visibility, ….

�What is the likelihood that there are defects ?●new technology, complexity, interfacing, ….

Improve Quality Services B.V. 9

Product Risk MatrixProduct Risk Matrix

IIIIII

H

xx

x

Must TestCould Test

X1

MoSCoW priorities

Improve Quality Services B.V. 10

IVIVIIIIIIL

M

L HM

x

x

x

Likelihood

Impact

Should Test“Won’t Test”

X2

RiskRisk--BasedBased Test Test ApproachApproach

IIIIII

HH

MM

xxx

Must TestCould TestTest design techniquesReview designSupport module testsReview module testsFull re-test – 2 iterations

Exploratory TestingSupport module testsReview module testsRe-test defect only

Improve Quality Services BV 11

IVIVIIIIII

LL

MM

LL HMM

x

x

xLikelihood

Impact

Should Test“Won’t Test” Test design techniquesReview designFull re-test

Exploratory TestingRe-test defect only

Agile “One Page Test Plan”

Must Test….. Test Approach …..

Should test…… Test Approach …..

should includea definition of “done”

Improve Quality Services B.V. 12

…… Test Approach …..

Could Test….. Test Approach …..

Would Test….. Test Approach …..A 3 hour session

to achieve a one pageMaster Test Plan

RiskRisk--BasedBased ReportingReporting ((exampleexample))

TS1 TS2 TS3 TS4 TS5 TS6 TS7 TS8

Risk item 1

Risk item 2

Improve Quality Services B.V. 13

Risk item 2

Risk item 3

Risk item 4

Risk item 5

CanCan we release the product?we release the product?

TS1 TS2 TS3 TS4 TS5 TS6 TS7 TS8

Risk item 1

Risk item 2

Management view

Improve Quality Services B.V. 14

Risk item 2

Risk item 3

Risk item 4

Risk item 5

Core Practice # 3: ReviewsCore Practice # 3: Reviews

Upstream Focused Reviews

Testing is preventing defects - Hetzel

CommunicationCommunicationCommunicationCommunicationCommunicationCommunicationCommunicationCommunication & Feedback& Feedback& Feedback& Feedback& Feedback& Feedback& Feedback& Feedback

Improve Quality Services B.V. 15

� Priority to the profitable� Apply review practices that make

the difference

CommunicationCommunicationCommunicationCommunicationCommunicationCommunicationCommunicationCommunication & Feedback& Feedback& Feedback& Feedback& Feedback& Feedback& Feedback& FeedbackCustomer collaborationCustomer collaborationCustomer collaborationCustomer collaborationCustomer collaborationCustomer collaborationCustomer collaborationCustomer collaboration

Why? and What?Why? and What?

� 60% of the defects have already been made before coding/implementation has started

� Upstream defects are the most important− for costs, but also for project success− for costs, but also for project success− requirements, architecture documents

� Focus on the high risk items − you usually can’t afford anything else− a ‘risk based’ review plan helps enormously

�Limited number, but thorough reviews

Improve Quality Services B.V. 16

Don’t fool yourself ....Don’t fool yourself ....

� Different review types for different objectives (e.g. Walkthrough / Informal Review / Inspection)

� Trained moderators

− ‘moderator du jour’ does not work− ‘moderator du jour’ does not work

� Apply roles and kick-off

� Entry check

� Not too many pages (± 20) or lines of code

� Reasonable checking rate (± 10)Improve Quality Services B.V. 17

Core Practice # 3: Unit TestingCore Practice # 3: Unit Testing

Perform Real Unit Testing

WorkingWorkingWorkingWorkingWorkingWorkingWorkingWorking software over software over software over software over software over software over software over software over

Improve Quality Services B.V. 18

� Buddy Testing (XP)� Different from debugging� Pair Inspection (or Static Analysis)

Testing requires independence - Hetzel

WorkingWorkingWorkingWorkingWorkingWorkingWorkingWorking software over software over software over software over software over software over software over software over comprehensivecomprehensivecomprehensivecomprehensivecomprehensivecomprehensivecomprehensivecomprehensive documentationdocumentationdocumentationdocumentationdocumentationdocumentationdocumentationdocumentation

Unit Test Unit Test Process (IEEE 1008)Process (IEEE 1008)

PlanCheck

coverageCheckresults

Runtests

Writetests

Unit test plan- technique- environment- exit criteria- level of independence

Will often refer to:Project Unit Test Plan

Unit test spec.- test input- expected result- test objective

May act as:automatedtest Script

Unit test log- test cases- test results- test coverage- tester- environment- etc

Improve Quality Services B.V. 19

Practical Practical ImplicationsImplications

� Buddy (independent) testing� Buy and install a coverage tool� Use formal exit criteria

Remember: most high level testing will only achieve 30 - 40% code coverage

− Risk based / Statement and Decision Coverage (be realistic!!)

� Free format test designs (code)� Developers are testing and it is fun!!� Higher quality delivered to the testers

Improve Quality Services B.V. 20

Core Practice # 4: Test DesignCore Practice # 4: Test Design

Use Informal Techniques Formally

RespondingRespondingRespondingRespondingRespondingRespondingRespondingResponding to to to to to to to to changechangechangechangechangechangechangechange, , , , , , , , CustomerCustomerCustomerCustomerCustomerCustomerCustomerCustomercollaborationcollaborationcollaborationcollaborationcollaborationcollaborationcollaborationcollaboration and No and No and No and No and No and No and No and No comphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensive doc’sdoc’sdoc’sdoc’sdoc’sdoc’sdoc’sdoc’s

21

� Test Use Cases

� Exploratory Testing

� Classification Tree Method

collaborationcollaborationcollaborationcollaborationcollaborationcollaborationcollaborationcollaboration and No and No and No and No and No and No and No and No comphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensivecomphrehensive doc’sdoc’sdoc’sdoc’sdoc’sdoc’sdoc’sdoc’s

Testing is finding defects - Myers

Improve Quality Services BV 21

Exploratory TestingExploratory Testing

� Simultaneous exploration, test design and execution� Finding the most important defects in the time available

Let yourself be Let yourself be distracteddistracted

‘cause you never ‘cause you never knowknowwhat what you’ll find ….’you’ll find ….’

newnew

PRODUCT

tests

2222

distracteddistracted what what you’ll find ….’you’ll find ….’

But periodically take But periodically take stockstockof of your status against your missionyour status against your mission

test ideastest ideas

newnew

test ideastest ideas

Improve Quality Services B.V. 22

Exploratory Testing ProcessExploratory Testing Process

Test Plan

Risk Matrix

Keep track of

bug/fix ratio !!

To be reviewed with

business & developmentCharter

Sessions

- exploration

- design

- execution

Debriefing

notes

Charter

HeuristicsList of

commondefects

Improve Quality Services B.V. 23

Are you really doing ET or just EG?Are you really doing ET or just EG?

� Are you using test charters?� Are stakeholders involved in producing

testcharters?� Is there a standard list of ideas / common defects?� Do you work in pairs?

Illustration only

� Do you work in pairs?� Is the maximum length of a test session 1 day?� Are test logs available from the test sessions?� Are new risk, ideas and issues documented?� Are test experiences discussed on a daily basis

during debriefing sessions?

Improve Quality Services B.V. 24

Core Practice # 5: PeopleCore Practice # 5: People

Build Experienced & Skilled Testers

IndividualsIndividualsIndividualsIndividualsIndividualsIndividualsIndividualsIndividuals and and and and and and and and interactionsinteractionsinteractionsinteractionsinteractionsinteractionsinteractionsinteractionsover over over over over over over over processesprocessesprocessesprocessesprocessesprocessesprocessesprocesses and toolsand toolsand toolsand toolsand toolsand toolsand toolsand tools

25

� Training (-on-the-job)

� Exchanging Experiences

� Test Team Dynamics

� Not only for Testers !!

over over over over over over over over processesprocessesprocessesprocessesprocessesprocessesprocessesprocesses and toolsand toolsand toolsand toolsand toolsand toolsand toolsand tools

Testing is an extremely complex and

intellectually challenging task - Myers

Improve Quality Services BV 25

Testing SkillsTesting Skills

Test knowledge- Test principles

- Techniques

IT knowledge- Software development

- Requirements (IREB)

via (exchanging) via (exchanging) Practical Practical ExperiencesExperiences, , CoachingCoachingand (formal) and (formal) TrainingTraining

Improve Quality Services BV 26

- Techniques

- Tools, etc.

- Requirements (IREB)

- Configuration mgt.

Domainknowledge- Business process

- User characteristics

Soft skills- Communication

- Critical mindset

- Presentation & reporting

Test Team DynamicsTest Team Dynamics

Effective teams don’t just happen!�Correct experience and skills�Correct attitude�Common purpose�Common purpose�Team management�The right team mixture, e.g. Belbin profiles

Balanced teams are more effectiveNobody is perfect but a team can be ……..Improve Quality Services B.V. 27

SummarySummary

�Focus ‘agile testing’ on …−Risk-Based Test Approach & Reporting−Upstream Focused Formal Reviews−Apply Real Unit Testing−Use Informal Techniques Formally−Build Experienced & Skilled Testers

�This will make you survive in almost any project ….

Improve Quality Services B.V. 28