Exploratory testing by Rik Marselis

Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016

© Sogeti 2016 1

Exploring software;

Combine experience,

structure & techniques


© Sogeti 2016 2

|

Todays presentation

Our flavour of Exploratory Testing

A little theory

Some exercises and dynamic testing

Exploratory Testing © Sogeti 2016 2

This little fellow is R0B3.

He will assist us in this

session.


© Sogeti 2016 3

Our flavour of exploratory testing.


© Sogeti 2016 4

|

E.T. sources:


Cem Kaner first used “Exploratory Testing” in 1984

Good books:

“Explore it!” by Elisabeth Hendrickson

“Exploratory Software Testing” by James Whittaker

Website:

www.TMap.net

And many more sources…

http://www.tmap.net/


© Sogeti 2016 5

|

E.T. has many flavours, this is our flavour:


• Focus on confidence (risk-based)

• Structured (charter, log, debriefing)

• Session-based & Timeboxed (not too short, not too long)

• Tandem-approach (two people)

• Combine experience-based and coverage-based testing

• Simultaneous test design, test execution and learning

• Flexible (fit for Agile and DevOps)

• Prepared (test ideas, testing tours)

• Tools (heuristics, checklists, test design tooling)

• Serious Fun !!


© Sogeti 2016 6

Focus on confidence

Confidence


© Sogeti 2016 7

|

Testing = Minimizing difference between expected and delivered


Expected Delivered

Business value achieved

Business value missed

Product risks

materialized and

money wasted


© Sogeti 2016 8

|

Definition of testing


TMap®(Test Management approach):

Testing is a process

that provides insight into,

and advice on,

quality and the related risks


© Sogeti 2016 9

|

Testing contributes to gaining CONFIDENCE

© Sogeti 2016 9

Quality

Risks

ConfidenceQuality attributes

Analysis and mitigation


© Sogeti 2016 10

|

Definition of exploratory testing


Elisabeth Hendrickson:

Exploratory Testing is

simultaneously designing and executing tests to learn about the system,

using your insights from the last experiment

to inform the next.


© Sogeti 2016 11

|

What is the main deliverable of testing?


Go

No Go

ClientTester


© Sogeti 2016 12

|

How do you get the test report?

Investigate the situation

Prepare a test (input & expected result)

Execute a test (compare the actual result)

Use the outcome to report

Testing à la façon TMap © Sogeti 2016 12

Use a testing method


© Sogeti 2016 13

|

Purpose of exploratory testing


Fast Feedback LearningRevealing the

unexpectedGaining confidence


© Sogeti 2016 14

Tandem-approach.


© Sogeti 2016 15

|15

How testing is often perceived

Two IT people are cycling up a hill on a tandem-bike. When they finally reach the summit the one in front, the developer, says: ‘It was hard but we made it!’ Then the one at the back, the tester, comments: ‘Yes and I hope you appreciate that I was using the brakes all the time because otherwise we would have rolled down the hill backwards’

Make sure you are a tester that helps cycling up the hill !!


© Sogeti 2016 16

|

Tandem approach: 2 people


Why work in pairs?• Two people know more than one

• One subject-matter expert and one testing-expert

• One does the testing, the other the logging

Alternative: one expert and one trainee

Beware:

not suited for very dominant or very introvert people


© Sogeti 2016 17

|

Why 2 people see more than 1



© Sogeti 2016 18

|

Why 2 people see more than 1



© Sogeti 2016 19

Structured (charter, log, debriefing)


© Sogeti 2016 20

|

Information /

Conclusion /

AdviceSession log

Organize with charters: 3 major activities


Create

charters

Execute

charterDebriefing

Backlog of

charters

Session log

Session log

Information /

Conclusion /

Advice

Information /

Conclusion /

Advice


© Sogeti 2016 21

|

Our pragmatic simple charter template


CharterLog

(notes)

De-

briefing

Test ideas: e.g.

testing tours,

test data, etc.


© Sogeti 2016 22

|

Determine test units, assign test charters


Someone organizes exploratory testing upfront by creating charters(e.g. a test manager, scrum master, team member, etc.)

Divide the test object in logical test units

Create one or more charters per test unit

This may involve any kind of risk analysis (e.g. risk poker)

Create a few extra “empty” charters that can be assigned based on

conclusion and advice in the debriefing.


© Sogeti 2016 23

Prepared (testideas, testing tours)


© Sogeti 2016 24

|

Test ideas

Inspired by: Elisabeth Hendrickson Exploratory Testing © Sogeti 2016 24

A test idea is

any useful thought, piece of data, technique,

heuristic or whatever that you write down on your charter

so that during your test session you have an

abundance of possibilities to vary your testing.

Make sure you also think of non-functional quality characteristics!


© Sogeti 2016 25

|

Testing tours


Landmarks of Amsterdam

Rijksmuseum

Dam square

Red lights

Van Gogh museum


© Sogeti 2016 26

|

Testing tours


James Whittaker distinguishes many different possible testing tours.

Examples:

The landmark tour: think of all the “landmarks” that you must have

seen during your exploratory session, while travelling between these

landmarks you explore the rest of your test object.

The supermodel tour: Only the outside matters, focus testing on

the look-and-feel.

The intellectual tour: Prepare “hard questions” for the test object,

e.g. if you must open a file, find the hardest file to open.

(for more tours read: “Exploratory Software Testing” by James Whittaker)


© Sogeti 2016 27

|

Exercise: test ideas for line-following robot


R0B3 is a robot with a light sensor.

It can follow a line

It will stop if it detects a red color

What test ideas come to your mind?


© Sogeti 2016 28

Session-based & Timeboxed.


© Sogeti 2016 29

|

Running the exploratory testing session


No distractions

No interruptions

Timebox:

½ hour – 3 hours

(as specified in charter)Charter


© Sogeti 2016 30

|

Structure: session based & chartered


Keep the scope of your charter in mind, but keep some room to follow your

insights to explore somewhat beyond the scope to learn more.

Room for wandering !!


© Sogeti 2016 31

|

Test log


Inputs Actions Expected output Actual outputObservations,Defects, etc.


© Sogeti 2016 32

|

Possible debriefing outcomes


DebriefingNot able to

execute test

New risks

and/or test

ideas

Return

charter to

backlog

New

charter on

backlog

Information /

Conclusion /

Advice

Information /

Conclusion /

Advice


© Sogeti 2016 33

Combine Experience & Coverage.


© Sogeti 2016 34

|

Always combine both approaches

combine

Exploratory Testing © Sogeti 2016


© Sogeti 2016 35

|

Four groups of Coverage based testing

Better know & use

one technique

of each group

than

all techniques

of only one group



© Sogeti 2016 36

|

Experience based approaches with TMap HD

Largely intuitive & ad hoc

Structured, chartered

Experience brought together



© Sogeti 2016 37

|

Test the triangle program

What would you want to test?

Why?

What is your expected result?

What is the actual result?

What is your conclusion?

© Sogeti 2016 37


© Sogeti 2016 38

|

Expected outcome is ESSENTIAL


Be sure FIRST to think of your expected outcome

before you execute a test.

1 + 1 = ???

Expected: 2

Actual:


© Sogeti 2016 39

Simultanuous

test design,

test execution,

and learning.


© Sogeti 2016 40

|

Why combine structure and flexibility?


Exercise

The trainer has an object in mind (can be anything in the universe)

The participants ask a yes/no questionThe trainer answers (yes or no only)

If a participant knows the object shout it out!Lee Copeland

http://www.google.nl/url?url=http://www.sqetraining.com/sme-profiles/lee-copeland&rct=j&frm=1&q=&esrc=s&sa=U&ei=woLQVNT9EILlUtrogng&ved=0CBYQ9QEwAA&usg=AFQjCNHTJfayG47m7jxbpV_G1vlgoZT6hg

http://www.google.nl/url?url=http://www.sqetraining.com/sme-profiles/lee-copeland&rct=j&frm=1&q=&esrc=s&sa=U&ei=woLQVNT9EILlUtrogng&ved=0CBYQ9QEwAA&usg=AFQjCNHTJfayG47m7jxbpV_G1vlgoZT6hg


© Sogeti 2016 41

|

Exciting…

© Sogeti 2016 41


© Sogeti 2016 42

|© Sogeti 2016 42


© Sogeti 2016 43

Flexible (fit for Agile and DevOps)


© Sogeti 2016 44

|

Exploration is a feedback loop


TEST

LEARN


© Sogeti 2016 45

|

Early testing


Also known as

“Reviewing”Also known as

“Testing”

Start testing as early as

possible.

Combine static and

dynamic testing

whenever possible.


© Sogeti 2016 46

Tools (heuristics, checklists,

test design tooling, test

execution tooling, etc.)


© Sogeti 2016 47

|

Use heuristics to get test ideas


A heuristic is a rule of thumb, simplification, or educated guess

Numbers:

- Zero

- One

- Many

- Too many

- Too few

Selecting:

- None

- One

- Some

- Many

- All

Timing:

- Always

- Sometimes

- Never

CRUD:

- Create

- Read

- Update

- Delete


© Sogeti 2016 48

|

Test design techniques are tools !


Use coverage based techniques as “test ideas”(thus your start-set of test cases can be made long before …)


© Sogeti 2016 49

|

Tool: nightmare headline game



© Sogeti 2016 50

Serious fun !!


© Sogeti 2016 51

|

Bug hunt


Get a charter

Analyse

Pair testing

one hour

Judgeevalutedefects

Score sheet

update

The “Friday afternoon-sport”:

competition in short iterations

to determine

which team finds

the most (real!) bugs

(of course there’s a prize!!)


© Sogeti 2016 52

|

Conclusion

source: TMap HD, Niki van Dreumel, ASR Exploratory Testing © Sogeti 2016 52

In the hands of experts,

Exploratory Testing is a structured way

to measure the quality of the information system

in a relatively short time

and to reach an objective view on the risks involved.


© Sogeti 2016 53

Happy

exploration !


© Sogeti 2016 54

Questions ?

Answers !Discussion