Exploratory Testing in An Agile Context Elisabeth Hendrickson

Quality Tree Software, Inc. esh@qualitytree.com

What is Exploratory Testing?

Simultaneously…

…learning about the software

…designing tests

…executing tests

using feedback from the last

test to inform the next

(Term coined by Cem Kaner. Definition refined by James Bach. Others have contributed to ET including

James Lyndsay, Jonathan Kohl, Michael Bolton.)

Exploratory Testing and Emergent Behavior

This security story passes my acceptance

tests.

The unit tests all pass! The

build is green.

Did anyone notice that a user can make themselves

an administrator?

Thinking is Mandatory

All the usual test design practices apply…

• Variable analysis

• Heuristics

• Boundary analysis

• State modeling

• Decision tables

• Personas

• Etc.

Organize the Work with Charters

A template: Explore area/feature [ with resources, conditions, or constraints ] to discover information

Examples: Explore the Calculator feature in the Fly to discover conditions or sequences that generate inaccurate results.

Explore drawing calculators with irregular, small, and too close buttons and lines to discover how the Fly behaves under these conditions.

Time Box Exploration in Sessions

Session

Session

Session

Session

Session

Session

Explore the login feature in combination with bookmarks & history to discover information about interactions.

Explore the site with various browsers and settings to discover risks related to unsupported configurations.

Explore error handling using data type attacks to find problems related to data validation.

Charters Sessions

Where Do Charters Come From?

• Stakeholder Questions “What happens if we have more than 5000 records?”

• Quality Attributes “Explore the features associated with users, groups, and security to find security holes.”

• Risks Imagine the worst thing that could possibly happen, and consider what might contribute to it happening?

• Any Other Need for Information Testing yields information. Charters can come from any information need.

Exploratory Testing on Agile Projects

Within an Iteration

Automated Unit Tests

Automated Acceptance or

Story Tests

Manual Exploratory

Testing

Define “done”, represent executable

requirements

Drive design, represent executable

specifications

Provides additional feedback

ATDD: Part of an Agile Testing Strategy

(A variation on Brian Marick’s Agile Testing Quadrants as published in his essay “Agile Testing Directions”)

Business-facing

Code- facing

Support Critique

Acceptance Tests Exploratory Testing

Unit Tests Reviews, inspections,

pairing, code quality metrics

Result: Feedback on Process AND Product

I’ll add things like that to the

acceptance criteria.

We need to think about that kind of thing in future designs too.

We’ve automated the regression test when

we fixed the bug.