Exploratory testing – Black or white? - LTH · Exploratory testing – Black or white? Using...

Exploratory testing – Black or white?Using Focus Groups to Enhance Collaboration

ELIZABETH BJARNASON LU, PETER VISURI SONY

KAI PETERSEN BTH, PER RUNESON LTH

The Reality of Testing Today

§Huge Legacy§More and more complex product§ Less people working on it§ Increase quality focus§Need a quicker way towards good quality

(Exploratory…?)

ScopeScope

CostCostTimeTime

QualityQuality

Scope

CostTime

Quality

What is ET?

Exploratory software testing (ET) is astyle of software testing that

Emphasizes the personal freedom andresponsibility of the individual tester

to continually optimize the value of her workby treating test-related learning, test design, testexecution, and test result interpretation as mutuallysupportive activities that

run in parallel throughout the project.

Variations of Exploratory Testing

Freestyle Pure scripted

Variations of Exploratory Testing

Freestyle Pure scripted

Test objectonly

Test object,test steps,test data

Variations of Exploratory Testing

Freestyle Pure scripted

Test objectonly

Test object,test steps,test data

Test goals,constraints

Actionable Exploratory Testing

Workshop agenda• Introduction (10 min): research context, team &

participants• The principles of exploratory testing (5 min)• Alternative types of test charters (20 min)• Exercise: Write test cases according to test charter

templates (15 + 25 min)• Reflect on improvements (10 min)• Closing (5 min): Sum up; next steps

Findings: Plus & Minuses

+ Trace of coverage - Initial cost of tc creation- → large test suites

+ Rapid feedback

+ Less admin/ barriers to testing

+ Resilient to changes

+ Changes managed informally

+ Supports learning

- Easy to miss specific requirements

- Requires knowledge

- Changes managed informally

Outcome – guidance for future

Test Priority1. Most important use cases (customer/legal)2. NFR’s (Stability, Performance, Power…)3. ET to cover most other areas

Good to consider for ET- mixed experienced testers- mix different levels and degrees of exploration when

designing/executing tests

Further contacts

Elizabeth Bjarnason Peter Visurielizabeth.bjarnason@cs.lth.se peter.visuri@sonymobile.com

Kai Peterson Per Runesonkai@bth.se per.runeson@cs.lth.se

Further reading

• Itkonen J, Mäntylä M, Lassenius C (2007) Defect DetectionEfficiency: Test Case Based vs. Exploratory Testing. ESEM’07, pp61–70

• Itkonen J., Mäntylä M. V. and Lassenius, C. The Role of the Tester'sKnowledge in Exploratory Software Testing IEEE Transactions onSoftware Engineering (2013) 39(3):707–724

• Micalef M, Porter C, Borg A, Do Exploratory Testers Need FormalTraining? An Investigation Using HCI Techniques, TAIC-PART 2016

• Afzal W, Ghazi, A N, Itkonen, J, Torkar, R, Andrews A, Khurram Bhatti,An Experiment on the Effectiveness and Efficiency of ExploratoryTesting, Empir Software Eng (2015) 20:844–878

Exploratory testing – Black or white?Using Focus Groups to Enhance Collaboration

ELIZABETH BJARNASON LU, PETER VISURI SONY

KAI PETERSEN BTH, PER RUNESON LTH