{

Context Driven Testing in an Agile Context

Mark Richards

@MarkAtScale

http://www.agilenotanarchy.com

MY AGILE TESTING JOURNEY

2000

2005

2012

Extreme Programming“The team consists of developers and customers – testers are redundant”

Scrum“The team is cross functional – testers are there to help your developers figure out the

automated tests that need writing”

Context driven testing“You know nothing John Snow – you’re

confusing checking with testing”

Agile Manifesto - Declaration of Interdependence“Improve effectiveness and reliability through situational-specific

strategies, processes and practices”

CONTEXT DRIVEN TESTING

The Seven Basic Principles

• The value of any practice depends on its context

• There are good practices in context, but there are no best practices

• People, working together, are the most important part of any project’s context

• Projects unfold over time in ways that are often not predictable

• The product is a solution. If the problem isn’t solved, the product doesn’t work

• Good software testing is a challenging intellectual process

• Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products

James Bach, Cem Kaner –http://context-driven-testing.com

A NOTE ON QUALITY

All tests are experiments performed to answer a question between what a product is and what it should be – Kaner, Bach, Pettichord

Quality is value to some person – Weinberg

THE SATISFICE CONTEXT MODEL

Test Process

Mission

Requirements

Test LabTest Team

Development

Source: Lessons learned in software testing: A context-driven approach – Kaner, Bach, Pettichord

THE SATISFICE CONTEXT MODEL - DEVELOPMENT

Test Process

Mission

Requirements

Test LabTest

Team

Development

Source: Lessons learned in software testing: A context-driven approach – Kaner, Bach, Pettichord

DevelopmentThe system that produces the product you will test. How do you receive the product? How testable is it?

• You should receive testable product early and often. The goal is shippable product every sprint

• Design will be emergent. Many features will be iterated upon

• Focus on refactoring will mean the balance between progression and regression will be radically different from waterfall

• Where will the “agile devs” or coach be on their agile testing journey?

THE SATISFICE CONTEXT MODEL - REQUIREMENTS

Test Process

Mission

Requirements

Test LabTest

Team

Development

Source: Lessons learned in software testing: A context-driven approach – Kaner, Bach, Pettichord

RequirementsThe criteria for a successful product. What are the product risks? Whose opinion about quality matters?

• Requirements are likely to be light-weight (User Stories)

• Immature agile teams are likely to lack effective contexting of user-stories within larger scenarios

• In theory the product owner is the sole opinion that should matter for quality – in practice, rarely so.

• You’re involved in requirements – influence testability.

• Can you improve the understanding of requirements by probing acceptance criteria and definition of done?

• Consider conference, inference and reference

THE SATISFICE CONTEXT MODEL – TEST TEAM

Test Process

Mission

Requirements

Test LabTest

Team

Development

Source: Lessons learned in software testing: A context-driven approach – Kaner, Bach, Pettichord

Test teamThe people available to test the product. Do you have the right staff? Are they up to speed on the technology

• An agile team is 7+-2 people, typically with 2 embedded testers

• Immature teams will “waterfall sprints” – if you think testing begins when you’re handed product to test, you’re in trouble

• How do you ensure diversity of testing domain experience?

• What is needed to co-ordinate test efforts across multiple teams?

• What is the integration process?• How do you grow the team’s thinking

without generating resistance?

THE SATISFICE CONTEXT MODEL – TEST LAB

Test Process

Mission

Requirements

Test LabTest

Team

Development

Source: Lessons learned in software testing: A context-driven approach – Kaner, Bach, Pettichord

Test LabThe system, tools, and materials that allow you to do your job. Do you have the right equipment? Is your bug-tracking system in good order?

• Test environments will be needed on Day 1 of an agile project

• Lightweight tools are prevalent. Prefer using the same tool as the developers for lifecycle/bug tracking.

• Are your test environments shared with waterfall projects? How will this impact you?

THE SATISFICE CONTEXT MODEL

Test Process

Mission

Requirements

Test LabTest

Team

Development

Source: Lessons learned in software testing: A context-driven approach – Kaner, Bach, Pettichord

MissionThe problems you must solve to be seen as successful by your clients. Find important bugs fast? Produce an accurate assessment of quality?

The object of planning is to make choices about test process that allow you to test within the constraints of the project environment, while exploiting your resources, enabling you to achieve your mission.

SOME THOUGHTS ON AUTOMATION

Many Agilists will claim automation is the panacea ….

• Expect, encourage and nurture good unit test automation practices by the developers. It’s great for white-box testing and eliminates waste from higher-order testing

• You can no more automate testing than you can automate programming – but automating checking leaves you time to focus on high-return testing

• If you have bad tests, automation can help you do bad testing faster

Prioritizing automation

• Automation of build and environment standup/promotion should come first. It enables more reliable, more regular delivery of product and is useful both to the devs for unit testing and you for higher order testing

• Automation of smoke tests should come second. It provides fast feedback on bad builds and avoids wasted testing effort.

• Consider minor scripts that aid testing without covering E2E scenarios• GUI automation should come last. Avoiding a crushing maintenance

burden and fragility requires significant skill

PRIORITIES

• You cannot test everything

Source: Lessons learned in software testing: A context-driven approach – Kaner, Bach, Pettichord

• It’s possible to test without questioning, but it’s not possible to test well .. If you do not question, your testing will be aimless and mechanical. How do you increase the power of your questions?.

• We expect the tester to learn, throughout the project, about the product, its market, its risks, and the ways in which it has failed previous tests. New tests are constantly created and used. They’re more powerful than older tests because they’re based on the tester’s continuously increasing knowledge.

• Every test we do, every document we write, every meeting we attend takes time away from running other tests that could expose a key defect

• Testing is prioritized in terms of the probability that some feature of the program will fail and the probable cost of failure, if this feature does fail. The greater the probability of an expensive failure, the more important it is to test that feature as early and as carefully as possible

CLOSING THOUGHTS

Good agile avoids the late stage tradeoff between reliability and time – exploit this!

Thought-leaders who are challenging me to grow• Gerry Weinberg• James Bach• Michael Bolton• Ilari Henrik• Griffon Jones• Lynne Mckee• Henrik Andersson