BUILD THE “RIGHT” REGRESSION SUITE USING BEHAVIOR DRIVEN TESTING - BDT Anand Bagmar

Test Practice Lead

@BagmarAnand about.me/anand.bagmar

ABOUT ME

Think ….

CASE STUDY

•  Multiple, long running projects

•  Legacy applications

•  Integrated

•  Limited / flaky / long-running Automation

•  Long regression cycle

•  Huge cost of fixing defects

FUNCTIONAL TESTING CHALLENGES

•  Brittle

•  Little / less value

•  No visibility into what is tested

•  Maintenance nightmare

•  Outdated

•  Expensive

AS A RESULT

•  Trust deficit on the team •  In existing Automation

•  In manual testing

•  Finger pointing / blame game

BIGGEST PROBLEM

My Case Study

Concept #1 Given -> When -> Then

Expected Functionality:

The customer should be able to withdraw money from his account via an ATM machine

BDD Example:

Given the account has balance of 5000

When the customer requests 1000

Then the account is debited by 1000

13

Workshop – Iteration 1 Requirements

PHASE 1 REQUIREMENTS

•  Guest User is able to search for a flight for a single traveller

•  Select option from Search Results page

•  Specify contact information for person booking the flight

A Guest User can …

WORKFLOW

Search Search Results

Contact Information

SEARCH MODULE

SEARCH RESULTS

CONTACT DETAILS

CONTACT DETAILS

PHASE 1 REQUIREMENTS

•  Guest User is able to search for a flight for a single traveller

•  Select option from Search Results page

•  Specify contact information for person booking the flight

A Guest User can …

WORKFLOW

Search Search Results

Contact Information

LETS GET STARTED … WRITE 2 TEST SCENARIOS

Iteration #1 Showcase

Concept #2 Test Specification Styles

•  Imperative

•  Declarative

TEST SPECIFICATION STYLES

A Guest User can …

AN EXAMPLE

Search Search Results

Contact Information

IMPERATIVE STYLE

Given I am a guest user on the home page

And I choose “round” trip option

And I select “Chicago” from the origin dropdown

And I select “San Francisco” from the destination dropdown

And I select departure date as “5 December 2013”

And I select returning date as “25 December 2013”

When I click on Search

Then I should see the search results page

And I should see at least 1 option for my criteria

...

IMPERATIVE STYLE …

…

When I select the “first” option

Then I am on the Contact Information page

When I enter first name as “Anand”

And I enter last name as “Bagmar”

…

…

And I click the “Select and Continue” button

Then I should be on the next page

DECLARATIVE STYLE

Given I am a guest user

When I search for flight options for a “one-way” trip for “1” “Adult” from “Chicago” to “San Francisco”

And I select the “first” flight

And I enter “valid” contact details for “traveller1”

Then I am able to Save and Continue

•  Ben Mabey - http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html

•  Alister Scott - http://watirmelon.com/2010/12/10/cucumber-imperative-or-declarative-that-is-the-question/

REFERENCES

Workshop – Iteration 2 Evolved Requirements

•  User registration from Contact information page

•  Registered user can login from home page (below search criteria)

•  For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information

PHASE 2 REQUIREMENTS

Login Search Search Results

Contact Information

•  Pre-filled

WORKFLOW … ENHANCED

REGISTERED USER

•  User registration from Contact information page

•  Registered user can login from home page (below search criteria)

•  For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information

PHASE 2 REQUIREMENTS

Login Search Search Results

Contact Information

•  Pre-filled

WORKFLOW … ENHANCED

LETS GET STARTED … WRITE 2 TEST SCENARIOS

Iteration #2

Showcase

WHY IS THIS IMPORTANT?

Concept #3 The Test Pyramid

Test Automation

A PRACTICE THAT MAKES TEAMS SUCCESSFUL?

Test Automation

A PRACTICE THAT MAKES TEAMS UNSUCCESSFUL?

What is Test Automation?

Automation -

A Safety Net

Why do we do Test Automation?

QUICK FEEDBACK

How to get quick feedback from

Test Automation?

•  “Succeeding with Agile” – Mike Cohn

•  Martin Fowler – Test Pyramid

martinfowler.com/bliki/TestPyramid.html

THE TEST PYRAMID

IDEAL TEST AUTOMATION PYRAMID

Unit

Manual / Exploratory

Integration

JavaScript

View

Web Service

UI

Cost

Impact

Time

Business- facing Tests

Technology- facing Tests

REALITY

ICE-CREAM CONE ANTI PATTERN

Manual / Exploratory

UI

Web Service

View

JavaScript

Integration

Unit

Web Service

View

JavaScript

Integration

Unit

Developers Test Pyramid QA Team Test Pyramid

DUAL TEST PYRAMID ANTI-PATTERN

Manual / Exploratory

UI

Web Service

CUP-CAKE ANTI-PATTERN

Concept #4 Behavior Driven Testing

BDT

BEHAVIOR DRIVEN TESTING (BDT)

•  How is this functionality going to be used by the end-user?

•  What is their thought process going to be when using this product?

•  What “core-business-value” should be delivered to the end-user?

THE BIG PICTURE

Identify the “right” type of tests that sit on top of the Test Pyramid

My Case Study

HOW TO SOLVE THE PROBLEM?

•  Remove the ambiguity

•  Same visibility for all

SOLUTION

•  Whiteboards

•  Mind maps

•  Flow charts

•  Collaboration!!

TOOLS TO ENABLE BDT

HOW DID WE ACCOMPLISH THIS?

IDENTIFY BUSINESS FLOWS

•  Identify business flows based on personas, and how the personas use the business functionality, or,

•  Identify business flows, and how different user personas can use the business flows

NOTE: Business flows are different from UI flows

HOW TO IDENTIFY BUSINESS FLOWS ?

PRIORITIZE BUSINESS FLOWS

1st candidate •  Automate! •  High Value, Low Cost

2nd candidate •  Automate! •  High Value, High Cost

Can Automate? •  Manual Regression? •  Low Value, Low Cost

Do Not Automate •  Manual Regression! •  Low Value, High Cost

High Value

High Cost

Low Value

Low Cost

AUTOMATE THE TOP LAYER USING

•  BDD Tools, or,

•  Directly in any appropriate programming language / tools

•  Identification of Tests

•  Automate it!

WHAT’S NEXT?

Caution!!

AUTOMATE TESTS THAT ARE VALUABLE, NOT BECAUSE YOU CAN OR LIKE TO!

CAUTION!

Who is doing the action?

What is the business functionality?

Depicts a user flow / user journey

Actions & Verifications happen implicitly

Given auctioneer creates an auction And 5 vehicles are added to the auction When auctioneer starts the auction And auctioneer sets a starting bid of 5000 And buyer_1 bids And buyer_2 bids Then auctioneer Sells the vehicle And buyer_2 wins the vehicle

EXAMPLE

Delete profile

UPDATE BUSINESS FLOWS

1st candidate •  Automate! •  High Value, Low Cost

2nd candidate •  Automate! •  High Value, High Cost

Can Automate? •  Manual Regression? •  Low Value, Low Cost

Do Not Automate •  Manual Regression! •  Low Value, High Cost

High Value

High Cost

Low Value

Low Cost

Project Management tool Mingle

Test case repository (manual + automated)

Cucumber .feature files with appropriate tagging

Feature files / Scenarios 33 / 65

# of Manual Tests 0

Tests running on CI (Jenkins) Since the 1st test was automated

Smoke tests execution time 15 minutes

Regression tests execution time

45 minutes

FINAL STATE … IN MY CASE STUDY

•  Builds and deployments were automated

•  Tests ran on demand, automatically! •  Smoke / Regression / Javascript / Unit

•  Functional Performance

•  Concurrent Testing

ADDITIONALLY …

How did this technique (BDT) become

successful?

CONTEXT IS KING!

Domain

Distributed Teams

Skills

Capabilities Product-under test

Tech Stack

Process

Tools

BDT – Lessons Learnt

NOT work in isolation

BDT WILL …

BDT REQUIRES …

Lower layers of Test Pyramid to be well automated!

BDT REQUIRES …

Dedicated and Focused testing (manual / exploratory)

BDT helps identify the “right” type of regression tests!

Executable Specs

BDT Scenarios

Business Idea

Features

Epics

Stories

Iterations

Releases

BDT VALUE PROPOSITION

BDT VALUE PROPOSITION …

Incrementally build

Business Flows

Regress Business Value

Test in-sync with Product Functionality

Effective Tests

BDT VALUE PROPOSITION …

Remove ambiguity

Understand new functionality

Living Documentation

Onboarding

abagmar@gmail.com

Anand.Bagmar@thoughtworks.com

@BagmarAnand

about.me/anand.bagmar

THANK YOU