Structured Exploratory Testing – Method in Madness

© 2012 WIPRO LTD | WWW.WIPRO.COM1

Structured Exploratory Testing – Method in Madness

Vivek DevarajanSenior Consultant, Wipro Technologies


Scripted versus Non-Scripted Testing

Some Exploratory Testing Ideas2

1

Session Based Test Management3

Benefits of Exploratory Testing4

Case Studies5





Freestyle

Testing

Scripted Testing

Structured Exploratory Testing / Session Based Test Management

•Elaborate Documentation•Static Test Suite•Higher Testing Costs•Limited Defect Detection

•Concise Documentation•Dynamic Test Optimization•Lower Testing Costs•Better Defect Detection

•No Accountability•No Predictability•High Possibility of Defect Slippage

Leveraging the benefits of both Testing StylesMethodical Madness


Some Exploratory Testing Ideas


Exploratory Testing using Question PatternsWhile testing a system, how often have you felt “I have tested something similar before?”

The Domain FactorOn the other hand, how often have you been bogged down by an application, just because it was from a different domain that you are not so familiar with?

The Reuse FactorIs there a way to develop testing ideas once and reuse them across domains / applications?

The Knowledge FactorIs there a way an experienced tester or domain expert can impart his/her experience to others?

The Exploratory Testing Context Is there a way to explore the application systematically and find bugs?


Question Patterns and Exploratory TestingQuestion Patterns: A Set of interrelated Questions grouped together, focused on an aspect of the system under test that•Enable someone to understand the application better•Can be used as a technique for requirements capture

Patterns help the tester explore the application systematically and find bugs

Example: Parent and Child Entity PatternScenario A).Consider an online shopping application where the user can add / modify / delete multiple items in his shopping cart order. In this case, the shopping cart is a business entity which has multiple child entities (items) within itself (a one-to-many relationship between the cart and the items) where the child entity (item) can be added/updated or deleted.Scenario B). Consider an application where the user can apply for an insurance policy online, to which he can add / modify / delete multiple beneficiaries. In this case, the policy is a business entity which has multiple child entities (beneficiaries) within itself (a one-to-many relationship between the policy and the beneficiaries) where the child entity (beneficiary) can be added/updated or deleted.


Question Patterns and Exploratory TestingQuestions that can be asked: Parent and Child Entity Pattern•What are the business entities involved?

•What is the relationship between them? (One to Many, One-to-One, etc)

•What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search)

•What are the field level and business validations for each entity?

Answers to such questions enable testers to find gaps in requirements (or even capture requirements).

Answers to such questions also serve as a guideline and a checklist during exploratory testing.


Question Patterns and Exploratory TestingQuestions that can be asked: Parent and Child Entity Pattern•What are the business entities involved?

•What is the relationship between them? (One to Many, One-to-One, etc)

•What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search)

•What are the field level and business validations for each entity?

Answers to such questions enable testers to find gaps in requirements (or even capture requirements).

Answers to such questions also serve as a guideline and a checklist during exploratory testing.


Exploratory Testing by Looking for Specific Problems

•Purpose of Testing – Looking for Bugs/Problems that could impact the stakeholders

•Defect Seeking v/s Defect Finding

•Make a list of possible issues

•Design tests to find ways to stimulate the possible issues identified


Exploratory Testing by Analyzing Data Flow

•Most Applications process / manage data

•Identify the input and output data

•Verify the output data for various combinations of input data (including green path values, invalid values, boundary conditions, large values, etc)

•Try modifying the data at various stages in the flow and see the impact

•Identify the right combinations / representatives of data


Exploratory Testing by Analyzing Flows

•Identify end-to-end flows and branching

•Depict the branching graphically, if needed

•Try parallel flows

•Try varying the sequencing / timing of the flows

•Analyze the impact of input data values on the flows

•Check the main flows as well as alternate/exceptional flows

•Check for state transition flows


Session Based Test Management – Accountable and Manageable Exploratory

Testing


Session Based Test Management - Approach

Test Execution is divided into time-boxed Test Sessions, where the tester does (uninterrupted) exploratory testing with a specific charter

Tester takes brief notes about the testing done and the issues found

Tester uses a checklist of testing ideas as a guide for testing

Further tests are dynamically optimized depending on the results of the current test

The Test Lead conducts a ‘Debrief’ session with the testers on the test findings and plans further executions based on the findings


Session Based Test Management – Session Contents

The contents of a typical test session are as follows:

•Charter: Contains the mission of the given session. Typically it mentions the application areas to be covered / explored, at a high level.•Tester name, Session Time Start and End•Task Breakdown: Time taken for Test execution v/s Bug Reporting, Charter (Original Session Charter) v/s Opportunity (Diversions taken)•Test Notes: Notes of what the tester executed, in brief.•Bugs/Issues found


Session Based Test Management – Accountability

•Tester’s Activities: Tracked via Test Session reports

•Coverage and Testing Progress Tracking: • High Level Coverage: Number of functional areas and quality

criteria covered• Granular Coverage: Number of checklists/items/test ideas

covered for each functional area•Traceability: Each test idea and item in the checklist can be mapped to requirements


Benefits of Exploratory Testing


Exploratory Testing - Benefits

•Lesser Preparation and lesser effort• •Quicker Bug Detection

•Leverages on Tester’s Abilities, experience and creativity

•Works even if requirements are not fully frozen


Skilled Testing v/s Deskilling – The Classic Debate

•Deskilling via Detailed Test Cases leads to higher test design and maintenance costs and limited bug detection capabilities

•Testers with lesser experience can be trained in skilled exploratory testing

•Skilled Testing is more cost effective and results in better product quality


Case Studies


Case Study 1 – The Problem

The Context:•Previous customer concerns over quality in beta releases

•Large volume of functional tests to be executed in a short time span

- Conventional Testing Estimations: 3000 test cases to be designed and executed in 2 months (including KAP phase), requiring 14 testers

•Inexperienced Testers with no domain knowledge

•Customer more focused on bugs/issues than in testing processes

A Police IT Application for a Government Customer


Case Study 1 – Testing Approach and Results

•Question Patterns used to arrive at checklist of testing ideas- Entity Relationship pattern- State Transition Pattern- Master Data – Transaction Data Pattern

•Checklists consisting of Business Conditions and Testing Ideas

•Initial Test Sessions focused on exploration of the business operations of entities

•Deep Dive Test Sessions focused on finding specific problems

•Test Sessions for End to End Testing focused on Application Flow and Data flow

Results: 65% reduction in testing efforts and 60% better defect detection

A Police IT Application for a Government Customer


Case Study 2 – The Problem

The Context:

•The application is intended to enable healthcare providers and pharmacies to place orders quickly

•Bugs not being found early enough

•High cost of testing

•Frequent patch releases with incomplete functionality

•Mix of experienced and inexperienced testers, with fair domain knowledge

•Parallel run of conventional testing and Session Based Test Management

An Order Management Application for a Healthcare customer


Case Study 2 – Testing Approach and Results

•Question Patterns used to arrive at checklist of testing ideas for regression testing

•Checklists consisting of Business Conditions and functional changes to be checked for

•Initial Test Sessions focused on the exploration of new functionalities

•Subsequent Test Sessions focused on user experience

Results: 50% reduction in test design efforts and quicker defect detection

An Order Management Application for a Healthcare customer


Thank youVivek DevarajanSenior Consultant, Wipro Technologies

[email protected]


Appendix


Artifacts Created

Checklists Created for Session Based Test Management:

Sample Test Session:

Microsoft Office Excel Worksheet

Microsoft Office Excel 97-2003 Worksheet





Session Based Test Management

Freestyle

Testing

Scripted Testing


Exploratory Testing

SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles

S. No.

Factor Scripted Testing Session Based Test Management

Freestyle Testing

1 Cost Very High Medium to Low Very Low2 Defect

Detection• Limited Effectiveness• Sometimes late in the cycle

•Highly Effective•Early in the cycle

•Highly Effective•Very unpredictable

3 Accountability Very High Very High Very Low


SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles

S. No.

Factor Scripted Testing Session Based Test Management

Freestyle Testing

4 Predictability Very High Medium Very Low5 Possibility of

Defect Slippage

Medium Low High

6 Documentation Level

Very High Medium Very Low

7 Requirements Coverage

Very High Very High Very Low

8 Requirements Traceability

Very High High Very Low



Comparison of Structured v/s Freestyle Testing

Freestyle Testing

Scripted Testing

Benefits: •High Predictability•Deskilling (Test Execution)•High AccountabilityDrawbacks:•Very High Costs (Resources + Tools) for Test Creation as well as Maintenance•Limited Defect Detection Capabilities (Lack of Proactive/Creative Approaches)•Needs well documented requirements

Benefits: •Very Low Costs for Test Creation/Maintenance•Highly effective in quick Defect Seeking, especially effective for tricky bugs•Does not need well documented requirementsDrawbacks:•Low Accountability•Relies on the creativity of the Tester



Freestyle

Testing

Scripted Testing

Automated Testing

Session Based Test Management Exploratory

Testing

•Elaborate Documentation•Static Test Suite•Higher Testing Costs•Limited Defect Detection

•Concise Documentation•Dynamic Test Optimization•Lower Testing Costs•Better Defect Detection

•No Accountability•No Predictability•High Possibility of Defect Slippage

SBTM - Leveraging the benefits of both Testing Styles


Evolution of Session Based Test Management

Adhoc / Freestyle Testing

Exploratory Testing

Session Based Test

Management

Evol

utio

n


Salient Features of Session Based Test Management• Application Learning, Test Design and Test Execution happen in

Parallel

• A Structured, measurable and accountable method of Exploratory Testing

• Crisp Checklists to guide the tester on what needs to be tested

• Execution happens in Test Sessions (Uninterrupted Test Execution Time) with specific charters, after which tester takes notes and makes a report

• Tester takes a different execution route, if required, depending on issues found in a Test Session

• Future test sessions are planned based on results from current session


Metrics for Session Based Test Management


Test Session Metrics

Cumulative Defect Trend

Defect TrendExecution Metrics

E:\Projects\Cardinal\Test Execution Prod E:\Projects\Cardinal\

Defect Trend.bmp

E:\Projects\Cardinal\Cumulative Defects

Documents

Structured Exploratory Testing – Method in Madness