19 - 20 August 2016Bangalore
www.xpconference.in
#XPIndia2016
Pairwise Testing in Agile WorldClassic Combinatorial Test case design technique that can be applied to agile testing to save testing cost & time.
#XPIndia2016
How many combinations does it take to test this?
#XPIndia2016
Four fundamental challenges of software testingSource : Cem Kaner
• Complete testing is impossible• Testers misallocate resources because they fall for
the company’s process myths• Test groups operate under multiple missions, often
conflicting, rarely articulated• Test groups often lack skilled programmers, and a
vision of appropriate projects that would keep programming testers challenged
#XPIndia2016
Complete testing is IMPOSSIBLE (Cem Kaner)
• Test every possible input to every variable• Test every possible combination of inputs to every
combination of variables• Test every possible sequence through the program• Test every hardware / software configuration,
including configurations of servers not under your control
• Test every way in which the user might try to use the program
#XPIndia2016
Motivation for Pairwise Testing
• Wallace and Kuhn of NIST (National Institute of Standards & Technology) determined that 98% of the reported software defects in recalled medical devices could have been detected by testing all pairs of parameter settings.
D.R. Wallace, D.R. Kuhn, “Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data,” Intl. Journal of Reliability, Quality, and Safety Engineering, vol. 8, no. 4.
#XPIndia2016
Essence of Pairwise Testing
Total Possible Configurations:4 x 2 x 5 x 3 x 2 x 2 x 4 = 1920
Parameter Name Values No of ValuesAge 10, 14, 30, 50 4
Gender Male, Female 2
Certainty Probable, Doubtful, Possible, Established, Suspected
5
Severity Minor, Moderate, Major 3
Food True, False 2
Ethanol True, False 2
Medication COUMADIN 10 MG ORAL TABS, ASPIRIN 325 MG ORAL TABS, STENDRA TABS, ABILIFY TABS
4
#XPIndia2016
Software Failure Analysis The simplest bugs in a system are
generally triggered by a single input parameterE.g., failure occurs if
• pressure < 10 (1-way interaction) The next simplest category of bugs
consists of those dependent on interactions between pairs of parametersE.g., failure occurs if
• pressure < 10 & volume > 300 (2-way interaction)
#XPIndia2016
Software Failure Analysis Bugs involving interactions between 3
or more parameters are progressively less common, while at the same time progressively more expensive
Most complex failure required 4-way interaction
#XPIndia2016
Software Failure Analysis – Medical Devices
#XPIndia2016
The Pairwise Technique
Don’t test all combinations. Test all pairs. Pairwise testing – an approach to combinatorial testing that
executes a pairwise test data set.
Pairwise test data set - A set of test cases that covers all combinations of the selected test data values for every pair of a system’s input variables.
Pairwise testing is also known as 2-way testing; other approaches include 3-way, 4-way, … , n-way.
#XPIndia2016
An Example
X={A, B} Y={C, D} Z={E, F}
There are 2^3 = 8 Combinations
System S
#XPIndia2016
All Combinations (Test Cases)
Test ID X Y ZT1 A C E
T2 A C F
T3 A D E
T4 A D F
T5 B C E
T6 B C F
T7 B D E
T8 B D F
X={A, B} Y={C, D} Z={E, F}
#XPIndia2016
Pairwise Reductions by handYou can get rid of test case T2, since
AC is covered in T1 CF is covered in T6 AF is covered in T4
What other test cases can you get rid of?
#XPIndia2016
Pairwise Reductions by hand
Test ID X Y Z
T1 A C E
T4 A D F
T6 B C F
T7 B D E
#XPIndia2016
Pairwise Reductions
Number of inputs
Number of selected test data values
Total Number of combinations
Size of pairwise test set
3 2 8 47 2 128 8
13 3 1594323 18
40 3 1.2 x 1019 21
#XPIndia2016
Pairwise Reduction - Tools
#XPIndia2016
TestCover Commercial off the shelf tool Algorithm based on Bell Labs research Mathematically optimal among other tools Determines test scenarios/cases based
on input Test Factor Test Factor Values Documented Constraints https://testcover.com/sub/index.php
#XPIndia2016
TestCover ExampleTest Factor Number of Values Test Factor Values
Operating system 4 XP, Vista, MacOS, Linux
Display resolution 3 Low, medium, high
Connection 4 wi-fi, dsl, cable, lan
Browser 4 IE, Firefox, Opera, Safari
Application 3 App1, App2, App3
Total Configurations = 576
#XPIndia2016
The TestCover Request Header
Test Factor
Test Factor Values
Partition
#XPIndia2016
The Reduced Pairwise Subset
#XPIndia2016
Another Example – with Constraints
Header
Block
Block
Block
Partition
#XPIndia2016
The Result
#XPIndia2016
Caveat• Pairwise testing will not be effective if you choose
the wrong input test data values • We often assume defects can be immediately
observed. In fact, there may be slow internal corruption occurring
• High-use (including defaults) or high-risk combinations probably don’t get enough attention
Note: These dangers are not exclusive to pairwise testing. They affect all testing efforts
References http://www.pairwise.org https://testcover.com/sub/index.php http://
csrc.nist.gov/groups/SNS/acts/documents/verify07.pdf http://www.bcs.org/upload/pdf/lcopeland-070312.pdf