Chapter 14 Software Testing Techniques

These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 1

Software Engineering: A PractitionerSoftware Engineering: A Practitioner’’s Approach, 6/es Approach, 6/e

Chapter 14Chapter 14Software Testing TechniquesSoftware Testing Techniques

copyright © 1996, 2001, 2005R.S. Pressman & Associates, Inc.

For University Use OnlyMay be reproduced ONLY for student use at the university level

when used in conjunction with Software Engineering: A Practitioner's Approach.Any other reproduction or use is expressly prohibited.


TestabilityTestability OperabilityOperability——it operates cleanlyit operates cleanly ObservabilityObservability——the the results of each test case are readilyresults of each test case are readily

observedobserved ControllabilityControllability——the degree to which testing can bethe degree to which testing can be

automated and optimizedautomated and optimized DecomposabilityDecomposability——testing can be targetedtesting can be targeted SimplicitySimplicity——reduce complex architecture and logic toreduce complex architecture and logic to

simplify testssimplify tests StabilityStability——few changes are requested during testingfew changes are requested during testing UnderstandabilityUnderstandability——of the designof the design


What is a What is a ““GoodGood”” Test? Test?

A good test has a high probability of findingA good test has a high probability of findingan erroran error

A good test is not redundant.A good test is not redundant. A good test should be A good test should be ““best of breedbest of breed”” A good test should be neither too simple norA good test should be neither too simple nor

too complextoo complex


Test Case DesignTest Case Design

"Bugs lurk in corners "Bugs lurk in corners and congregate at and congregate at boundaries ..."boundaries ..."

BorisBoris Beizer Beizer

OBJECTIVEOBJECTIVE

CRITERIACRITERIA

CONSTRAINTCONSTRAINT

to uncover errorsto uncover errors

in a complete mannerin a complete manner

with a minimum of effort and timewith a minimum of effort and time


Exhaustive TestingExhaustive Testing

loop < 20 Xloop < 20 X

There are 10 possible paths! If we execute oneThere are 10 possible paths! If we execute onetest per millisecond, it would take 3,170 years totest per millisecond, it would take 3,170 years totest this program!!test this program!!

1414


Selective TestingSelective Testing

loop < 20 Xloop < 20 X

Selected pathSelected path


Software TestingSoftware Testing

Methods

Strategies

white-boxmethods

black-box methods


White-Box TestingWhite-Box Testing

... our goal is to ensure that all ... our goal is to ensure that all statements and conditions have statements and conditions have been executed at least once ...been executed at least once ...


Why Cover?Why Cover?

logic errors and incorrect assumptions logic errors and incorrect assumptions are inversely proportional to a path's are inversely proportional to a path's execution probabilityexecution probability

we often we often believebelieve that a path is not that a path is not likely to be executed; in fact, reality is likely to be executed; in fact, reality is often counter intuitiveoften counter intuitive

typographical errors are random; it's typographical errors are random; it's likely that untested paths will contain likely that untested paths will contain some some


Basis Path TestingBasis Path TestingFirst, we compute the cyclomatic complexity:

number of simple decisions + 1

or

number of enclosed areas + 1

In this case, V(G) = 4


Cyclomatic ComplexityCyclomatic ComplexityA number of industry studies have indicated A number of industry studies have indicated that the higher V(G), the higher the probability that the higher V(G), the higher the probability or errors.or errors.

V(G)V(G)

modulesmodules

modules in this range are modules in this range are more error pronemore error prone


Basis Path TestingBasis Path TestingNext, we derive the Next, we derive the independent paths:independent paths:

Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths

Path 1: 1,2,3,6,7,8Path 1: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 3: 1,2,4,7,8Path 3: 1,2,4,7,8Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8

Finally, we derive testFinally, we derive testcases to exercise these cases to exercise these paths.paths.

11

22

3344

55 66

77

88


Basis Path Testing NotesBasis Path Testing Notes

you don't need a flow chart, you don't need a flow chart, but the picture will help when but the picture will help when you trace program pathsyou trace program paths

count each simple logical test, count each simple logical test, compound tests count as 2 or compound tests count as 2 or moremore

basis path testing should be basis path testing should be applied to critical modulesapplied to critical modules


Graph MatricesGraph Matrices A graph matrix is a square matrix whose sizeA graph matrix is a square matrix whose size

(i.e., number of rows and columns) is equal to(i.e., number of rows and columns) is equal tothe number of nodes on a flow graphthe number of nodes on a flow graph

Each row and column corresponds to anEach row and column corresponds to anidentified node, and matrix entries correspondidentified node, and matrix entries correspondto connections (an edge) between nodes.to connections (an edge) between nodes.

By adding a By adding a link weightlink weight to each matrix entry, the to each matrix entry, thegraph matrix can become a powerful tool forgraph matrix can become a powerful tool forevaluating program control structure duringevaluating program control structure duringtestingtesting


Control Structure TestingControl Structure Testing

Condition testingCondition testing —— a test case design method that a test case design method thatexercises the logical conditions contained in a programexercises the logical conditions contained in a programmodulemodule

Data flow testingData flow testing —— selects test paths of a program selects test paths of a programaccording to the locations of definitions and uses ofaccording to the locations of definitions and uses ofvariables in the programvariables in the program


Loop TestingLoop Testing

Nested Nested LoopsLoops

ConcatenatedConcatenated Loops Loops Unstructured Unstructured

LoopsLoops

Simple Simple looploop


Loop Testing: Simple LoopsLoop Testing: Simple Loops

Minimum conditionsMinimum conditions——Simple LoopsSimple Loops1. skip the loop entirely1. skip the loop entirely2. only one pass through the loop2. only one pass through the loop3. two passes through the loop3. two passes through the loop4. m passes through the loop m < n4. m passes through the loop m < n5. (n-1), n, and (n+1) passes through 5. (n-1), n, and (n+1) passes through the loopthe loopwhere n is the maximum number where n is the maximum number of allowable passesof allowable passes


Loop Testing: Nested LoopsLoop Testing: Nested Loops

Start at the innermost loop. Set all outer loops to their Start at the innermost loop. Set all outer loops to their minimum iteration parameter values.minimum iteration parameter values.Test the min+1, typical, max-1 and max for the Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their innermost loop, while holding the outer loops at their minimum values.minimum values.Move out one loop and set it up as in step 2, holding all Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until other loops at typical values. Continue this step until the outermost loop has been tested.the outermost loop has been tested.

If the loops are independent of one another If the loops are independent of one another then treat each as a simple loop then treat each as a simple loop else* treat as nested loops else* treat as nested loopsendifendif* * for example, the final loop counter value of loop 1 is for example, the final loop counter value of loop 1 is used to initialize loop 2.used to initialize loop 2.

Nested LoopsNested Loops

Concatenated LoopsConcatenated Loops


Black-Box TestingBlack-Box Testing

requirementsrequirements

eventseventsinputinput

outputoutput


Black-Box TestingBlack-Box Testing How is functional validity tested?How is functional validity tested? How is system behavior and performance tested?How is system behavior and performance tested? What classes of input will make good test cases?What classes of input will make good test cases? Is the system particularly sensitive to certain inputIs the system particularly sensitive to certain input

values?values? How are the boundaries of a data class isolated?How are the boundaries of a data class isolated? What data rates and data volume can the systemWhat data rates and data volume can the system

tolerate?tolerate? What effect will specific combinations of data haveWhat effect will specific combinations of data have

on system operation?on system operation?


Graph-Based MethodsGraph-Based Methods

newfile

menu select generates(generation time < 1.0 sec)

documentwindow

documenttext

is represented ascontains

Attributes:background color: whitetext color: default color

or preferences

(b)

object#1

Directed link(link weight)

object#2

object#3

Undirected link

Parallel links

Node weight(value)

(a)

allows editing of

To understand theTo understand theobjects that areobjects that aremodeled in softwaremodeled in softwareand the relationshipsand the relationshipsthat connect thesethat connect theseobjectsobjects

In this context, we considerIn this context, we considerthe term the term ““objectsobjects”” in the in thebroadest possible context. Itbroadest possible context. Itencompasses data objects,encompasses data objects,traditional componentstraditional components(modules), and object-(modules), and object-oriented elements oforiented elements ofcomputer software.computer software.


Equivalence PartitioningEquivalence Partitioning

useruserqueriesqueries mousemouse

pickspicks

outputoutputformatsformats

promptsprompts

FKFKinputinput

datadata


Sample Equivalence ClassesSample Equivalence Classes

user supplied commandsuser supplied commandsresponses to system promptsresponses to system promptsfile namesfile namescomputational datacomputational data physical parameters physical parameters bounding values bounding values initiation values initiation valuesoutput data formattingoutput data formattingresponses to error messagesresponses to error messagesgraphical data (e.g., mouse picks)graphical data (e.g., mouse picks)

data outside bounds of the program data outside bounds of the program physically impossible dataphysically impossible dataproper value supplied in wrong placeproper value supplied in wrong place

Valid dataValid data

Invalid dataInvalid data


Boundary Value AnalysisBoundary Value Analysis

useruserqueriesqueries mousemouse

pickspicks

outputoutputformatsformats

promptsprompts

FKFKinputinput

datadata

outputoutputdomaindomaininput domaininput domain


Comparison TestingComparison Testing

Used only in situations in which the reliability ofUsed only in situations in which the reliability ofsoftware is absolutely critical (e.g., human-ratedsoftware is absolutely critical (e.g., human-ratedsystems)systems) Separate software engineering teams develop independentSeparate software engineering teams develop independent

versions of an application using the same specificationversions of an application using the same specification Each version can be tested with the same test data to ensure Each version can be tested with the same test data to ensure

that all provide identical outputthat all provide identical output Then all versions are executed in parallel with real-timeThen all versions are executed in parallel with real-time

comparison of results to ensure consistencycomparison of results to ensure consistency


Orthogonal Array TestingOrthogonal Array Testing Used when the number of input parameters is small and

the values that each of the parameters may take areclearly bounded

One input item at a time L9 orthogonal array

XY

Z

XY

Z


Object-Oriented TestingObject-Oriented Testing


OOTOOT——Test Case DesignTest Case DesignBerard Berard [BER93] proposes the following approach:[BER93] proposes the following approach:

1.1. Each test case should be uniquely identified and should be explicitlyEach test case should be uniquely identified and should be explicitlyassociated with the class to be tested,associated with the class to be tested,2.2. The purpose of the test should be stated,The purpose of the test should be stated,3.3. A list of testing steps should be developed for each test and shouldA list of testing steps should be developed for each test and shouldcontain [BER94]:contain [BER94]:

a.a. a list of specified states for the object that is to be testeda list of specified states for the object that is to be testedb.b. a list of messages and operations that will be exercised as a list of messages and operations that will be exercised as

a consequence of the testa consequence of the testc.c. a list of exceptions that may occur as the object is testeda list of exceptions that may occur as the object is testedd.d. a list of external conditions (i.e., changes in the environment externala list of external conditions (i.e., changes in the environment external

to the software that must exist in order to properly conduct the test)to the software that must exist in order to properly conduct the test)e.e. supplementary information that will aid in understanding or supplementary information that will aid in understanding or

implementing the test.implementing the test.


Testing MethodsTesting Methods Fault-based testingFault-based testing

The tester looks for plausible faults (i.e., aspects of the implementation of the The tester looks for plausible faults (i.e., aspects of the implementation of thesystem that may result in defects). To determine whether these faults exist, testsystem that may result in defects). To determine whether these faults exist, testcases are designed to exercise the design or code.cases are designed to exercise the design or code.

Class Testing and the Class HierarchyClass Testing and the Class Hierarchy Inheritance does not obviate the need for thorough testing of all derived classes.Inheritance does not obviate the need for thorough testing of all derived classes.

In fact, it can actually complicate the testing process.In fact, it can actually complicate the testing process. Scenario-Based Test DesignScenario-Based Test Design

Scenario-based testing concentrates on what the user does, not what the productScenario-based testing concentrates on what the user does, not what the productdoes. This means capturing the tasks (via use-cases) that the user has to perform,does. This means capturing the tasks (via use-cases) that the user has to perform,then applying them and their variants as tests.then applying them and their variants as tests.


OOT Methods: Random TestingOOT Methods: Random Testing

Random testingRandom testing identify operations applicable to a classidentify operations applicable to a class define constraints on their usedefine constraints on their use identify aidentify a minimum test sequenceminimum test sequence

an operation sequence that defines the minimum lifean operation sequence that defines the minimum lifehistory of the class (object)history of the class (object)

generate a variety of random (but valid) test sequencesgenerate a variety of random (but valid) test sequences exercise other (more complex) class instance life historiesexercise other (more complex) class instance life histories


OOT Methods: Partition TestingOOT Methods: Partition Testing Partition TestingPartition Testing

reduces the number of test cases required to test a class inreduces the number of test cases required to test a class inmuch the same way as equivalence partitioning formuch the same way as equivalence partitioning forconventional softwareconventional software

state-based partitioningstate-based partitioning categorize and test operations based on their ability to changecategorize and test operations based on their ability to change

the state of a classthe state of a class attribute-based partitioningattribute-based partitioning

categorize and test operations based on the attributes that theycategorize and test operations based on the attributes that theyuseuse

category-based partitioningcategory-based partitioning categorize and test operations based on the generic functioncategorize and test operations based on the generic function

each performseach performs


OOT Methods: Inter-Class TestingOOT Methods: Inter-Class Testing Inter-class testingInter-class testing

For each client class, use the list of class operators toFor each client class, use the list of class operators togenerate a series of random test sequences. The operatorsgenerate a series of random test sequences. The operatorswill send messages to other server classes.will send messages to other server classes.

For each message that is generated, determine theFor each message that is generated, determine thecollaborator class and the corresponding operator in thecollaborator class and the corresponding operator in theserver object.server object.

For each operator in the server object (that has been invokedFor each operator in the server object (that has been invokedby messages sent from the client object), determine theby messages sent from the client object), determine themessages that it transmits.messages that it transmits.

For each of the messages, determine the next level ofFor each of the messages, determine the next level ofoperators that are invoked and incorporate these into the testoperators that are invoked and incorporate these into the testsequencesequence


OOT Methods: Behavior TestingOOT Methods: Behavior Testingempty

acctopen setup Accnt

set up

acct

deposit(initial)

working

acct

withdrawal(final)

deadacct close

nonworkingacct

deposit

withdrawbalance

credit

accntInfo

Figure 14.3 State diagram for Account class (adapted from [KIR94])

The tests to beThe tests to bedesigned shoulddesigned shouldachieve all stateachieve all statecoveragecoverage[KIR94]. That is,[KIR94]. That is,the operationthe operationsequencessequencesshould cause theshould cause theAccount class toAccount class tomake transitionmake transitionthrough allthrough allallowable statesallowable states


Testing PatternsTesting PatternsPattern name:Pattern name: pair testing pair testing

Abstract: Abstract: A process-oriented pattern, pair testing describes a technique that A process-oriented pattern, pair testing describes a technique thatis analogous to pair programming (Chapter 4) in which two testers workis analogous to pair programming (Chapter 4) in which two testers worktogether to design and execute a series of tests that can be applied to unit,together to design and execute a series of tests that can be applied to unit,integration or validation testing activities.integration or validation testing activities.

Pattern name: Pattern name: separate test interface separate test interfaceAbstract: Abstract: There is a need to test every class in an object-oriented system, There is a need to test every class in an object-oriented system,including including ““internal classesinternal classes”” (i.e., classes that do not expose any interface (i.e., classes that do not expose any interfaceoutside of the component that used them). The separate test interfaceoutside of the component that used them). The separate test interfacepattern describes how to create pattern describes how to create ““a test interface that can be used toa test interface that can be used todescribe specific tests on classes that are visible only internally to adescribe specific tests on classes that are visible only internally to acomponent.component.”” [LAN01] [LAN01]

Pattern name: Pattern name: scenario testing scenario testingAbstract: Abstract: Once unit and integration tests have been conducted, there is aOnce unit and integration tests have been conducted, there is aneed to determine whether the software will perform in a manner thatneed to determine whether the software will perform in a manner thatsatisfies users. The scenario testing pattern describes a technique forsatisfies users. The scenario testing pattern describes a technique forexercising the software from the userexercising the software from the user’’s point of view. A failure at this levels point of view. A failure at this levelindicates that the software has failed to meet a user visible requirement.indicates that the software has failed to meet a user visible requirement.[KAN01][KAN01]

Åben Uddannelse

Software Engineering, Efteråret 2002 1

Test i XP

Automatisk test

Test i XP 2

Software testing

“ Testing is the process ofexecuting a program with the

intent of finding errors.”

Glen Myers

Åben Uddannelse


Test i XP 3

Testing You couldn’t possibly write all those tests? It

would take too much time. ”Programmerswon’t write tests!”

Unless The design is as simple as it can be You are programming with a partner, so if you

couldn’t think of another test your partner can You feel good when you see the tests all running Your customer feels good about the system when

they see all of their tests running

Test i XP 4

Test-First Development Tests are specified before software is written

Users specify inputs and outputs as part of test “Outside-in” perspective improves the software design Ensures software delivers expected results Writing tests is part of s/w design process

Detailed test knowledge captured in tests Functional Tests specify requirements in detail Unit Tests developed as soon as component/class identified;

defines software interface Tests drive the development process

Software is only written because tests are failing. Ensures all software is tested Ensures all requirements are captured as tests

Åben Uddannelse


Test i XP 5

Timelines Acceptance need to be available in the same

iteration as the story is scheduled The only way to know that the stories are done is

that the acceptance tests are passed Programmers have the right to know what is

needed Therefore insist on automated functional tests

Customers have the right to see progress in arunning system

Proven to to work by automated tests that you specify

Test i XP 6

Functional Tests Specified by the user Implemented by users, developers,

and/or test team Automated Run at least daily Part of the specification

Åben Uddannelse


Test i XP 7

Test Metrics

Tests failed

Tests passed

1 2 3 4 5 6 7

Key:

Test i XP 8

Unit Tests Testing bit by bit

Tests made by the programmers forthe programmers

Åben Uddannelse


Test i XP 9

Unit Tests

A unit test is written BEFORE THE CODE ISWRITTEN for each piece of the system.

When you create your tests first, before the code, you will find it mucheasier and faster to create your code.

The combined time it takes to create a unit test and create some codeto make it pass is about the same as just coding it up straight away.

But, if you already have the unit tests you don't need to create themafter the code saving you some time now and lots later.

Eases regression testing JUnit (for Java), SUnit (for Squeak) allow for running all unit tests.

When a bug is found, a new unit test is written to catch for that bug in the future

Test i XP 10

Unit Tests Written by developers Always run at 100% Support design, coding, refactoring,

and quality.

Åben Uddannelse


Test i XP 11

Testing steps Create a test class

Name it like ClassNameTestCase Code a setup Write a test Run the test Modify the model code and repeat steps four

and five until the test runs successfully Release the code and the test case

12

Automated Testing Continuous Automated Testing throughout system lifecycle

during development & integration before delivery, after installation Tests are rerun many times every day

Low cost regression testing since most tests are automated Higher initial cost recouped before initial delivery; free

thereafter “one button” testing, anytime, anywhere

Improves Developer Morale Developers enjoy testing, because it is coding Removes developers’ fear of breaking something Reduces maintenance cost significantly

Åben Uddannelse


Test i XP 13

What is JUnit? A small testing framework written in Java. A series of extensible classes that do much

of the testing grunt work for us. (i.e. countingand reporting errors and failed tests, runningtests in batch, etc.)

Very handy for Extreme Testing… Developed by

Kent Beck, [email protected] Erich Gamma, [email protected]

http://www.junit.org/

Test i XP 14

Essential Classes junit.framework.TestCase

Allows running multiple tests Does all counting and reporting of errors

junit.framework.Assert A set of assert methods Test fails if the assert condition does not hold If a test fails it is counted and reported

junit.framework.TestSuite A collection of tests Uses Java introspection to find all the methods start with "test“ and

have void parameters. TestSuite's run method executes all the tests

Åben Uddannelse


Test i XP 15

Test Runners

Classes that run tests for a test case Can be Text or UI based Text Based:

Accepts the name of a TestCase class as an argument If this class defines a static suite method it will be invoked

and the returned test is run. Otherwise all the methodsstarting with "test" having no arguments are run.

AWT UI Based: A handy tool for running test cases Runs test cases and reports failures and errors

Test i XP 16

Class diagram

Åben Uddannelse


Test i XP 17

JUnit How-To (1) Define a test case (of course) Create a Class which extends TestClass Define the fixture for the test class, these are all

variables required to carry out the tests Override the setUp and tearDown methods of

TestCase to initialize members or free upresources, respectively

Test i XP 18

JUnit How-To (2) Define the test methods required for the test case.

Additional methods may be added at any time Define a suite method which creates a TestSuite

from the methods of the test class Define a main method which uses some test runner

to run the tests in the TestSuite Simply run the test case class

Åben Uddannelse


Test i XP 19

JUnit How-To (3) Multiple test cases my be run by creating a class

which contains a suite of TestCase subclasses These suites can be run in batch or individually by

simply running the whole class containing the suiteor by calling run with the name of the test case youwant to run. Each test case constructor can accepta string which is the name of the test case

Test i XP 20

Junit User Interface

Åben Uddannelse


Test i XP 21

A Quick Example (1) Class Person Person contains a first and a last name Also contains the method hashCode methods for computing hashcodes

for first and last

public class Person {public Person() { first = ""; last = ""; }...public int hashCode() { int result = 0;

result += first.hashCode();result += last.hashCode();return result; }

...}

Test i XP 22

A Quick Example (2) Class PersonTest extends the

Junit class TestCase The fields first and last make up

the fixture PersonTest contains a test

which test if the hashCodemethod returns the proper values

The method suite is inheritedfrom TestCase

Åben Uddannelse


Test i XP 23

A Quick Example (3)

The PersonTest Constructor

public PersonTest(String name) {

super(name);

}

Accepts the name of the test case and passes it to the superclass

Test i XP 24

A Quick Example (4)

SetUp of one PersonTest

private static final Person personThree =

new Person("First3", "Last3");

Åben Uddannelse


Test i XP 25

A Quick Example (5)The PersonTest test method testHashCode

public void testHashCode() {int code = personThree.hashCode();

Person test = new Person("First3","Last3");assertEquals(code,test.hashCode());

test = new Person( "First1", "Last1");assertTrue((code == test.hashCode()) == false);……}

Test i XP 26

A Quick Example (6)The PersonTest suite method

public static Test suite ( ) {

return new TestSuite (PersonTest.class));

}

Creates a new TestSuite containing the tests definedin the PersonTest class

Åben Uddannelse


Test i XP 27

A Quick Example (7)Run multiple test suites (class AllTests) Define new suite - TestSuite Add suites to TestSuite

public static Test suite ( ) {

TestSuite suite= new TestSuite("All JUnit Tests");

suite.addTest(new TestSuite(PersonTest.class));suite.addTest(new TestSuite(LinkedListTest.class));

return suite;

}

Test i XP 28

A Quick Example (8)The public class AllTests

public static void main (String[] args) {

/** junit.textui.TestRunner.run (suite());*/

junit.swingui.TestRunner.run (AllTests.class); }

Creates a new TestSuite that runs all the tests definedin the PersonTest and LinkedListTest classes

Documents

Chapter 14 Software Testing Techniques