Dynamic Testing

1

Dynamic Testing

Test Design Techniques

2

Dynamic Testing

• Testing that involves the execution of the software of the component or system

• Execution of the test object on a computer• Need a test bed

3

Test Bed

Test Object

Stub1

Stub 2

Stub k

Test stubs

Run time environment,Analysis tools, monitors

TestCase 1

TestCase 2

TestCase n

TestDriver

Testoutput

comparison

PoC

PoO

PoC – Point of Control PoO – Point of Observation

4

Incremental approach to execute tests

• Steps are:– Determine conditions and preconditions for the test and the

goals that are to be achieved– Specify individual test cases– Determine how to execute the tests (usually chaining together

several test cases)• Eg: group test cases in such a way that a whole sequence of

test cases is executed (test sequence or test scenario)• Document it in test procedure specification• Need test script, e.g., JUnit

5

Techniques for Testing

• They are test case design techniques• Different techniques are:

– Black Box Testing– White Box or Glass Box or Open Box Testing

6

Black Box Testing Techniques(1)

• Test object is treated as a black box• The inner structure and design of the test object is unknown• Test cases are derived/design using the specification or the

requirements of the test object• The behavior of the test object is watched from the outside (PoO is

outside the test object)• It is not possible to control the operating sequence of the object

other than choosing the adequate input test data (PoC is situated outside of test object)

• Used for higher levels of testing• Any test design before the code is written (test-first programming,

test-driven development) is black box driven

7

Black Box Testing Techniques(2)

Poc and PoO “outside” the test object

Test Object

TestOutputdata

PoC

PoO

Test input data

8

Black Box Testing Techniques: Equivalence Class Partitioning Method(1)

• The domain of possible input data for each input data element is divided in equivalence classes

• An equivalence class is a group of data values where the tester assumes that the test object processes them the same way– the behavior of the component or system is assumed to be the

same– Test of one representative of the equivalence class is seen as

sufficient

• test equivalence class for correct input as well as incorrect input

9

Black Box Testing Techniques: Equivalence Class Partitioning Method(2)• Example of correct (valid)

equivalence class vEC

Parameter Equivalence Class Representative values

Bonus calculation program, duration of employment in years

vEC1 : 0 <= x <= 3

vEC2 : 3 < x <= 5

vEC3 : 5 < x <= 8

vEC4 : x > 8

2

4

7

12

10

Black Box Testing Techniques: Equivalence Class Partitioning Method(3)• Example of incorrect (invalid)

equivalence class iEC

Parameter Equivalence Class Representative values

Bonus calculation program, duration of employment in years

iEC1 : x < 0(“negative” – thus incorrect – staff membership in a company)

iEC2 : x > 40(unrealistically long and incorrect staff membership in a company

-3

45

11


Systematic Derivation of the Test Cases• Determine the domain of all possible inputs for every input data element

– E.g., function/method parameter at component tests or input screen field at system test

• Partition the domain into equivalence classes– First the subdomain of correct inputs

• The test object should process these according to specification– Then the values outside this domain as equivalence classes for invalid values

• Observe how the test object behaves• Refine the equivalence classes

– The equivalence classes should be divided until each different requirement corresponds to an equivalence class

• Choose a representative value for every single equivalence class for testing• Define the preconditions and the expected result for every test case

12


• the same principle of dividing into equivalence classes can also be used for the output data

• Identification of the individual test case is more expensive, because for every output-representative the corresponding input value combination causing this output must be determined

• Regard also the output values for the equivalence classes with incorrect values

13


Boundaries of the equivalence classes• Partitioning into equivalence classes and selecting the

representative values should be carefully done• Probability of failure detection is highly dependent upon the quality

of the partitioning, as well as which test cases are executed• The best values are those verifying the boundaries of the

equivalence classes– Often misunderstanding or inaccuracies in the requirements at

these spots– E.g., “… more than 32 credits…” might mean

• The value 32 being inside equivalence class (EC : x >= 32) or• Outside equivalence class (EC : x > 32)• Additional test case with x=32 might detect a misunderstanding

14


Equivalence class construction for integer values(1)• All possible classes are to be identified• Example : based on integer parameter extras of the

function calculate_price()– the domain of inputs partitioned into equivalence classes

Parameter Equivalence classes

extras vEC1 : [MIN_INT,…,MAX_INT]iEC1 : Not a Number

Incorrect values:Numbers that are greater orSmaller than the range of theApplicable interval or everyNonnumeric value includingFloating point numbers

15


Equivalence class construction for integer values(2)• Further divide the equivalence classes with correct values because

negative and positive values often must be treated differently• Zero is a further input value that often lead to failures• Refinement of Equivalence classes and representatives

• Result: the equivalence class method, taking into account boundary values, generate the following test values:(“f”, MIN_INT, -123, -1, 0, 654 MAX_INT)

Parameter Equivalence classes Representativesextras vEC1 : [MIN_INT,…,-1]

vEC2 : [0,…,MAX_INT]iEC1 : Not a Number

-123654“f”

16


Guidelines for Determining Equivalence class(1) For the input as well as for the outputs, identify the restrictions and

conditions in the specification For every restriction or condition, partition into equivalence classes:

• If a continuous numerical domain is specified, then create one valid and two invalid ECs

• E.g., “the item count can be from 1 to 999”, identify one vEC (1 < item < 999) and two iEC (item count < 1 and item count > 999)

• If a number of values should be entered, then create one valid (with all possible correct values) and two invalid ECs (less and more than)

• E.g., “one through six owners can be listed for the car”, identify one vEC and two iEC (no owner and more than six owners)

17


Guidelines for Determining Equivalence class(2) For every restriction or condition, partition into equivalence classes:

• If a set of values is specified where each value may possibly be treated differently, then create one vEC for each value of the set (containing exactly this one value) and one additional iEC (containing all possible other values)

• E.g., “type of vehicle must be BUS, TRUCK, TAXI, MOTORCYCLE”, identify one vEC from the set and one iEC (“BICYCLE”)

• If there is a condition that must be fulfilled, then create one vEC and one iEC

• E.g., “first character of the identifier must be a letter”, identify one vEC (it is a letter) and one iEC (it is not a letter)

If there is any doubt that the values of one EC are treated equally, then the EC should be further divided into subclasses

18


Test Cases (1)• Combination of the representatives

– Rules for test case determination• The representative values of all vECs should be combined to

test cases, meaning that all possible combinations of vECs will be covered. Those combinations build “valid test case” or “positive test case”

– Separate test for invalid value• The representative value of an iEC can only be combined with

representatives of other vECs. Thus every iEC leads to an additional “invalid test case” or “negative test case”.

– If a test case combines more than one incorrect value, defect masking may result

– Thus only one of the possible exceptions is actually triggered and tested

19

Black Box Testing Techniques: Equivalence Class Partitioning Method(11)Test Cases (2)• Restriction of the number of test cases

– Reducing the number of valid test cases as the result of multiplicative combination of representatives

• Rules for test case restriction Combine the test cases and sort them by frequency of occurrence.

Prioritize the test cases in this order. That way only relevant test cases are tested

Test cases including boundary values or boundary value combinations are preferred

Combine every representative of one EC with every representative of other EC (dual combination)

Ensure as minimum criteria that every representative of an EC appears in at least one test case

Representatives of iECs should not be combined with representatives of other iECs

20


Test Completion Criteria• Define as the percentage of executed ECs in comparison

to the total number of specified ECEC-coverage = (number of tested EC / total number of EC) * 100%

• Test coverage is a measurable criterion for finishing testing

21


The value of ECP• Contribute to a complete test where specified conditions and

restrictions are not overlooked• Minimize the generation of unnecessary test cases• EC can be determined by for input and output• Can also be prepared for internal values and states, time dependent

values, and interface parameters• Can be used in system testing, integration testing, and component

testing• Only single input or output conditions are considered, while possible

dependencies or interactions between conditions are ignored• May not consider boundary problems

22

Black Box Testing Techniques: Boundary Value Analysis (BVA) (1)

• Faults often appear at the boundaries of ECs or boundary conditions

• This is because of misunderstanding or unclear definition• Boundary conditions are those situations directly on,

above, and beneath the edges of input ECs and output ECs

• BVA differs with ECP in two aspects:– Rather than selecting any element in an EC as representative,

BVA requires the one or more elements be selected such that each edge of the EC is the subject of a test

– Rather than just focusing attention on the input conditions, test cases are also derived by considering the output ECs

23


Guidelines for BVA• If an input condition specifies a range of values, write

test cases for the ends of the range, and invalid-input test cases for situations just beyond the ends.– E.g., if the valid domain is [-1.0,…,+1.0], write the test cases for

the situations -1.0, +1.0, -1.001, +1.001

• If an input condition specifies a number of values, write test cases for the minimum and maximum number of values and one beneath and beyond these values– E.g., if an input file can contain 1-255 records, write test cases

for 0, 1, 255, 256

24

Black Box Testing Techniques: Boundary Value Analysis (BVA) (3)Guidelines for BVA• If an output domain serve as the basis, then the analysis can be

done as follows:– The output of the test object is an integer value between 500 and 1000.

Test output that should be achieved: 500, 1000, 499, 1001• If the permitted number of output values is to be tested, proceed just

as with the number of input values: if outputs of 1 to 4 data values are allowed, the test outputs to produce are: 1, 4 as well as 0 and 5 data values

• If the input or output of a program is an ordered set (a sequential file, a linear list or a table), focus attention on the first and last elements of the set

• If complex data structures are given as input or output, an empty list or zero matrixes can be considered a boundary value

25


Guidelines for BVA• For numeric calculations, values that are close together,

as well as values that are far apart, should be taken into consideration as boundary values

• For iECs, BVA is only useful when different exception handling for the test object is expected depending on an EC boundary

• Extremely large data structures, lists, tables, etc. should be chosen. For example, those that exceed buffer, file, or data storage boundaries; to check behavior in extreme cases

26


Test Cases• The valid boundaries inside an EC may be combined as

test cases• The invalid boundaries must be verified separately and

cannot be combined with other invalid boundaries

27


Test Completion Criteria

BV-Coverage = (number of tested BV/total number of BV) *100%

The Value of the Technique• Should be done together with ECP• Require a lot of creativity to define test data at boundaries

28

Black Box Testing Techniques: State Transition Testing(1)• Base on history of execution or event or input, influences

of the output and how the test object behaves• Test object starts from an initial state and then come into

different states• Events trigger state transitions

– Event is normally a function invocation• State transitions can involve actions• The test object can be a complex system with different

system states, as well as a class in OOS with different states

• Finite state machine, state diagrams, or state transition tables model this behavior

29

Black Box Testing Techniques: State Transition Testing(2)

• Example: Stack

empty filled full

initialize

delete

push

Pop [height = 1] pop

Push[height = max – 1]

TopPop [height > 1]Push [height < max -1] Push*

top

30


• A possible test case for accepting stringsPrecondition: stack is initialized; state is “empty”Input: push (“hello”)Expected result: stack contains “hello”Post-condition: state of the stack is “filled”

• Further test cases for the stack example– Test case 1: initialize [empty], push [filled], push, push, push [full]– Test case 2: initialize [empty], push [filled], top, pop [empty], delete*note: not all states have been reached

31


Test criteria• Should execute all specified functions of a certain state at least once• The compliance between the specified and the actual behavior can

thus be checked.• To identify test cases

– The finite state machine is transformed into a transition tree which includes certain sequences of transitions

32


Building a transition tree1. The initial or start state is the root of the tree2. For every possible transition from the initial state to a following state

in a transition diagram, the transition tree receives a branch from its root to a node, representing this next state

3. The process for step 2 is repeated for every leaf in the tree (every newly added node) until one of the following two end-conditions is fulfilled:

• The corresponding state is already included in the tree on the way from the root to the node. This end-condition corresponds to one pass of a cycle in the transition diagram

• The corresponding state is a final state.

33


Building a transition tree (example)initial

empty

deletedfilled

empty filled full filled filled

full full filled

initialize

deletepush

pop

push push poppop

toppush*

pop

34


• For the example: there are 8 different paths. Each path represents a test case

• In addition, the reaction of the state machine for wrong usage must be checked– The functions are called in state in which they are not supposed

to be called (test of robustness)• E.g., to delete a stack while in “full” state

– So, extend the transition tree to include a branch for every function from every node

35


Building a transition tree (example)initial

empty

deletedfilled

empty filled full filled filled

full full filled

initialize

deletepush

pop

push push poppop

toppush*

pop

failure

failure

failure

failure

pop

topdelete

delete

36


• Also good for system testing• E.g., to test GUI by modeling it as a finite state machine

– Screen and user controls as states– Input reactions as state transition

• Appropriate test cases and test coverage can be identified from the model

37

Black Box Testing Techniques: State Transition Testing(10)Test Cases• The following information is necessary for a complete definition of a

state-based test case– The initial state– The input– The expected behavior– The expected final state

• Further, for each expected transition of the test case the following aspects must be defined– The state before the transition– The initiating event that triggers the transition– The expected reaction triggered by the transition– The net expected state

38


Test Completion Criteria• Every state has been reached at least once• Every transition has been executed at least once• Every transition violating the specification has been

checked• Percentages can be defined for the above

39


The value of this technique• Apply where states are important and where the

functionality is influenced by the state• State transition testing is important for object-oriented

testing– Takes into account the special aspects of object orientation

40

Black Box Testing Techniques: Cause-effect Graphing Technique(1)

• Previous techniques do not regard combinations of input circumstances

• Cause-effect graphing is a technique that uses the dependencies for identification of the test cases

• The logical relationships between causes and their effects in a component or a system are displayed in a cause-effect graph– Every cause is described as a condition that consists of input

conditions (or combinations of those) can be TRUE or FALSE– The conditions are connected with logical operators (e.g, AND,

OR and NOT)

41


Basic cause-effect graph symbols

a bidentify

If a is TRUE, b is TRUEelse b is FALSE

a bNOT

If a is TRUE, b is FALSEelse b is TRUE

a

b

c

d

a

b

c

OR

If a or b or c is TRUEd is TRUEelse d is FALSE

If a and b are TRUE,c is TRUEelse c is FALSE

AND

42

Black Box Testing Techniques: Cause-effect Graphing Technique(3) - example• withdrawing money at an ATM. In order to get the money

from the machine, the following conditions must be fulfilled:1. The bankcard is valid2. The PIN must be correctly entered3. The maximum number of PIN input is three4. There is money in the machine, and in account

• The following actions/effects are possible at the machine:10. Reject card11. Ask for another PIN input12. “eat” the card13. Ask for alternate dollar amount14. Pay the requested amount of money

43

Black Box Testing Techniques: Cause-effect Graphing Technique(4)- example

1

2

3

4

10

11

12

13

14

44

Black Box Testing Techniques: Cause-effect Graphing Technique(5)- example• The graph must be transformed to a decision table• A decision table has four quadrants

– The upper left is list of the inputs (causes)– The upper right is combinations of inputs– The lower left consists of the effects– The lower right is the expected effects

• The steps are:– Choose an effect– Looking in the graph, find combinations of causes that have the effect

and combinations that do not have this effect– Add one column into the table for every one of these cause-

combinations and the cause states of the remaining effects– Check if decision table entries occur several times and, if yes, delete

them

45

Black Box Testing Techniques: Cause-effect Graphing Technique(6)- exampleCondition/cause 1 2 3 4 5

1. Bankcard is valid N Y Y Y Y

2. PIN is correct - N N Y Y

3. 3 incorrect PIN - N(exit) Y - -

4. Money available - - - N Y

Effect/action

10. Reject card Y N N N N

11. Ask new PIN N Y N N N

12. Eat card N N Y N N

13. Ask new amount N N N Y N

14. Pay money N N N N Y

46


• Test cases– Every column is a test case

• The table defines the logical test cases• Must feed with concrete data values• Annotate necessary preconditions and post-conditions

• The value of the technique– May reveal combinations which are not included when using

other test case design techniques– However, errors can result from optimization of the table– Graph and table may grow very quickly and lose readability

when the number of conditions and dependent actions increases– Without support by tools, the technique is not easily applicable

Documents

Dynamic Testing