Ch6: Software Verification

2

Decision table based testing

Applicability: Spec. is described by a decision table.

Tables describe: How combinations of inputs generate different

outputs. Advantages:

Tables are easy to understand Supports a systematic derivation of tests

3

Decision table based testing (contd..)

“The word-processor may present portions of text in three different formats: plain text (p), boldface (b), italics (i). The following commands may be applied to each portion of text: make text plain (P), make boldface (B), make italics (I), emphasize (E), super emphasize (SE). Commands are available to dynamically set E to mean either B or I (we denote such commands as E=B and E=I, respectively.) Similarly, SE can be dynamically set to mean either B (command SE=B) or I (command SE=I), or B and I (command SE=B+I.)”

Example

4

Decision table based testing (contd..)

P

B

I

E

SE

E = B

E = I

SE = I

SE = B

SE = B + I

p b i b i b i b,i b,iaction

*

*

*

* *

*

*

* * *

*

*

*

*

*

5

Cause effect graphs

Based on a formal way of structuring complex input-output specifications

Transformation of inputs and outputs: Transform into Boolean values. Program transformation converted into a Boolean

function.

6

Cause effect graphs (contd..)

B I P E E = B SE E = I SE = B SE = I SE = B + I

b

i

p

A N D

O R

A N DO

R

The AND/OR graph represents the correspondence betweencauses and effects

7

Cause effect graphs (contd..)

Constraints may be added to describe legal output

a

b

c

e

a

b

c

o

a

b

r

a

b

m

at mostone

a

b

c

iat leastone

one and onlyone

requires masks

8

Cause effect graphs (contd..)

“Both B and I exclude P (i.e., one cannot ask both for plain text and, say, italics for the same portion of text.) E and SE are mutually exclusive.”

B I P E E = B SE E = I SE = B SE = I SE = B + I

m

m b

i

p

m m

A N D

O R

A N DO

R

9

Cause effect graphs (contd..)

Complete coverage principle: Generate all possible combinations of inputs & check if o/p occurs according

to specs. Consistency & completeness: o/p corresponding to i/p violates

compatibility, or no i/p comb. Generates o/p. May reduce the number by going backwards from outputs

OR node with true output:

• Generate only one test case with at least one input true.

• Output will be true if at least one variable is true. AND node with false output:

• Use only one combination with one false output. May highlight when admissible input violates graph’s consistency requirements May show incompleteness

10

Testing boundary conditions

Input domain partitioning: Partition i/p domain in classes, assuming that the

behavior is similar for data within a class. Some typical programming errors occur at the boundary

between different classes X < y, x <=y.

Suggestion: Test using values both within & at the boundaries.

Applies to both white and black-box testing

11

The oracle problem

How to inspect the results of test executions to reveal failures? Oracles are reqd. at each stage of testing. Automated test oracles for running a large # of tests. Oracles are difficult to design – no universal recipe.

12

Testing in the large: Testing and Modularity

How to test large software systems? If we apply the same techniques we used to test

modules to test systems, extremely complex. Try enumerating paths through the whole system.

Software architecture of the system may be used to drive verification. Test modules individually and then the whole

system. Benefits of modular testing:

Easier to localize errors, discover errors early, classify errors as to their scope.

Good design techniques enhance verifiability.

13

Testing in the large: Testing and modularity

Module testing: Verifies if a given module has been implemented correctly according to expected ext. behavior.

Integration testing: Testing that occurs as the system is being gradually integrated.

System testing: Test the entire system, as a collection of all the modules, usually in the delivery environment.

Acceptance testing: System testing by a customer.

14

Module testing

Modules often cannot be tested in isolation May need to provide a temporary context in which the

module might execute: Any modules it calls. Any non-local data structures it accesses. Simulate self-calls.

15

Module testing (contd..)

Stub: Module may call other modules. Procedure that emulates the behavior,Has the same i/o behavior as the called module, but simple impl. May be a Look-up table. Driver: Module may get called by other modules, piece of code that simulatesthe behavior of the calling module.

PROCEDURE UNDER TEST DRIVERSTUB

CALL CALL

ACCESS TO NONLOCAL VARIABLES

Harness or scaffolding: Support s/w needed for testing.

16

Integration testing

Big-bang testing: test individual modules & then system: No system testing, all inter-module dependencies resolved

during testing, many interactions may lead to high complexity.

Incremental testing: Apply incrementality principle to integration testing: Modules are developed and tested incrementally. Easier to localize errors. May identify collection of modules as autonomous

subsystems. Reduces the need for stubs & drivers for module

testing. Can be bottom-up, top-down.

17

Integration testing (contd..)

Bottom-up aggregation: Start aggregation and testing via USES hierarchy. Need drivers to emulate calls.

Top-down aggregation: Start from the top-level modules. Use stubs to simulate lower level modules.

Top-down and bottom-up approaches may be used in conjunction with each other

18

Integration testing (contd..)

A

B C

D E

If integration and testproceed bottom-uponly need drivers

Otherwise, if we proceedtop-down only stubs areneeded

19

Integration testing (contd..)

M1 M2

2,1 2,2M M

M1 USES M2 and M2 IS_COMPOSED_OF {M2,1, M2,2}

CASE 1Test M1, providing a stub for M2 and a driver for M1 Then provide an implementation for M2,1 and a stub for M2,2

CASE 2Implement M2,2 and test it by using a driver, Implement M2,1 and test the combination of M2,1 and M2,2 (i.e., M2) by using a driverFinally, implement M1 and test it with M2, using a driver for M1

20

Separation of concerns in testing

In general, testing involves several phases, with different goals, performed by different people

Use principle of separation of concerns to design test cases for different qualities: Performance, robustness, user friendliness.

A sample of different concerns Overload testing: Test behavior under peak conditions. Robustness: Test under unexpected conditions, erroneous

user commands, power failure. Regression testing: Test whether correctness & other

qualities are maintained after mods.

21

Testing object-oriented systems

General testing principles may be applied to object-oriented systems

Classes are similar to components in the traditional approach

Testing in the small: Add a few methods for testing. Other classes may be

stubbed in. Testing in the large:

Inheritance, interactions, public interface, private implementation.

22

Testing object-oriented systems (contd..)

New issues Inheritance Genericity Polymorphism Dynamic binding

Open problems still exist

23

Testing object-oriented systems (contd..)

“Flattening” the whole hierarchy and considering every class as a totally independent component

Finding an ad-hoc way to take advantage of the hierarchy

A sample strategy: A test that does not have to be repeated for any child class A test that must be performed for child class X and all of its

further children A test that must be redone by applying the same input

data, but verifying that the output is not (or is) changed A test that must be modified by adding other input

parameters and verifying that the output changes accordingly

24

Testing object-oriented systems (contd..)

Black-box testing: Test if class provides the functionality, ignore

implementation. Essentially testing of public interface.

Complete coverage principle – test each method. White-box testing:

Test hidden implementation. Ignores if public implementation. Apply coverage criteria to test cases.

Must perform both black and white box testing