COM3082C2 - Systems Development slide 6- 1

COM3082C2 - Systems Development slide 6- 1

Software Testing

Dr Z He

University of Ulster


Lecture 6: Software Testing Objectives

» Issues of software testing

– Software testing, Software testing principles, Need for testing

» Verification and validation

» Testing stages

– unit, integration, system, acceptance

» Test Planning

– Test plan

» Techniques for testing

– static: reading, walkthrough and inspections, correctness proofs, stepwise abstraction

– dynamic: functional, structural

» Testing Coverage

» Further Testing Stages


Software Testing» once source code has been generated, software must be tested to

uncover(and correct) as many errors as possible before delivery to the customer

» the goal is to design a series of test cases that have a high likelyhood of finding errors

» software testing techniques provide systematic guidance for designing tests that

– exercise the internal logic of software components, and

– exercise the input and output domains of the program to uncover errors in program function, behaviour and performance

» is a process of executing a program with intent of finding an error

» a good test case is one that has a high probability of finding an uncovered error

» a successful test is one that uncovers an undiscovered error

» cannot show the absence of errors and defects, it can only show that software error and defect are present


Software Testing Principles

All tests should be traceable to customer requirements Tests should be planned before testing begins. Test

planning can begin as soon as the requirements model is complete. Detailed definition of test cases can begin as soon as the design model has been solidified

Pareto principle applies to software testing (80% of all errors uncovered during testing will likely to be traceable to 20% of all program components)

Testing should begin in small (components) and progress toward testing in large(ultimately the entire system)

Exhaustive testing is not possible To be most effective, testing should be conducted by an

independent third party


Need for Testing 30-85 errors per 1000 lines of code (Boehm 1981) After testing, 0.5-3 errors per 1000 lines of code

(Myers 1986) Errors are expensive The later errors are found the more expensive they

are to fix

Costto fix

Requirements Operation

Phase


Need for Testing

Testing shows presence of errors but cannot prove their absence

In most cases it is impossible to test exhaustively E.g.

for i from 1 to 100 do

print (if a[i] = true then 1 else 0 endif);

has 2100 different outcomes - approx. 3x1014 years to exhaustively test. Given a machine can execute 10 million print instructions per second.


Verification and Validation

Verification - ‘Are we building the product right’?

Validation - ‘Are we building the right product’?

1. At the requirements analysis stage cheapest to correct at this stage Test criteria:

» completeness - are all the requirements gathered

» consistency - no contradictions within the system or with external components

» feasibility - benefits should outweigh costs

» testability - requirements should be unambiguous, otherwise they cannot be tested



2. At the Design stage criteria as for requirements stage. Elements from the requirements should be traced to the

design simulation, walkthroughs, inspections to test quality



3. Implementation stage Programs can be checked by reading through them (not by

the author) Stepwise abstraction - determine the function of code from

a number of its steps Tools are available to support the testing of code

» Static analysis - test the code without running it - E.g. have all the variables been declared?

» Dynamic analysis - test by executing the code


Unit testing

» testing of individual components Integration testing

» testing collections of modules integrated into sub-systems System testing

» testing the complete system prior to delivery Acceptance testing

» testing by users to check that the system satisfies requirements. Sometimes called alpha() testing

Testing Stages


Testing Stages

Sub-systemtesting

Unittesting

Systemtesting

Acceptancetesting


Test Plan

»The testing process

»Requirements traceability

»Tested items

»Testing schedule

»Test recording procedures

»Hardware and software requirements

»Constraints

Test Planning


Techniques for Testing - Static1. Reading, Walkthroughs and inspections All involve someone else looking at the code Look for

» inappropriate usage of data. E.g. uninitialised variables, arrays out of bounds, dangling pointers

» errors in declarations. E.g. undeclared variables, repeated names

» Faults in computations. E.g. division by zero, overflow, type mismatches, erroneous operator order (*,/,+,- etc.)

» faults in logical operators. E.g. < instead of >

» faults in control. E.g. infinite loops, loop executing the wrong number of times

» faults in interfaces. E.g. Wrong number of parameters

Many of these are detected by the compiler Can be applied to documents Can be applied to all stages of life cycle


Techniques for Testing - Static2. Correctness Proofs Does a program meet its specification. If specification is expressed formally and proved the

program can be proved Difficult to use. Cannot prove every aspect. Validation can only be done by testing.

3. Stepwise Abstraction bottom-up process start with code and derive functions check these with the requirements


Techniques for Testing - Dynamic

Dynamic Testing Techniques» Black Box Testing

– also called functional testing, or behavioural testing

– focuses on the functional requirements of the software

– enables the software engineer to derive sets of input conditions that will fully exercise all functional requirements for a program (test cases derived from specifications)

– tends to be applied during later stages of testing

– purposely disregards control structure, attention is focused on the information domain

– exhaustive functional testing is normally not feasible

– use characteristics of input domain to find the minimum set of test cases to reveal most defects

– we will study: Equivalence Partitioning Boundary value analysis



Dynamic Testing Techniques» White Box Testing

– also called glass box testing, or structural testing

– uses the control structure of the procedural design to derive test cases

– exhaustive structural testing is normally not feasible

– to find the minimum set of test cases to exercise most important logical paths

– We will study: Basis Path Testing Loop Testing Testing Coverage



Equivalence Partitioning » a black box /functional testing technique

» in determining a set of test cases, this technique is to partition the program domain into a (small) number of equivalent classes

» the assumption is that if a representative element from a class is tested and passed, all other elements in the class need not to be tested assuming they would pass the test

» How to define equivalent classes:

– if a input condition specifies a range, one valid and two invalid equivalent classes are defined

– if an input condition requires a specific value, one valid and two invalid equivalent classes are defined

– if an input condition specifies a member of a set, one valid and one invalid equivalent classes are defined

– if an input condition is Boolean, one valid and one invalid class are defined



Equivalence Partitioning » Example 1: Online banking

– The user can access the bank using his personal computer, providing a 6-digit password, and follow with a series of typed commands that trigger various banking functions. The software supplied for baking application accepts data in the form:

area code: blank or three digit number not beginning with 0 or 1 password: six digit alphanumeric string commands: “check”, “deposit”, or “bill pay”

– Equivalent classes can be identified for each field area code(boolean, range, specific value):

»ec1: blank; ec2: 3-digit numbers (<200); ec3: 3-digit numbers (200-999); ec4: less than 3-digit numbers; ec5: more than 3-digit numbers (at least one value is needed for test case design)



Equivalence Partitioning password (boolean, specific value, set):

»ec6: blank; ec7: 6-digit alphanumeric strings(valid passwords); ec8: 6-digit alphanumeric strings(invalid passwords); ec9: less than 6-digit alphanumeric strings; ec10: more than 6-digit alphanumeric strings

commands(boolean, set):

»ec10: blank; ec11: valid (“check”, “deposit”, “bill pay”); ec12: invalid

» Example 2: an input condition “each book has a 6-digit number” (specific value) can be classified as either valid or invalid equivalent classes

– one valid class: 6-digit numbers

– two invalid classes: more or less than 6 digits

– test cases can be constructed to cover the three equivalent classes

» Weak point: partitioning may be inadequate so that the assumption may not hold



Boundary Value Analysis (BVA)» complements the equivalent class partitioning technique

» Assumes software usually fails at boundaries.

» concentrate on the extreme (boundary) values from each class (rather than random values from each class)

» BVA leads to a selection of test cases that exercise bounding values

» output can also be partitioned into equivalent classes, and extreme values from each class are tested as well

» Guidelines for BVA:

– if an input condition specifies a range bounded by value a and b, test cases should be designed with values a and b, just above and just below a and b, respectively (a, b, --a, a++, --b, b++)

– if an input condition specifies a number of values, test cases should be developed that exercise the minimum and maximum numbers. Values just above and below minimum and maximum are also tested



Boundary Value Analysis (BVA)– Apply guidelines 1 and 2 to output conditions. For example, assume

that a temperature vs. pressure table is required as output from an engineering analysis program. Test cases should be designed to create an output report that produces the maximum (and minimum) allowable number of table entries.

– If internal program data structures have prescribed boundaries (e.g., an array has a defined limit of 100 entries), be certain to design a test case to exercise that data structure at its boundary.



Basis Path Testing» a white-box testing technique, proposed by Tom McCabe, 1976

» to derive a logical complexity measure of a procedural design, and use this measure as a guide for defining a basis set of execution paths

» test cases derived to exercise every statement and branch in the program at least once during testing (statement/branch coverage)

» if every condition in a compound condition is considered, condition coverage can be achieved

» Steps:

– Draw a (control) flow graph, using the flowchart or the code

– Calculate the cyclomatic complexity, using the flow graph

– Determine the basis set of linearly independent paths

– Design test cases to exercise each path in the basis set


Basis Path Testing Flow Graph

» used to depict program control structure

» can be drawn from a flowchart (a procedural design representation)

» can be drawn from a piece of source code

» Flow Graph Notation

– a flow graph composed of edges and nodes

– an edge starts from a node and ends to another node

Sequence if-then-else While Repeat-until Case

...



» Draw a flow graph from source code

1 procedure insert(a, b, n, x)

2 begin bool found:=false;

3 for I:=1 to n do

4 if a[I]=x

5 then found:=true; goto leave endif

6 enddo;

7 leave:

8 if found

9 then b[I]:=b[I]+1

10 else n:=n+1; a[n]:=x; b[n]:=1 endif

11 end insert

1, 2

3

64

5

7

8

9 10

11



» Draw a flow graph from a flowchart

h

gf

e

d

c

b

ainsert

X found?

found=true

x found?

Save xCount-x =1

Yes No

end

Count-x +1

All checked?

found=false

NoNo

Yes

Yes


Basis Path Testing Cyclomatic Complexity

» a software metric that provides a quantitative measure of the logical complexity of a program

» Basis set: is a maximal linearly independent set of paths through a graph

» An independent path: is any path through a program that introduces at least one new set of processing statements or a new condition (I.e. at least one new edge in a flow graph)

» Cyclomatic complexity defines the number of independent path in the basis set of a program

» gives an upper bound for the number of tests that must be conducted to achieve statement/branch/condition coverage

» How to calculate cyclomatic complexity:

cc = e - n + 2p» e - number of edges; n - number of nodes; p - number of components;

» if all nodes in a graph are connected, then p = 1, thus

cc = e - n + 2


Basis Path Testing: Example 11. Draw a flow graph

0 /*Finding the maximum of three integers*/

1 #include <stdio.h>

2 intmaximum(int, int, int);

3 main(){

4 int a,b,c;

5 printf((“Enter three integers: “);

6 scanf(“%d%d%d”, &a,&b,&c);

7 printf(“Maximum is: %d\n”, maximum(a,b,c));

8 }

9 int maximum(int x, int y, int z){

10 int max=x;

11 if(y>max)

12 max=y;

13 if(z>max)

14 max=z;

15 return max

16 }16

15

14

13

12

11

10

9

f

e

d

c

b

a

==>


Basis Path Testing: Example 1

2. Calculate cyclomatic complexity» e=7, n=6, p=1; so that cc=7-6+2=3

3. Identify a basis set of independent paths» p1: a-b-c-d-e-f (y>x, z>y)

» p2: a-b-d-e-f (y<=x, z>x)

» p3: a-b-c-d-f (y>x, z<=y)

4. Design test casesTestcase

Test data(x,y,z)

Expectedresult

Actualresult

Pass/fail

p1 2, 3, 4 4

p2 3, 1, 6 6

p3 5, 7, 3 7


Basis Path Testing: Example 2

1. Draw a flow graph» see slide 6-24: source code, flow graph

2. Calculate cyclomatic complexity» e = 12; n = 10; p = 1

» cc = 12 - 10 + 2 x 1 =4

3. Determine a basis set of independent paths

» expect to specify 4 independent paths

» p1: 1-2-3-7-8-9-11

» p2: 1-2-3-4-5-7-8-9-11

» p3: 1-2-3-4-5-7-8-10-11

» p4: 1-2-3-4-6-3-7-8-10-11 (1 or more times)

» HOWEVER: by reading source code, we found

– 3-7 => 10; 5 => 9

– p1 and p3 must be modified

1, 2

3

64

5

7

8

9 10

11


Basis Path Testing: Example 2 3. Determine a basis set of independent paths

– if p3 modified, it would be the same as p2. Thus p3 should be deleted.

– But the new paths introduced by p3 (8-10-11) must be covered by other paths! We found p4 covers them.

– Modify p1, delete p3, we can have three independent paths p1: 1-2-3-7-8-10-11 p2: 1-2-3-4-5-7-8-9-11 p3: 1-2-3-4-6-3-7-8-10-11

– if you study the program carefully, you will find the following is better p1: 1-2-3-7-8-10-11 (insert x when a[] is empty) p2: 1-2-3-4-5-7-8-9-11(insert x when a[1]=x) p3: 1-2-3-4-6-3-4-5-7-8-9-11 (insert x when a[i]=x,i>1, n>=i) p4: 1-2-3-4-6-3-7-8-10-11 (insert x when a[] is not empty and x is

not in a[]; p4 does not introduce any new edge but it exercises a new combination of the program logic!)


Basis Path Testing: Example 2 4. Design test cases

» Path 1 test case: 1-2-3-7-8-10-11 (insert x when a[] is empty)

– input data: n=0; x=8; a[1]=0; b[1]=0;

– expected results: a[1]=8; b[1]=1; n=1;

» Path 2 test case: 1-2-3-4-5-7-8-9-11(insert x when a[1]=x)

– input data: n=3; x=9; a[1]=9; a[2]=2; a[3]=3;b[1]=2;b[2]=5;b[3]=8;

– expected results: b[1]=3

» Path 3 test case: 1-2-3-4-6-3-4-5-7-8-9-11 (insert x when a[i]=x,i>1, n>=i)

– input data: n=3;x=3;a[1]=9;a[2]=2;a[3]=3;b[1]=3;b[2]=2;b[3]=8;

– expected results: b[3]=9

» Path 4 test case: 1-2-3-4-6-3-7-8-10-11 (insert x when a[] is not empty and x is not in a[])

– input data: n=3;x=6;a[1]=9;a[2]=2;a[3]=3;b[1]=3;b[2]=2;b[3]=8;

– expected results: a[4]=6; b[4]=1; n=4;



Loop Testing» a white box/structural testing technique

» focuses exclusively on the validity of loop constructs

» four different classes of loops can be defined:

simple loops nested loops concatenated loops unconstructed loops



Loop Testing» simple loops

– if n is the maximum number of allowable passes through the loop

– the following set of test cases (7 cases) can be applied skip the loop entirely (0) only one pass through the loop (1) two passes through the loop (2) m passes through the loop where m<n (m) n-1, n, n+1 passes through the loop (n-1, n, n+1)

– Class Exercise use the example for the basis path testing to derive 7 test cases for real testing, you need to write a program to test the function



Loop Testing» nested loops

– start at the innermost loop. Set all other loops to minimum values

– conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter values.

– Work outward, conducting tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values

– continue until all loops have been tested

» concatenated loops

– if concatenated loops are independent, use the approach defined for simple loops

– if not independent(the loop counter for loop 1 is used as the initial value for loop 2), then the approach applied to nested loops is recommended



Testing Coverage» Statement coverage - Run a series of tests which ensure that every

statement is tested at least once. A tool is used to keep track of how many times a statement has been executed. BUT - not every situation is dealt with!

» Branch coverage - Run a series of tests that ensure that every possible branch is tested at least once. Again a tool is required.

» Condition coverage - Run a series of tests that ensure that every possible outcome of each condition is tested at least once.

» Path Coverage - Run a series of tests that cover all possible combination of paths. This may be impossible so paths have to be selected

» All-definition-use-path coverage - Create a test case for all paths that lead from a definition e.g. x=1 to a variable x = x +1.


Further Test Stages

So far we have dealt with module testing or unit testing Integration Testing

» Modules need to be tested when combined

» Start with low level modules and move up

– difficult to get an impression of the overall system

» Start at the top and work down to subsystems

– may prove repetitive and laborious

» A combination of both approaches can be used System Test: Testing the whole system

Acceptance Test: Testing the usability

Installation Test: If the operating environment is different from the development environment

Documents

COM3082C2 - Systems Development slide 6- 1