Testing (1)

Principles of testing

The testing process

Methods used in testing & debugging

Let’s look at :

Testing (2)

DesignAnalysis Code Test

Testing (3)

Narrow View (unit level)

Run a program component with the purpose of finding errors prior to delivery of product

Broad View

The process of ensuring that the software conformsto its specification and meets user requirements

Validation

A good test is one that has a high probability of finding a new errorA successful test is one that discovers a new error

Verification

Testing (4)

Testing Principles

Pareto Principle – 80% of errors occur in 20% of classes

Tests should be planned long before testing begins – test planning should really be done with program design

All tests should be traceable to user requirements

Testing should begin “in the small” and proceed towards testing “in the large”

Exhaustive (complete) testing is usually not possible

To be effective, testing should be conducted by an independent 3rd party

20% of the citizens own 80% of the wealth

20% of the land grows 80% of the crops

Testing (5)

Who tests the software?

Developer?

Independent tester?

Understands the system but will test It “gently” and be motivated by need to deliver the product

Needs to learn about the system but will attempt to break it/crash it and is driven by quality

Testing (6)

Features of Testable Software

Operability- The better it works the more easily it can be tested (Bugs are easier to find in software which at least executes)

Observability- The results of each test should be easy to observe

Controlability- If we can control the execution of a separate parts of the software then it will be easier to set up specific test cases and perhaps to automate testing

Simplicity- Simple system architectures are easier to test than complex ones

Stability- Changes disrupt test planning and test cases

Test case – see next slide

Testing (7)

A test case is a controlled experiment that tests all or part of the system with defined test data

Test process -

However, this is very often not the case and test design often done badly and in an ad hoc manner

Constraints – with a minimum of time & effort

Criteria – in a complete manner

Objective – to uncover errors

“Make it up as we go along”Remember that “bugs lurk in corners and congregate at boundaries!”

Why?

Testing (8)

Two nested loops containing four if .. then .. else statements. Each loop can execute up to 20 times.

There are 10^14 possible paths if we count each single iteration.

If we execute one test per millisecond, it would take 3170 years to test this program!

Exhaustive Testing (not feasible)

Testing (9)

Selective Testing (feasible)

Test a carefully selected execution path.

Note that it cannot be comprehensive

Testing (10)

Testing Methods

Black Box testing – examines fundamental interface without looking at internal processing.In short, is the program’s output correct for a givenset of inputs?

White (Glass) Box testing – examines in detail the internal processing done by software components

Debugging – fixes the errors identified during testing

?

?

??

?

?

Testing (11)

The objective in white box testing is to ensure that all statements and conditions have been executed at least once.

Derive test cases that:

1. Exercise all independent execution paths

2. Exercise all logical decisions on both the true and the false sides

3. Execute all loops at their boundaries and within operational bounds

4. Exercise all internal data structures to ensure they are valid and that read/write accesses are as they should be

Testing (12)

Why cover all paths?

Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed

We often may be inclined to believe that a logical path is not likely to be executed when, in fact, it may be frequently executed

Typographical errors occur at random and so it is likely that untested paths will contain some

Testing (13)

Basis Path Testing -

Provides a measure of the logical complexity of a method or code component and provides a guide for defining a basis set of execution paths

It uses flow graph notation to represent the flow of control where nodesrepresent processing and arrows represent control flow.

WhileIfSequence

Testing (14)

Flow Graphs – Compound Conditions

Separate nodes are created for each arm of a compound condition (e.g. a and b are separate nodes in the conditionif(a && b))

Example:

If(a || b){ x();}else{ y()}Z()

b

a

z

y x

Testing (15)

Cyclomatic Complexity is a software metric that gives a quantitative measure of the logical complexity of a program

Find the cyclomatic complexity, V(G), ofa flow graph G:

- Number of simple predicates (decisions) + 1 or

- V(G) = E – N+2 (where E are edges and N are nodes or

- Number of enclosed areas + 1

In this case, V(G) = 4

Testing (16)

Analysis has shown that the number of errors and the maintenance Increases significantly for modules with a V(G) > 10.

Another use of cyclomatic complexity is that V(G) identifies the number ofindependent paths though a program that need to be tested

Testing (17)

Basis Path testing

V(G) is the number of linearly independentpaths through the program (each has at least one edge not covered by any other path)

Path 1: 1-2-3-8

Path 2: 1-2-3-8-1-2-3-8

Path 3: 1-2-4-5-7-8

Path 4: 1-2-4-6-7-8

Test design must prepare test cases that willforce the execution of each path in the basis set

1

2

3

4

5 6

7

8

Testing (18)

Basis Path testing

Example:Draw the flow graph, calculate the cyclomatic complexity, and list the basis paths using the following C++ piece of code:

While(value[i]!= -999.0 && totinputs < 100{ totinputs++ if(value[i] >= min && value[i] <= max) { totvalid++ sum = sum +value[i] } i++}

While(value[i]!= -999.0 && totinputs < 100{ totinputs++ if(value[i] >= min && value[i] <= max) { totvalid++ sum = sum +value[i] } i++}

1 2

34

5

6

7

1

2 4

3

5

6

7

Testing (19)

Basis Path testing

1

2 4

3

5

6

7

V(G) = number of enclosed areas + 1 = 5

V(G) = number of simple predicates + 1 = 5

V(G) = E – N + 2 = 10 – 7 + 2 = 5

Basis Paths to be Tested are:

1. 1-7 (value[i] = -999.0)

2. 1-2-7 (value[i] = 0, totinputs=100)

3. 1-2-3-6-1-7

4. 1-2-3-4-6-1-7

5. 1-2-3-4-5-6-1-7

Testing (20)

Other White Box Methods

Condition testing: exercises the logical (boolean) conditions in a program

Data Flow testing: selects test paths according to the location of the definition and use of variables in a program

Loop testing: focuses on the correctness of loop structures

Testing (21) Loop Testing

Nested Loops

Simple Loop

Concatenated Loops

Unstructured Loops

Testing (22) Loop Testing

Test Cases for simple Loops

2. Only one pass through the loop

1. Skip the loop entirely

3. Two passes through the loop

4. m passes through the loop (m<n)

5. (n-1), n and (n+1) passes through the loop

where n is the maximum number of allowable passes

Testing (23) Loop Testing

Testing Nested Loops

2. Test the min, min+1, typical, max-1 and max for the inner loops

1. Start at the innermost loop. Set all the outer loops to their minimum iteration parameter variables (i.e. loop control variable)

3. Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this until the outermost loop has been tested

Testing Concatenated Loops

If the loops are independent of one another then treat each as a simple loop, otherwise treat as nested loops

Testing Unstructured Loops

Don’t bother! Re-design!

Testing (24) Black Box Testing

Requirements

Events

Inputs Outputs

Black Box Testing is complementary to white box testing. Decide on what external conditions (i.e. inputs, requirements, events) that fully exercise allfunctional requirements (i.e. test all functions encoded in software)

Testing (25) Black Box Testing

Black Box Strengths

Attempts to find errors in the following categories:

Incorrect or missing functions

Interface errors

Errors in data structures or external database access

Behaviour or performance errors

Initialisation or termination errors

Black box testing is performed during later stages of testing

Testing (26) Black Box Methods

Black Box Methods

Equivalence Partitioning

- Divide input domain into classes of data

- Each test case then uncovers whole classes of errors

- Examples: valid data (user supplied commands, files names, graphical data (e.g. mouse selections),

invalid data (data outside bounds, physically impossible data e.g. negative value when only positive possible),

valid data supplied in an invalid situation (e.g. an order_quantityof 5,000 might be valid for an item named chalk but would most likely be invalid for an item called projector.

Testing (27) Black Box Methods

Black Box Methods

Boundary Value Analysis

- More errors tend to occur at the boundaries of the input domain

- Select test cases that exercise bounding values

- Examples: an input condition specifies a valid range of input values bounded by values a and b. Test cases should be designed with values a and band just above and below a and b.

Testing (28) Debugging

>..

.

.

.

.

.

.

Debugging

Suspected causes

New test cases

Regression tests

Corrections Identified causes

Test cases Execution of cases

Testing is a structured process that identifies an error’s “symptoms”

Debugging is a diagnostic process that identifies an error’s “cause”

Testing for unintended knock-on effects

Testing (29) Debugging Effort

Time required to correct the errorand conduct regression tests

Time required to diagnose the symptomand determine the cause

Regression testing means re-execution of a subset of test cases to ensure that changes made to correct errors do not have unintended side effects

Testing (30) Bugs- Symptoms & Causes

Symptom

Cause

Symptom and cause may be geographically separated

Symptom may disappear when another problem is fixed

Cause may be due to a combination of non-errors

Cause may be due to a system or compiler error

Cause may be due to an assumption that everyone believes

Symptom may be intermittent

Testing (31) Not All Bugs are Equal !

damage

Bug type

annoying

mild

infectious

catastrophic

extreme

serious

disturbing

Bug Categories: function-related bugs, data bugs, coding bugs,design bugs, documentation bugs, standards violations, etc

Testing (32) Debugging Techniques

Brute Force:

Backtracking:

Cause Elimination:

- Use when all else fails

- Try memory dumps and run-time traces

- Search through mass of information which may lead to source of error

- Can work in small programs where there are few backward paths

- Trace the source code backwards from the error to the source

- Create a set of “cause – hypothesis” for each error

- Use error data (program output) or further tests to prove or disprove these hypotheses

Some people seem to have intuitive skill at debugging and can find the source of errors quickly

Testing (33) Debugging Tips

Don’t immediately dive into the code, think about the symptom you are seeing

Use tools (e.g. dynamic debuggers) to gain further insight about the error problem

If you can’t solve the problem and locate the source of the error, get help from someone

Ask these questions before attempting to “fix” the bug:

1. Is the cause of the bug reproduced in another part of the program, i.e. are there duplicates of the error in the code?

2. Could another bug be introduced by the fix?

3. What could have been done to fix the bug at a design or coding-plan level in the first place?

Be absolutely sure to conduct regression tests when you do “fix” the bug