Software Testing. ‘The process of executing a program... with the intention of finding errors’ (Myers, 1976) Example: Test a module that inputs three

Software Testing

Software Testing

‘The process of executing a program... with the intention of finding errors’ (Myers, 1976)

Example:Test a module that inputs three integers (i1, i2, and i3) and

outputs that they • make an equilateral triangle• make an isosceles triangle• make a scalene triangle• do not make a triangle (an error)

Adapted from Binder (1999), originally proposed by Myers (1979).

Testing example

Take the time now to work out what tests need to be done, for example:

i1 not > 0 i1 + i2 <= i3 i1, i2, i3 all > 0 and all = etc

Use this space to write some test cases

Testing example (2)

Testing example (3)How many cases did you identify?

If you thought that 10 or fewer tests were enough, you were missing essential cases

If you thought that 20 or fewer tests were enough, you were making assumptions about the implementation

Testing example (4)

To test the module thoroughly (exhaustively), we need to test every possible input

• For 32 bit integers, we have 4x10^9 possible values

• 3 integers have 6.4 x 10^28 values• If we test 1000 cases per second, testing this

module exhaustively would take more than a billion times the age of the universe!

Clearly, we will need to be more selective in the way we test!

Finding Errors

‘The process of executing a program... with the intention of finding errors’ (Myers, 1976)

There are two kinds of errors Errors of omission

– we have left something out that we should put in– test that all the requirements and specifications are satisfied

Errors of commission– we have put something in that we should leave out– impractical to test that only the requirements and

specifications are satisfied, ie the product does nothing else

Context of testing

Since we cannot test exhaustively, we compliment our testing with Quality Assurance techniques such as code

walkthroughs inspections reviews

Equivalence Classes

Many exhaustive tests would be testing the same logical path, so we assume these can be combined• Such a set of cases is called an

equivalence class• These equivalence classes can be derived

from the requirements or the specification

Hazards

We test for known or likely causes of error Test every line of code Test every path in a module Test equivalence class boundary conditions Test complex logic Test interfaces Re-test super-class in sub-class Test object life cycles Test using “real” data volumes etc

Selecting tests

From experience we have learned that we can get good results by

• Testing for known hazards• Testing every known “business

scenario” and “use case”

Major Decisions:WHO WHAT HOW

Who does the Testing?

» The programmer ?» An independent team ?» The users ?

Everyone!

What to Test

• Unit testing

procedural function, OO method• Module testing

procedural program, OO class• Subsystem testing• Integration testing• Acceptance testing

What to Test ...Unit Testing

Testing is not debugging– If our software fails a test, we do debugging to find the cause

Test» The lowest level functions and procedures in isolation» Each logic path in the component specifications» Every line of code (white box)

What features of the Eiffel language assist with testing?» What do preconditions do?» What do postconditions do?

What to Test ...Module Testing

Test» The interaction of all the related

components of a module / class» Tests the module as a stand-alone entity

What do Eiffel invariants do?

In an OO system how would you create a ‘test harness’ or ‘test driver’ for a module?

What to Test ...

Subsystem Testing» Tests the interfaces between the modules» Scenarios or Use Cases are employed to test

module interaction

What to Test ...

Integration (System) Testing• Tests interactions between sub-systems and

components• System performance• Stress testing• Volume testing• Scenarios or Use Cases specified in the

requirements

What to Test ...

Acceptance Testing• Tests the whole system with Scenarios or Use

Cases specified in the requirements• Tests the whole system with live data• Establishes the ‘validity’ of the system

How to Test Software

1. Use a Test Plan2. Establish a test strategy Top/Down Bottom/Up Techniques

– White Box– Black Box

3. Use appropriate tools Automate as much testing as possible

How to Test Software...

1. Use a Test Plan

The Test Plan establishes:» Standards, deliverables» Major testing phases» Traceability of testing to requirements» Test schedule and resource allocation» Test record documentation» Testing strategies

Examples of Test Plans are provided in the tutorial notes and the assignment

How to Test Software …

Assignment Testing Documents1. To be completed from the requirement specifications before any code is written, i.e., Expected Results

- Test Plan

- Test Summary

2. To be completed during testing,

i.e., Expected + Actual Results

- Test Report


Assignment Testing Documents …- Test Plan

- Describe how you are going to verify that your system modifications will work as specified- Only include tests for your modifications (+ any missing tests from Stage 0))

- Test Summary- show all the tests, including regression tests, i.e. show all Stage 1 tests as well as your own tests- use as a checklist of all tests


Assignment Testing Documents …• Test Report

Includes black and white box testing:

• Test Plan • showing actual results for your tests

• Test Summary • of all tests with indication of all working tests

(i.e. a checklist ‘tick’ for all working tests)


2. Establish a Test StrategyTop-Down and Bottom-Up Testing

Top-Down Testing Major Features:

. Control program is tested first

. Modules are integrated one at a time

. Major emphasis is on interface testing

. Starts at sub-system level with modules as ‘stubs’

. Then tests modules with functions as ‘stubs’

. Used in conjunction with top-down program development


2. Establish a Test Strategy

Top-Down and Bottom-Up Testing

Bottom-Up Testing

Major Features: • Allows early testing aimed at proving feasibility of particular

modules• Modules can be integrated in various clusters as desired• Major emphasis is on module functionality and performance


Testing StrategiesTop-Down Testing

• Advantages• Design errors are trapped earlier• A working (prototype) system• Enables early validation of design

• Disadvantages• Difficult to produce a stub for a complex component• Difficult to observe test output from top-level

modules


Testing Strategies ...Bottom-Up Testing

» Advantages Easier to create test cases and observe output Uses simple drivers for low-level modules to provide

data and the interface Natural fit with OO development techniques

» Disadvantages No ‘program’ until all modules tested High-level errors may cause changes in lower

modules


Testing Strategies …Advantages (again)Top-Down No test drivers are needed

The control program plus a few modules forms a basic early prototype

Interface errors discovered earlyModular features aid debugging

Bottom-Up No test stubs are neededEasier to adjust manpower needsErrors in critical modules are found earlyNatural fit with OO development techniques


Testing Strategies …Disadvantages (again)Top-Down . Test stubs are needed

. Extended early phases dictate a slow manpower buildup

. Errors in critical modules at low levels are found late

Bottom-Up . Test drivers are needed. Many modules must be integrated

before a working program is available. Interface errors are discovered late


2. Establish a Test Strategy ... Testing Techniques

– Structural (White Box) Testing– Functional (Black Box) Testing– Equivalence Partitioning– Mathematical / Formal Verification– Paradigm / Language hazards


Testing Techniques Structural (White Box) Testing

» Use code to derive a program flow-graph which shows all logic paths

» Maximum test paths = program’s ‘Cyclomatic Complexity’ (McCabe)

» Use a dynamic program analyser to identify code not executed

White Box Testing attempts to test every line of code


Testing Techniques ...

Functional (Black Box) Testing» Test cases are based on functions as outlined in specification» Can be developed when program specifications are complete ie

before code is started» The Test Plan with the test cases can be developed from the

functional requirements

Black Box testing only checks the outputs for accuracy – not every line of code is tested.


Testing Techniques ...White Box testing should only be used for

» Unit testing

White Box and Black Box testing can be used for» Module testing

Only use Black Box testing for» Subsystem testing » Integration testing» Acceptance testing


3. Use Appropriate Testing Tools» Test data generators» Test control / evaluation environments» Program analysers

Static Dynamic

» File comparators» Simulators» Symbolic program dumps» Trace packages» Interactive debugging environments

Regression Testing

Maintenance/Development Testing Differences

Maintenance:– Only changes need to be reviewed– Only new test cases that exercise the change

need to be developed– Test results compared against previous test

results (Regression Testing)– Data collected during impact analysis identifies

testing at each level

When to Stop Testing ...

When do you stop testing?NOT - When delivery date arrives!!

- usually what happens!!

It should be:When test goals have been reached

» code coverage» case coverage» risk coverage» defect density (defects / LOC)» defect arrival rate

Testing Summary

When testing always: Use a Test Plan Use testing strategies and techniques

Top/Down - Bottom/Up White Box - Black Box

Use appropriate tools : automate tests For maintenance

– Regression Test