CSC 478CSC 478 11

Software TestingSoftware Testing

Chapter 23Chapter 23

CSC 478 2

The Testing ProcessThe Testing Process

► Component testing Component testing Testing of individual program components;Testing of individual program components; Usually the responsibility of the component developer Usually the responsibility of the component developer

(except sometimes for critical systems);(except sometimes for critical systems); Tests are derived from the developer’s experience.Tests are derived from the developer’s experience.

► System testingSystem testing Testing of groups of components integrated to create a Testing of groups of components integrated to create a

system or sub-system;system or sub-system; The responsibility of an independent testing team;The responsibility of an independent testing team; Tests are based on a system specification.Tests are based on a system specification.

CSC 478 3

Testing PhasesTesting Phases

Componenttesting

Systemtesting

Software developer Independent testing team

CSC 478 4

Defect TestingDefect Testing►The goal of defect testing is to discover The goal of defect testing is to discover

defects in programsdefects in programs►A A successfulsuccessful defect test is a test which defect test is a test which

causes a program to behave in an causes a program to behave in an anomalous wayanomalous way

►Tests show the presence not the absence of Tests show the presence not the absence of defectsdefects

CSC 478 5

Testing Process GoalsTesting Process Goals

► Validation testingValidation testing To demonstrate to the developer and the system To demonstrate to the developer and the system

customer that the software meets its requirements;customer that the software meets its requirements; A successful test shows that the system operates as A successful test shows that the system operates as

intended.intended.

► Defect testingDefect testing To discover faults or defects in the software where its To discover faults or defects in the software where its

behaviour is incorrect or not in conformance with its behaviour is incorrect or not in conformance with its specification;specification;

A successful test is a test that makes the system A successful test is a test that makes the system perform incorrectly and so exposes a defect in the perform incorrectly and so exposes a defect in the system.system.

CSC 478 6

► Only exhaustive testing can show a program is Only exhaustive testing can show a program is free from defects. However, exhaustive testing is free from defects. However, exhaustive testing is impossible,impossible,

► Testing policies define the approach to be used in Testing policies define the approach to be used in selecting system tests:selecting system tests: All functions accessed through menus should be testedAll functions accessed through menus should be tested Combinations of functions accessed through the same Combinations of functions accessed through the same

menu should be testedmenu should be tested Where user input is required, all functions must be Where user input is required, all functions must be

tested with correct and incorrect inputtested with correct and incorrect input

Testing PoliciesTesting Policies

CSC 478 7

System TestingSystem Testing

►Involves integrating components to create a Involves integrating components to create a system or sub-system.system or sub-system.

►May involve testing an increment to be May involve testing an increment to be delivered to the customer.delivered to the customer.

►Two phases:Two phases: Integration testingIntegration testing - the test team have access - the test team have access

to the system source code. The system is tested to the system source code. The system is tested as components are integrated.as components are integrated.

Release testingRelease testing - the test team test the - the test team test the complete system to be delivered as a black-box.complete system to be delivered as a black-box.

CSC 478 8

Integration TestingIntegration Testing

► Involves building a system from its components Involves building a system from its components and testing it for problems that arise from and testing it for problems that arise from component interactions.component interactions.

► Top-down integrationTop-down integration Develop the skeleton of the system and Develop the skeleton of the system and

populate it with components.populate it with components.► Bottom-up integrationBottom-up integration

Integrate infrastructure components then add Integrate infrastructure components then add functional components.functional components.

► To simplify error localisation, systems should be To simplify error localisation, systems should be incrementally integrated.incrementally integrated.

CSC 478 9

Testing ApproachesTesting Approaches

► Architectural validationArchitectural validation Top-down integration testing is better at discovering Top-down integration testing is better at discovering

errors in the system architecture.errors in the system architecture.► System demonstrationSystem demonstration

Top-down integration testing allows a limited Top-down integration testing allows a limited demonstration at an early stage in the development.demonstration at an early stage in the development.

► Test implementationTest implementation Often easier with bottom-up integration testing.Often easier with bottom-up integration testing.

► Test observationTest observation Problems with both approaches. Extra code may be Problems with both approaches. Extra code may be

required to observe tests.required to observe tests.

CSC 478 10

Release TestingRelease Testing

► The process of testing a release of a system that The process of testing a release of a system that will be distributed to customers.will be distributed to customers.

► Primary goal is to increase the supplier’s Primary goal is to increase the supplier’s confidence that the system meets its confidence that the system meets its requirements.requirements.

► Release testing is usually black-box or functional Release testing is usually black-box or functional testingtesting Based on the system specification only;Based on the system specification only; Testers do not have knowledge of the system Testers do not have knowledge of the system

implementation.implementation.

CSC 478 11

Testing GuidelinesTesting Guidelines

►Testing guidelines are hints for the testing Testing guidelines are hints for the testing team to help them choose tests that will team to help them choose tests that will reveal defects in the systemreveal defects in the system Choose inputs that force the system to generate Choose inputs that force the system to generate

all error messagesall error messages Design inputs that cause buffers to overflow;Design inputs that cause buffers to overflow; Repeat the same input or input series several Repeat the same input or input series several

timestimes Force invalid outputs to be generatedForce invalid outputs to be generated Force computation results to be too large or too Force computation results to be too large or too

smallsmall

CSC 478 12

Use CasesUse Cases

►Use cases can be a basis for deriving the Use cases can be a basis for deriving the tests for a system. They help identify tests for a system. They help identify operations to be tested and help design the operations to be tested and help design the required test cases.required test cases.

►From an associated sequence diagram, the From an associated sequence diagram, the inputs and outputs to be created for the inputs and outputs to be created for the tests can be identified.tests can be identified.

CSC 478 13

Performance TestingPerformance Testing

►Part of release testing may involve testing Part of release testing may involve testing the emergent properties of a system, such the emergent properties of a system, such as performance and reliability.as performance and reliability.

►Performance tests usually involve planning a Performance tests usually involve planning a series of tests where the load is steadily series of tests where the load is steadily increased until the system performance increased until the system performance becomes unacceptable.becomes unacceptable.

CSC 478 14

Stress TestingStress Testing

► Exercises the system beyond its maximum design Exercises the system beyond its maximum design load. Stressing the system often causes defects to load. Stressing the system often causes defects to come to light.come to light.

► Stressing the system test failure behaviour.. Stressing the system test failure behaviour.. Systems should not fail catastrophically. Stress Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or testing checks for unacceptable loss of service or data.data.

► Stress testing is particularly relevant to distributed Stress testing is particularly relevant to distributed systems that can exhibit severe degradation as a systems that can exhibit severe degradation as a network becomes overloaded.network becomes overloaded.

CSC 478 15

Component TestingComponent Testing

► Component or unit testing is the process of testing Component or unit testing is the process of testing individual components in isolation.individual components in isolation.

► It is a defect testing process.It is a defect testing process.► Components may be:Components may be:

Individual functions or methods within an objectIndividual functions or methods within an object Object classes with several attributes and methodsObject classes with several attributes and methods Composite components with defined interfaces Composite components with defined interfaces

used to access their functionalityused to access their functionality

CSC 478 16

Object Class TestingObject Class Testing

►Complete test coverage of a class involvesComplete test coverage of a class involves Testing all operations associated with an objectTesting all operations associated with an object Setting and interrogating all object attributesSetting and interrogating all object attributes Exercising the object in all possible statesExercising the object in all possible states

►Inheritance makes it more difficult to design Inheritance makes it more difficult to design object class tests as the information to be object class tests as the information to be tested is not localised.tested is not localised.

CSC 478 17

►Objectives are to detect faults due to Objectives are to detect faults due to interface errors or invalid assumptions about interface errors or invalid assumptions about interfaces.interfaces.

►Particularly important for object-oriented Particularly important for object-oriented development as objects are defined by their development as objects are defined by their interfaces.interfaces.

Interface TestingInterface Testing

CSC 478 18

Interface TypesInterface Types► Parameter interfacesParameter interfaces

Data passed from one procedure to another.Data passed from one procedure to another.

► Shared memory interfacesShared memory interfaces Block of memory is shared between procedures or Block of memory is shared between procedures or

functions.functions.

► Procedural interfacesProcedural interfaces Sub-system encapsulates a set of procedures to be Sub-system encapsulates a set of procedures to be

called by other sub-systems.called by other sub-systems.

► Message passing interfacesMessage passing interfaces Sub-systems request services from other sub-systems.Sub-systems request services from other sub-systems.

CSC 478 19

Interface ErrorsInterface Errors► Interface misuseInterface misuse

A calling component calls another component and A calling component calls another component and makes an error in its use of its interface e.g. parameters makes an error in its use of its interface e.g. parameters in the wrong order.in the wrong order.

► Interface misunderstandingInterface misunderstanding A calling component embeds assumptions about the A calling component embeds assumptions about the

behaviour of the called component which are incorrect.behaviour of the called component which are incorrect.

► Timing errorsTiming errors The called and the calling component operate at The called and the calling component operate at

different speeds and out-of-date information is different speeds and out-of-date information is accessed.accessed.

CSC 478 20

Interface Testing GuidelinesInterface Testing Guidelines► Design tests so that parameters to a called Design tests so that parameters to a called

procedure are at the extreme ends of their ranges.procedure are at the extreme ends of their ranges.► Always test pointer parameters with null pointers.Always test pointer parameters with null pointers.► Design tests which cause the component to fail.Design tests which cause the component to fail.► Use stress testing in message passing systems.Use stress testing in message passing systems.► In shared memory systems, vary the order in In shared memory systems, vary the order in

which components are activated.which components are activated.

CSC 478 21

Test Case DesignTest Case Design

►Involves designing the test cases (inputs Involves designing the test cases (inputs and outputs) used to test the system.and outputs) used to test the system.

►The goal of test case design is to create a The goal of test case design is to create a set of tests that are effective in validation set of tests that are effective in validation and defect testing.and defect testing.

►Design approaches:Design approaches: Requirements-based testingRequirements-based testing Partition testingPartition testing Structural testingStructural testing

CSC 478 22

Requirements Based TestingRequirements Based Testing

►A general principle of requirements A general principle of requirements engineering is that requirements should be engineering is that requirements should be testable.testable.

►Requirements-based testing is a validation Requirements-based testing is a validation testing technique where you consider each testing technique where you consider each requirement and derive a set of tests for requirement and derive a set of tests for that requirement.that requirement.

CSC 478 23

Partition TestingPartition Testing

►Input data and output results often fall into Input data and output results often fall into different classes where all members of a different classes where all members of a class are related.class are related.

►Each of these classes is an Each of these classes is an equivalence equivalence partitionpartition or domain where the program or domain where the program behaves in an equivalent way for each class behaves in an equivalent way for each class member.member.

►Test cases should be chosen from each Test cases should be chosen from each partition.partition.

CSC 478 24

Equivalence PartitioningEquivalence Partitioning

System

Outputs

Invalid inputs Valid inputs

CSC 478 25

Testing Guidelines (Sequences)Testing Guidelines (Sequences)►Test software with sequences which have Test software with sequences which have

only a single value.only a single value.►Use sequences of different sizes in different Use sequences of different sizes in different

tests.tests.►Derive tests so that the first, middle and last Derive tests so that the first, middle and last

elements of the sequence are accessed.elements of the sequence are accessed.►Test with sequences of zero length.Test with sequences of zero length.

CSC 478 26

►Sometime called white-box testing.Sometime called white-box testing.►Derivation of test cases according to Derivation of test cases according to

program structure. Knowledge of the program structure. Knowledge of the program is used to identify additional test program is used to identify additional test cases.cases.

►Objective is to exercise all program Objective is to exercise all program statements (not all path combinations).statements (not all path combinations).

Structural TestingStructural Testing

CSC 478 27

Path TestingPath Testing

►The objective of path testing is to ensure The objective of path testing is to ensure that the set of test cases is such that each that the set of test cases is such that each path through the program is executed at path through the program is executed at least once.least once.

►The starting point for path testing is a The starting point for path testing is a program flow graph that shows nodes program flow graph that shows nodes representing program decisions and arcs representing program decisions and arcs representing the flow of control.representing the flow of control.

►Statements with conditions are therefore Statements with conditions are therefore nodes in the flow graph.nodes in the flow graph.

CSC 478 28

Test AutomationTest Automation

► Testing is an expensive process phase. Testing Testing is an expensive process phase. Testing workbenches provide a range of tools to reduce workbenches provide a range of tools to reduce the time required and total testing costs.the time required and total testing costs.

► Systems such as Junit support the automatic Systems such as Junit support the automatic execution of tests.execution of tests.

► Most testing workbenches are open systems Most testing workbenches are open systems because testing needs are organisation-specific.because testing needs are organisation-specific.

► They are sometimes difficult to integrate with They are sometimes difficult to integrate with closed design and analysis workbenches.closed design and analysis workbenches.

CSC 478 29

Testing Workbench AdaptationTesting Workbench Adaptation

►Scripts may be developed for user interface Scripts may be developed for user interface simulators and patterns for test data simulators and patterns for test data generators.generators.

►Test outputs may have to be prepared Test outputs may have to be prepared manually for comparison.manually for comparison.

►Special-purpose file comparators may be Special-purpose file comparators may be developed.developed.