How to define efficient test goals and strategies...How to define efficient Anders Claesson 2004-09-03 test goals and strategies Workflow 1 Perform a test analysis. 2 Specify the test

How to define efficient Anders Claesson 2004-09-03 test goals and strategies

How to define efficient test goals and strategies

Anders [email protected]

www.qualtechconferences.com


Contents

• Introduction • Workflow• Experiences • Conclusions


IntroductionLearn:- To define measurable testing goals and efficient test strategies.- The input/output and steps in defining the test goals and strategies.- To increase the test efficiency by having the right test focus.

Added value:- Shorter lead times.- Increased quality of testing.- Better use of resources.


Workflow1 Perform a test analysis.2 Specify the test scope.3 Divide testing into phases.4 Define test goals.5 Specify test metrics.6 Define entry and exit criteria.7 Define test strategies.8 Define when to stop testing.9 Identify and adapt the

test process.10 Identify the test items (features,

functions, interfaces, etc.) to be tested.


1 - Perform a test analysisPurpose: To understand what system is to be tested, what is

important for the customer/user and what problems might occur.

1 Identify project goals (i.e. the purpose of the project).2 Analyse requirements and needs.3 Identify new and changed functions.4 Identify important system characteristics.5 Identify critical system parts (e.g. real time performance).6 Identify system architecture (interfaces, sub-systems).7 Identify how and when the system will be integrated.8 Analyse problems in the previous release.9 Analyse current risks and possible failure modes.


1 - Perform a test analysisInp Document type Document name Document id. Rev

T

T Use cases Use Case Specification for interlocking Abc-123 x.y

S Report Technical Issues regarding XXX Abc-123 x.y

T Interface Spec. Interface Specification for sub-system A - B Abc-123 x.y

T Standards UNISIG ERTMS Specification subset yyyy Abc-123 x.y

S

T

Tp

S

T

S

Requirements Customer requirements Abc-123 x.y

Specification System Definition XX Abc-123 x.y

Specification System Specification XX project Abc-123 x.y

Specification System Architecture Description Abc-123 x.y

Specification XX Product Structure Abc-123 x.y

Risk list Technical risks in XXX Abc-123 x.y

Glossary Project Glossary Abc-123 x.yHow to use the input documents:T - To be tested/covered in System Test (fully or partly depending on if the specification

covers more than the current release). Tp - To be partly covered regarding details in functional behaviour.S - Supporting document. To be used only as information.

Example: Input information. Test planning Baseline


1 - Perform a test analysisNo Document type Document name Document id. Rev

1

2 Detailed Test Plan Integration test plan for XXX Abc-123 x.y

7 Test Lab Guide Testers lab users guide. Abc-123 x.y

10 System Test Report Results from all system tests. Abc-123 x.y

3 Integration Test Specification Integration Test of sub-system A – B. Abc-123 x.y

4 System Test Specification Functional tests of XXX. Abc-123 x.y

5

6

8

9

Master Test Plan Master Test plan for all test levels and phases. Abc-123 x.y

System Test Specification Non-functional tests of XXX. Abc-123 x.y

Test Lab Specification Lab set up and drawings. Abc-123 x.y

Risk follow up Risks test results. Abc-123 x.y

Integration Test Report Results of all integration tests. Abc-123 x.y

Example: Output information. Test output Baseline


2 - Specify the test scopePurpose: To define which types of tests should be included and excluded

in the testing of this particular project.

1 Identify the scope of the project (e.g. from the project specification).2 Use the results from the test analysis regarding what functions/features,

characteristics, interfaces and risks should be covered.3 Include the overall scope and what should be covered in testing in a

Master Test Plan.

Example:Included Integration tests, functions described in the

System Requirement Specification, user interfaces, site data.

Excluded Protocol tests, GSM-r network, tests related to installation.


3 - Divide testing into levels and phases

Purpose: To decide levels and phases of testing based on how the system will be integrated and when different types of tests will be performed.

1 Identify the integration steps (e.g. unit, function and system level).2 Decide which test phases to include based on:

- Integration steps.- Test objectives and scope (e.g. acceptance test, field test).- System architecture (e.g. sub-system division).

Example:• Unit test.• HW and SW integration test.• Sub-system integration test.• System integration test.• System test.• Acceptance test.• Installation test.• Field test.


4 - Define test goalsPurpose: To define goals of what should be achieved in testing.

1 Define testing goals for each test phase and goals for the delivery to the customer.

2 Testing goals may cover the following areas:a) The quality of testing, i.e. test coverage of:

- Functions.- Characteristics.- Use cases.- Modules.- Source code.

b) The quality of the product/system, e.g. risk and error trend level (based on project goals).

c) The time frame in which testing should be completed (based on project goals).


4 - Define test goalsExample:Prio Goal Description

1

1

1

1

2

2

2 Operational manuals tested.

Usability testing of the operational manuals should be performed at system level testing. The service manuals are written per Sub-system and tested on Sub-system level. Installation and maintenance manuals are tested by ITC.

100% requirements coverage.

Each requirement specified in the SRS should be covered by at least two Test Cases (one positive and one negative). More Test Cases per requirement should be indicated by the risk analysis.

100% coverage of the main flows in the operational scenarios.

The main flow in each operational scenario should be covered by at least one Test Case (both for regression test and test of new functionality).

100% of the highest risks covered.

Each high priority risk (defined as intolerable in the risk analysis) shall be covered by at least one Test Case. Each risk should be traceable to corresponding interface specification, requirement and/or operational scenario.

100% of externally observable failure modes covered.

Each failure mode in the system, that is observable in the external interface, should be covered by at least one Test Case.

100% feature coverage.

Each feature (user service or operation) should be covered by at least one Test Case (both new tests and regression tests).

100% of externally observable system states covered.

Each system state that is observable in the external interface of the system shall be covered by at least one Test Case. All transitions between states may not need to be covered.


5 - Specify test metricsPurpose: To define the metrics to be used for follow up of testing activities.

1 Select a few metrics covering the high priority testing goals and other areas important to follow up.

2 Describe the metrics and how they will be collected, presented and followed up.

M/O Metric Scale

M Requirements coverage. % of covered requirements versus total number of requirements in the RS.

M Statement coverage. % of covered statements versus the total number of statements in a module.

O Planned versus actual start date of test execution.

Days difference from the originally planned date.

M Written Incident Reports. Number of written Incident Reports by testers. Measure for each test phase and in total.

M Remaining risks. Number of remaining risks in the system per severity level after finished testing.

M = Mandatory O = Optional

Example:


6 - Define entry and exit criteriaPurpose: To define when a test phase can start and when it can be

considered finished.

1 Identify the least amount of input to be able to start a test phase.2 Specify what needs to be fulfilled in each test phase before it can be

considered complete.

Example: Test Phase Entry Criteria – Test Execution

System Test

• One or more integrated functions on the system platform possible to test both stand-alone and interacting with other functions

• Test Cases prepared covering some part of the features and requirements to be tested.

• Required test tools integrated and tested in the test environment.• Test scripts written and tested.• Test data prepared.


6 - Define entry and exit criteriaExample: Test Phase Exit Criteria

Module Test

• 100% Software Design and Module Specifications covered.• 100% Statement Coverage of the source code.• Boundary Value Analysis Testing of all input/output.• Equivalence Classes and Input Partitioning Testing.• All Test Cases passed and no remaining faults to be corrected.• All code reviewed.• Known weaknesses described.• Module test reports written (including obtained test coverage).


7 - Define test strategiesPurpose: To define the approach how the project goals and the testing

goals should be achieved.

1 Use a checklist to find strategies covering the goals and common testing problems. What are the common design mistakes. Analyse bug-reports.

2 Draw a matrix of common strategies and test techniques per test phase.3 List strategies per test type, e.g. performance test, function test, etc.4 Define strategies for:

a) How to obtain sufficient test coverage.b) Test data and Test Case selection.c) Regression testing.d) Test environment.e) Risk handling.f) The use of tools, e.g. test automation, test

management, requirement test traceability.


7 - Define test strategiesExample: Strategies to be used per subject area.

3. Review input specifications

8. Specify Test Cases in thesame order as requirements are implemented.

4. Identify common and critical functionality.

11. Focus testing on the highest risks.

12. Run 50% normal cases, 50% negative Test Cases.

10. Have Test Case selection workshops.

6. Plan testable increments.

9. Educate in test techniques.

13. Measure important characteristics.

5. Build the test environment like the specific application.

2. Identify input specifications to be covered by tests.

Test Analysis

Test Case SelectionTest Planning

Test Environment

7. Create a logical Test Plan.

1. Use the new System Validation Test Process.

Test Process

14. Start by writing TS’sagainst the Operational Scenarios.

15. Cover Operational Manuals indirectly by TC.


8 - Define when to stop testingPurpose: To define when to continue or stop testing before delivery to the

customer.

1 Define a test stop criteria linked to the test goals and the system release criteria to determine when testing can be considered finished.

2 Define the criteria in the following areas:a) Test coverage.b) Product/system quality.c) Time to market.


8 - Define when to stop testingExample: Stop criteria.

All planned Test Cases should be executed and passed. Sufficient test coverage must be obtained according to the priority-one test goals.

Test Coverage

Testing should stop when: • The probability of remaining

faults is reduced to a level that can be accepted by the customer [B. Beizer].

• The systems’ risk level is within acceptable limits (i.e. no remaining high severity risks).

• The product value has been demonstrated and accepted (i.e.implicit and explicit quality attributes are satisfied).

Quality

When the agreed ship date has been reached.

Time


9 - Identify and adapt the test process

Purpose: To define which and how the test process should be used.To adapt the way of working for the current project and products.To define current project roles, who is doing what and when.

1 Decide how the standard test process in the company should be used.2 Add, change or remove any parts serving

the selected test strategies and test goals.

Example:• White box test should be performed during Unit Test by the developers. Use the

BS7925-2 standard in addition to the company test process.• Testing of system characteristics should be performed by separate teams

focused on the most important characteristics. Standards regarding usability (http://www.developingskills.com/ds.php?cat=use) should, for example, be used in addition to the company test process.


10 - Identify the test items to be tested.

Purpose: To list the current test items according to scope, goals and strategies.To decide which test strategies to use for each test item.

1 Prepare a test item list per test phase.2 Select test strategies and test techniques

applicable for each test item. Example:• Divide features to be tested into separate

Test Specifications.• A Use Case may be covered by one

Test Specification.• Use the following strategies for Use Case testing: Requirements based testing,

scenario testing, risk based testing, exploratory testing, state based testing, boundary value analysis, equivalence class partitioning, failure mode testing.

Test Item List1. Start up on-board units.

2. Restart of failed units.

3. Operational Scenario x.

4. .........

OK

Started

Failed

-

Strategy

Strategy

Strategy

Strategy


Experiences• A successful introduction of a new test process

was performed at Bombardier Transportation in Sweden (2003-2004).• Detailed work instructions and document templates included.• Testing became more efficient and focused on what's important.• More focus put on the requirements and how to get missing

information.• Workshops were performed to identify risks,

to define test goals and strategies and how to select Test Cases. This resulted in faster and more efficient test preparation.

• The quality of testing increased significantly. More problems were identified starting with the requirements and throughout all testing phases.


Conclusions• With the right test focus, i.e. on test goals, requirements,

features, coverage, risks, exploration themes, strategies, processes, etc. significant improvements can be achieved.

• You may start with a simple, yet effective approach regarding the use of test goals, strategies and test techniques.

• Remember that your questioning skills (i.e. asking the right questions about the system) and ability to structure your thinking is the key to good testing.

Documents

How to define efficient test goals and strategies...How to define efficient Anders Claesson 2004-09-03 test goals and strategies Workflow 1 Perform a test analysis. 2 Specify the test