Testing Terminology

• Review Class 2• Test Process Overview• Risk Analysis• The Testing Process• Planning Activities• Team Meeting

CEN 5076 Class 3 – 09/19

CEN 5076 Class 3 - 09/19 2

Laws of Testing Handbook of SSE

• Online debugging is more efficient than offline debugging (Sackman’s).

• Testing can show the presence but not the absence of errors (Dijkstra).

• A developer is unsuited to test his or her code (Weinberg’s).

• Approximately 80 percent of defects come from 20 percent of modules (Paret-Zipf-type)

• Performance testing benefits significantly from system-level benchmarks (Gray-Serlin)

CEN 5076 Class 3 - 09/19 3

Laws of Testing Handbook of SSE• Usability is quantifiable (Nielsen-Norman law).• Partition testing is more effective than random

testing (Gutjahr’s hypothesis).• The adequacy of a coverage criterion can

only be intuitively defined (Weyuker’s hypothesis).

• The test suite needed to verify an arithmetic path expression can be determined (Endres-Glatthaar hypothesis).

• Suspicion-based testing can be more effective than most other approaches (Hamlet’s hypothesis).

CEN 5076 Class 3 - 09/19 4

Taxonomy of OO Classes• Class Abstraction Technique (CAT) that

support implementation-based testing.• Catalogs classes based on the characteristics

of a class.• “Class characteristics for a given class C are the

properties of the features in C and the relationships C has with other classes in the implementation. The properties of the features in C describe how criteria such as types, accessibility, shared class data, polymorphism, dynamic binding, deferred features, exception handling, and concurrency are represented in the attributes and routines of C.” [Clarke and Malloy SEA ’04]

CEN 5076 Class 3 - 09/19 5

Taxonomy of OO Classes cont

Descriptors Type Nomenclature Attributes Routines Families (Nested) New (Constant) NA no type (Multi-Parents) Recursive New P primitive type (Friend) Concurrent Recursive P* reference to P (Has-Friend) Polymorphic Redefined U user-defined type Generic Private Concurrent U* reference to U Concurrent Protected Synchronized L library Abstract Public Exception-R L* reference to L Inheritance-free Constant Exception-H A any type (generics) Parent Static Has-Polymorphic A* reference to A External Child - Non-Virtual m <n> parameterized type Internal Child - Virtual m <n *> reference to - - Deferred parameterized type - - Private where m ε {U, L}- - Protected n is any combination of - - Public { P, P*, U, U*, L, L*, A, A* }- - Static -

CEN 5076 Class 3 - 09/19 6

Taxonomy of OO Classes cont

CEN 5076 Class 3 - 09/19 7

Types of Inheritance• Model Inheritance – is-a relation between

abstractions in the model.- Subtype - View - Restriction- Extension

• Variation inheritance – expresses relations within the software itself rather than the model.- Functional- Type- Uneffecting

CEN 5076 Class 3 - 09/19 8

Types of Inheritance• Software inheritance – describes a class by how it

differs from another class.- Reification- Structure- Implementation- Facility – constant, machine

[Bertrand Meyer ’96]

CEN 5076 Class 3 - 09/19 9

Planning for Testing• Testing requires considerable resources.

• Good planning and good management is required for the effective utilization of these resources.

• Want to plan a test process that complements your development process.

• Want to analyze the risks associated with verifying the required functionality.

• Need to develop test plans for the different levels and types of testing required for a comprehensive test process.

CEN 5076 Class 3 - 09/19 10

Testing Process Overview• Testing is applied at various points during

development.

• Development and testing processes have different goals and different measures of success.

• Development strives to build a product that meets a need.

• Testing strives to answer questions about the product, including whether the product meets the need that it is intended to meet.

CEN 5076 Class 3 - 09/19 11

Testing Process Overview• Example:

- The lower the defect rate (ratio of test cases that fail to the total number used), the more successful the development process is considered to be.

- The higher the defect rate, the more successful the testing process is considered to be.

• The roles of development and testing are usually assigned to different people.

• Developers are responsible for some testing, e.g., unit testing and integration testing.

CEN 5076 Class 3 - 09/19 12

Testing Process Overview• To ensure testing is carried out with the same

vigor when it is done by developers a buddy system is used.

• Buddy system – functionality is partitioned and 2 developers take turns in writing code and testing.

• The development and testing processes are in a feedback loop, recall testing model USDP diagram.

• Recall in the USDP increments of the system are usually developed.

CEN 5076 Class 3 - 09/19 13

Testing Process OverviewIncremental approach:

Increment 1 – Analysis , Design, Implementation, Testing.

Increment 2 – Analysis , Design, Implementation, Testing.

…

Increment n – Analysis , Design, Implementation, Testing.

CEN 5076 Class 3 - 09/19 14

Testing Process Overview• The testing perspective must be considered,

preferably by professional testers, when development methods and tools are selected.

• The form and quality of the requirements specification also affects the testing process.

• Product requirements comprise the source of test cases in system and acceptance testing.

• System testers should participate in the gathering and validation of the reqs. – need to understand reqs., assess risks, and check for testability.

CEN 5076 Class 3 - 09/19 15

Testing Process Overview• Testability:

1. The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met. [IEEE 610]

2. The degree to which a requirement is stated in terms that permit establishment of test criteria and performance of tests to determine whether those criteria have been met. [IEEE 610]

• Test criteria- The criteria that a system or component must

meet in order to pass a given test. [IEEE 610]

CEN 5076 Class 3 - 09/19 16

Testing Process OverviewThere are two types of testing criteria:1. Test data selection criterion –

represents a rule used to determine which test case to select.

2. Test data adequacy criterion – a rule used determine whether or not sufficient testing has been performed.

• Test data selection criterion serves as the basis for picking a test set to satisfy some goal, while a test data adequacy criterion checks to see whether a previously selected test set satisfies the goal. [Weyuker 93]

CEN 5076 Class 3 - 09/19 17

Testing Process OverviewSTEP Testing technique [Hetzel 84]

1. Analysis – product to be tested is examined to identify any special features that must receive particular attention and to determine test cases that should be constructed.

2. Construction – artifacts that are needed for testing are created. Test cases are translated into programming languages and scripting languages, or they are entered in a tool-specific language. Data sets required for testing are built.

CEN 5076 Class 3 - 09/19 18

Testing Process OverviewSTEP Testing technique [Hetzel 84] cont

3. Execution and Evaluation – most visible and recognized part of the test effort. Test cases are executed and the results examined to determine if the software passed or failed the test suite.

• Test suites must be maintained, i.e., 1. as reqs. changed so does the test suite,

2. as problems are found by users test cases will be added to catch those problems.

CEN 5076 Class 3 - 09/19 19

Risk Analysis – A Tool for Testing• Risk

1. Anything that threatens the successful achievement of a project’s goal.

2. An event that has some probability of occurring, and if it does there will be some lost.

• Risk-based testing principle:- Test most heavily those portions of the system

that pose the highest risk to the project to ensure that the most harmful faults are identified.

• Risk analysis – a procedure for identifying risks and for identifying ways to prevent potential problems from becoming real.

CEN 5076 Class 3 - 09/19 20

Risk Analysis – A Tool for Testing

• Each project requires it own individual analysis, therefore it is important to apply testing strategies that make sense.

[“Effective software testing: 50 specific ways to improve your testing”. E. Dustin 2003 – see updated online course syllabus]

Some of these strategies include:

• Select test-design techniques – numerous techniques are available.

CEN 5076 Class 3 - 09/19 21

Risk Analysis – A Tool for Testing• Select testing tools – decide on the vendor-

provided tools; how the tools will be used, which team member will use them.

• Develop in-house test harness or scripts.

• Determine test personnel and expertise required.- To write test harness and scripts a developer

must be included in the testing team.

- Automation skills required for capture/playback skills.

- Domain expertise usually required.

CEN 5076 Class 3 - 09/19 22

Risk Analysis – A Tool for Testing• Determine testing coverage

- It is essential that testers understand the coverage required e.g., may be a contractual agreement in SRD.

- There might be a code coverage requirement e.g., DOD.

- Determine test coverage given the resources, schedules, tools, task at hand, and risks of not testing an item.

- Beizer estimates that from 2% to 80% of the application size can be code to support testing.

CEN 5076 Class 3 - 09/19 23

Risk Analysis – A Tool for Testing• Establish release criteria

- Indicate when testing can be considered complete.

- Try to state in a quantifiable manner e.g., use cases 1, 3, and 4 need to be defect free.

• Set the testing schedule

• Consider the testing phases- Different test strategies apply to different test

phases. (see V-model for testing phases in traditional model)

CEN 5076 Class 3 - 09/19 24

Risk Analysis – A Tool for TestingHigh risk factors:• Short time-to-market

- Can prevent adequate testing.- Test strategies adapted for the time available.

• New design process- Intro of new design tools, techniques. - Example change from USDP to eXtreme

programming approach or vise versa.

• New technology• Complexity

- Identify high risk uses case during reqs. analysis.

CEN 5076 Class 3 - 09/19 25

Risk Analysis – A Tool for Testing• Frequency of use

- This functionality usually represents the core of the application.

• Untestable functional and non-functional reqs.- A reason to quantify non-functional requirements.

- Must make all requirements testable.

• Other sources of risk:- Prog. Languages: permits certain classes of

errors and inhibit others e.g., strong typing vs. weak typing.

CEN 5076 Class 3 - 09/19 26

Risk Analysis – A Tool for Testing

Class Project:

• Each use case should have a risk, frequency, and criticality associated with it.

• Review each of the use cases to ensure the above properties are quantified and valid.

• Produce a ranked list of use cases. Focus on the implemented use cases.

• Use the risk, frequency, and criticality to identify the testing effort (time and personnel) required for each use case during system testing.

CEN 5076 Class 3 - 09/19 27

A Testing Process

Dimensions of software testing (think of the attributes in italics as a continuum):

1. Who performs the testing?

- Developer, tester, independent tester.

2. Which pieces will be tested?

- Test nothing, test a sample, test everything.

- Use a systematic approach. P. 81

CEN 5076 Class 3 - 09/19 28

A Testing Process

Dimensions of software testing cont:

3. When will testing be performed?

- When components are developed, when all components are developed and integrated.

4. How will testing be performed?

- Specification, implementation.

5. How much testing is adequate?

- Not testing, exhaustive testing.

CEN 5076 Class 3 - 09/19 29

Roles in Testing Process

Test plan should:

• Identify the roles each person will be assigned.

• For each role allocate time and effort.

• Schedule time allocated for each part of the testing effort.

• Development schedule drives much of the testing schedule.

• Identify the resources needed for the testing effort e.g., h/w, s/w, expertise

CEN 5076 Class 3 - 09/19 30

Roles in Testing Process

Roles:

• Unit tester – responsibility is to test the individual classes (cluster of classes) as they are produced.

• Integration tester – responsibility testing a set of objects that are being brought together from different development sources e.g., individuals or teams.

• System tester – has domain knowledge and is responsible for independently verifying that the completed application satisfies the reqs.

CEN 5076 Class 3 - 09/19 31

Roles in Testing Process• Test manager – responsible for managing the

test process i.e., requesting, coordinating, and making effective use of the resources allocated.

Other Roles see handout [Dustin]:

• Team lead – technical leadership for the test program, including test approach.

• Test engineers (usability, manual, automated, network, security) – specialist testers in each of the areas.

• Test environment specialist – installs test tools and establishes test-tool environment

CEN 5076 Class 3 - 09/19 32

Test Plan

Class Project:

• Roles for testing each model: test manager, test engineer (system, integration, class) tester, and minute keeper.

• Keep a diary of all activities performed at each meeting.

• Use information in handout to identify the duties/skills of each person on the team e.g., P. 73 first entry in table 13.2 .

CEN 5076 Class 3 - 09/19 33

Roles in Testing Process• Each diary entry should contain: start time of

meeting, end time of meeting, persons present, location of meeting, description of topics discuss (bullet form), tasks assigned to team members.

• Roles in the team will be rotated as each USDP model is tested. Initially students that developed the applications will be domain experts.

• Use an automated tool to generate a testing schedule e.g., MS project. Identify tasks, milestones, and deliverables for the testing activity.

CEN 5076 Class 3 - 09/19 34

A Detailed Set of Test Activities

• Fig 3.11 Synopsis of testing activities, P. 88

• Domain analysis

• Application analysis

• Architectural design

• Detailed design

• Class implementation

• Application implementation

CEN 5076 Class 3 - 09/19 35

Documentation

IEEE 829 Standard Test Plan outline:

1.0 Introduction – high level view of the testing, includes type of testing e.g., class, subsystem, system, acceptance, release.

2.0 Test Items – defines the scope, h/w and s/w to be tested.

3.0 Tested Features – parts of the s/w spec to be tested.

4.0 Features Not Tested – includes features already tested

5.0 Testing Strategy and Approach

5.1 Syntax

CEN 5076 Class 3 - 09/19 36

Documentation5.2 Description of Functionality

5.3 Arguments for Test – includes preconditions

5.4 Expected Output

5.5 Specific Exclusions

5.6 Dependencies

5.7 Test Case Success/Failure Criteria

6.0 Pass/Fail Criteria for the Complete Test Cycle

7.0 Entrance Criteria/Exit Criteria

8.0 Test-Suspension Criteria and Resumption Criteria

CEN 5076 Class 3 - 09/19 37

Documentation

9.0 Test Deliverables/Status Communication Vehicles

10.0 Testing Tasks

11.0 H/w and S/w reqs.

12.0 Problem Determination and Correction Responsibilities

13.0 Staffing and Training Needs/Assignments

14.0 Test Schedules

15.0 Risks and Contingencies

16.0 Approvals

CEN 5076 Class 3 - 09/19 38

Documentation

Visit the following site for more information:

http://www.coleyconsulting.co.uk/IEEE829.htm

http://www.evolutif.co.uk/tkb/guidelines/ieee829/example.html

CEN 5076 Class 3 - 09/19 39

Project Test Plan

• Summarizes testing strategy to be employed for the project.

• Fig. 3.15 P. 97 summarizes the activities that are required, the frequency with which each activity will be employed, and the entity responsible for this testing phase. Required for class project!

• Fig. 3.16 P. 97 associates each of the testing phases with the specific strategy for that phase. Required for class project!

CEN 5076 Class 3 - 09/19 40

Component Test Plan

• Defines the overall strategy and specific test cases that will be used to test a certain component.

• One plan per significant component (see uses cases).

• Fig. 3.17 P. 100 shows a template for the test plan. Required for class project!

• Each section of the plan contains two types of guided information:

CEN 5076 Class 3 - 09/19 41

Component Test Plan1. Project criteria – standards that have been

agreed upon and how thoroughly each component will be tested, e.g., 100% of the postconditions on modifier methods

should be tested.

2. Project procedures – techniques that have been agreed upon as the best way to handle a task. Provides the details of the test strategies that were identified in the project plan.e.g., constructing a Parallel Architecture for Class

Testing (PACT) class for each component that will be tested.

CEN 5076 Class 3 - 09/19 42

Component Test PlanSections of template:

• Objectives for the class - prioritized list of objectives for the component.

• Guided Inspection Requirements – (see P.99).

• Building and Retaining Test Suites

- Process of creating test driver classes.

- Scheduled deadline for delivery of test cases.

- The specification of the test driver

- Relative number of test cases (prioritized ) in each of the following categories.

CEN 5076 Class 3 - 09/19 43

Component Test Plan• Functional Test Cases

- Approach used to develop test cases from the specification.

- The class invariant method

- Identify the “types” of objects being tested. Based on the initial state of the object.

• Structural Test Cases

- Info about test cases developed for code coverage and code-review process.

- How to use the required test-coverage tool

CEN 5076 Class 3 - 09/19 44

Component Test Plan• State-Based Test Cases

- State representation for object.

- Approach used to generate test cases.

• Interaction Test Cases

- How dependencies will be handled.

- Creation of stubs required to handle cycles.

CEN 5076 Class 3 - 09/19 45

Use Case Test Plan• Describes the system level test to be derived

from a single use case.

• Incorporated by reference into both the integration and systems test plan.

• Types of use cases: High-level – are abstract use cases that are the basis

for being extended to end-to-end use cases.

Functional sub-use cases – are aggregated into end-to-end system-level use cases.

End-to-end – represents a complete transaction in the operation of the system.

CEN 5076 Class 3 - 09/19 46

Use Case Test Plan• Other uses cases:

Report – access information in the system, summarize it, and format it for presentation to the user.

Boundary – describe startup, shutdown, and exceptional conditions.

• Figs 3.18, 3.19, and 3.20, Pgs 102, 103, and 104.

CEN 5076 Class 3 - 09/19 47

Integration Test Plan• Very important in the incremental development

environment.

• Integrates individual classes into a cluster (component), components into subsystems, subsystems into the system.

• In most cases there is a need for test drivers and stubs.

• Integration test provides information on the order of testing the individual classes, components, subsystems.

CEN 5076 Class 3 - 09/19 48

Integration Test Plan

• Test cases span the parts of the system being integrated.

• More complex and comprehensive than the typical unit test.

• Format follows that of the system test plan (Fig. 3.21 P. 105)

CEN 5076 Class 3 - 09/19 49

System Test Plan

• Summarizes the individual use case test plans and provides information on additional types of testing .

• Note that the most of the information provided by the IEEE test plan format have already been captured in the individual test plans.