CTFL Chapter 1 Fundamentals

7/24/2019 CTFL Chapter 1 Fundamentals

1/80

MSTB-GTB 2011Version 1.0

Chapter

Software Testing FoundationsCertified Tester

Fundamentals of Testing

1


2/80

Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 2

After this lecture you should.

Be able to use examples to describe the way in which a software defect can cause harm topeople, to the environment, or to a company;

Know what is meant by the terms deficiency, defect, failure, defect condition / defect and errorand explain this with examples and compare;

Be able to distinguish between the cause of a defect and its effects;

Derive by examples why testing is necessary;

Be able to describe why testing is part of quality assurance and give examples of how testingcontributes to a higher quality;

Know the quality characteristic according to ISO 9126;

Recall the general objectives and principles of testing;

Differentiate between testing and debugging;

Know how the fundamental test process looks like;

Describe how expected values are calculated during testing; Know and characterize the psychological problems during the testing;

Differentiate between the different mindset of a tester and a developer;

Know the code of ethics of a software tester.


3/80


Learning Objectives for Fundamentals of Testing(according to the Certified Tester Foundation Level Syllabus, Version 2011)

1.1 Why is Testing Necessary? (K2) LO-1.1.1 Describe, with examples, the way in which a defect in software can cause

harm to a person, to the environment or to a company (K2)

LO-1.1.2 Distinguish between the root cause of a defect and its effects (K2)

LO-1.1.3 Give reasons why testing is necessary by giving examples (K2)

LO-1.1.4 Describe why testing is part of quality assurance and give examples of howtesting contributes to higher quality (K2)

LO-1.1.5 Explain and compare the terms error, defect, fault, failure and thecorresponding terms mistake and bug, using examples (K2)

1.2 What i s Testing? (K2)

LO-1.2.1 Recall the common objectives of testing (K1) LO-1.2.2 Provide examples for the objectives of testing in different phases of the

software life cycle (K2)

LO-1.2.3 Differentiate testing from debugging (K2)


4/80


Learning Objectives for Fundamentals of Testing(according to the Certified Tester Foundation Level Syllabus, Version 2011)

1.3 Seven Testing Principles (K2) LO-1.3.1 Explain the seven principles in testing (K2)

1.4 Fundamental Test Process (K1)

LO-1.4.1 Recall the five fundamental test activities and respective tasks from planningto closure (K1)

1.5 The Psychology of Testing (K2)

LO-1.5.1 Recall the psychological factors that influence the success of testing (K1)

LO-1.5.2 Contrast the mindset of a tester and of a developer (K2)


5/80


Terms you should be familiar with

Incident

Error Guessing

Test Plan

M a s

t e r T e s

t P l a n

R i s k

T e s

t B a s

i s T e s

t C o n

d i t i o n

I n d e p e n

d e n c e

Fundamental Test Process

S o

f t w a r e

T e s

t

Error cause

Confirmation Test

Test Policy

Quality AssuranceDebugging

T e s

t D e s

i g n

T e s

t R u n

L o g

Regression Test

F a

i l u r e

ErrorTest Case

Defect

R e v i e w

RequirementError

T e s

t S u m m

a r y

R e p o r t

E x

i t C r i

t e r i a

T e s

t D a

t a

Testware

Test CoverageExhaustive Test

T e s t O

b j e c

t i v e

Q u a

l i t y

Testing Test Suite

Test Execution


6/80

Chapter


1

Fundamentalsof Testing

Terms and Motivation

Principles of Testing da

Fundamental Test Process Test Process

Test Cases, Expected Results and Test Oracles

Psychology of Testing

Ethics of Testing }


7/80


In General: What is a Defect or a Deficiency?

A situation can only be classified as defective if it is pre-defined, howthe expected, correct and therefore the non-defective situation shouldlook like.Defect

Is the inability to fulfill a specific requirement. It is a discrepancy

between the actual behaviour (during the execution of the tests oridentified operation) and the expected behaviour (in the testspecification or the stated requirements).Deficiency

Non-fulfilment of a requirement related to an intended or specifieduse. A deficiency is for example the impairment of usability, whilemeeting performance or failure to perform any reasonableexpectation.

Based on: DIN EN ISO 9000:2005Quality Management Systems Fundamentals and Terms (German)Beuth Verlag, Berlin, 2005


8/80


Connection:Error - Fault/Defect - Failure

Error

ISTQB Glossary:

Error: Human action that produces an incorrect result [after IEEE 610].

People make mistakes; they commit errors.

Example: A programmer commitsan error by re-using a

a piece of software which isnot intended in the contextof the current project(the Ariane 5 satellitelaunching rocket)


9/80Chapter 1 Software Testing Foundations Certified TesterPage 9 Copyright 2011 to MSTB/GTBV 1.0

Error Other Definitions

Other defintions:1. Human action (of the developer) that

produces an error condition in the software2. Human action (of the user) that produces

an unwanted result (failure) as aconsequence (misoperation)

3. Unknowingly, inadvertently or intentionallyperformed act, or omission which leads under givencircumstances (task, environment) to an impairment of therequired function of a product



ISTQB Glossary:

Defect: A flaw in a component or system that can cause the componentor system to fail to perform its required function, e.g. an incorrect statement ordata definition A defect, if encountered during execution, may cause afailure of the component or system.

Defect/Fault

in a program

Error


An error made by a person(e.g. a developer) can resultin a defect in the sofware.(Defect and Fault aresynonymous terms)

Example: After theprogrammer hasre-used the piece ofcode in the wrongcontext, the softwarecontains a defect/fault.

(the full definition follows on page 12 )




Failure

which appears

Defect/Fault

in a program

Error

ISTQB Glossary:Deviation of the component orsystem from its expected delivery,service or result [after Fenton].

A failure is the effect ofa defect when executinga program (test object)which appears to theoutside.

Example:The Ariane 5rocket crasheson its firstmission.Cost ~ $7 billion

Boom


12/80

Chapter 1 Software Testing Foundations Certified TesterPage 12 Copyright 2011 to MSTB/GTBV 1.0

Fault/Defect Failure Additional comments

Defect (full definition from ISTQB glossary): A flaw in acomponent or system that can cause the component or system tofail to perform its required function, e.g. an incorrect statement ordata definition. A defect, if encountered during execution, maycause a failure of the component or system.

Defect masking : An occurrence in which one defect prevents thedetection of another [after IEEE 610]

A failure can also be called a malfunction or an external defect can also (but rarely) be caused by cosmic radiation, electro-

magnetic fields or hardware failure. Such failures can causeslow execution, incorrect output or the termination of execution.

Debugging is the development activity that finds, analyzes andremoves the cause of the failure.


13/80


Testing Definition and Goals

The process consisting of all life cycle activities, both static anddynamic, concerned with planning, preparation and evaluation ofsoftware products and related work products to

determine that they satisfy specified requirements demonstrate that they are fit for purpose detect defects

Source: [ISTQB Glossary]


14/80


Validation and Verification - Definitions

Validation

Confirmation by examination and through provision ofobjective evidence that the requirements for a specificintended use or application have been fulfilled .

[ISO 9000] and [ISTQB Glossary]Did we implement the right system ?

Verification

Confirmation by examination and through the provisionof objective evidence that specified requirements havebeen fulfilled [ISO 9000] and [ISTQB Glossary]Did we implement the system right ?


15/80


What is Software Quality?

A feature or characteristic that affects an item's quality [IEEE 610]. A set of attributes of a software product by which its quality is

described and evaluated. A software quality characteristic may berefined into multiple levels of sub-characteristics [ISO 9126]*.

Quality attributes relate to requirements

Functional requirements (capabilities, interfaces,professionalism, ...) Non-functional requirements (quality and implementation

requirements, project-specific requirements, ...)*Remark:

The description of product quality attributes provided in ISO9126 isused as a guide to describing software quality attributes. Otherstandards, such as the ISO/IEC 25000 series, may also be of use.


16/80


Software Quality according to ISO/IEC 9126 (1)

Software Quality

ISO/IEC 9126: Rate of software products, quality characteristicsand guidance on usage

Quality in Use External andInternal Quality

Fulfillment of taskswithin theexpenditure limits

for users (time,costs, ...)

Fulfillment of taskswithin the limits ofrisk (life and

health,business...)

Subjective usersatisfaction withinthe context of use

Productivity Security Satisfaction

Fulfillment of taskswithin the

accuracy andcompleteness

limits

Effectiveness


17/80


Software Quality according to ISO/IEC 9126 (2)

Suitability

Accuracy

Interoperability

(Data) Security

Compliance

MaturityFault tolerance

Recoverability

Compliance

Understandability

Learnability

Operability

Attractiveness

Compliance

Time behavior /Performance

Resourceutilization

Compliance

Analyzability

Changeability

Stability

Testability

Compliance

Adaptivity

Installability

Co-Existence

Replaceability

Compliance

Software Quality

Functionality Reliability Usability Efficency Maintainability Portability

ISO/IEC 9126: Evaluation of software products, quality characteristics and guidance on usage

Quality in Use External andInternal Quality


18/80


External Quality Attribute: Functionality (1)

Existence of functions with specified attributes. These functions meet the specified or implied requirements. sub-attributes of the quality attribute functionality are described

below and on the next page.

Suitability

The capability of the software product to provide an appropriate setof functions for specified tasks and user objectives [ISO 9126].

Accuracy

The capability of the software product to provide the right or agreed

results or effects with the needed degree of precision [ISO 9126].

(Further sub-attributes of the quality attribute functionality follow)


19/80


External Quality Attribute: Functionality (2)

(Further sub-attributes of the quality attribute functionality)Interoperability

The capability of the software to interact with one or more specifiedcomponents or systems [after ISO 9126].

(Data) Security

Attributes of software that bear on its ability to preventunauthorised access, whether accidental or deliberate, to programsand data [after ISO 9126].

Compliance

Attributes of the software which demonstrate that the softwarefulfills application-specific standards or agreements or statutorydirectives or similar regulations. [after ISO 9126].

Note: this also applies to all other quality attributes (e.g. reliability)


20/80


External Quality Attribute: Reliability

Attributes that relate to the ability of the software to maintain a specifiedlevel of performance under given conditions and for a specified time period.

Maturity

The capability of the software product to avoid failure as a result ofdefects in the software [ISO 9126].

Fault tolerance

The capability of the software product to maintain a specified level ofperformance in cases of software faults (defects) or of infringementof its specified interface [ISO 9126].

Recoverability The capability of the software product to re-establish a specified level

of performance and recover the data directly affected in case of failure[ISO 9126].


21/80


External Quality Attribute Usability

Attributes that relate to the effort required to use, and on the individualassessment of such use by a specific or implied group of users.

Understandability

The capability of the software product to enable the user tounderstand whether the software is suitable, and how it can be usedfor particular tasks and conditions of use [ISO 9126].

Learnability

The capability of the software product to enable the user to learn itsapplication [ISO 9126].

Operability The capability of the software product to enable the user to operate

and control it [ISO 9126].


22/80


External Quality Attribute: Efficiency

Attributes that relate to the relationship between the performancelevels of the software and the amount of equipment used underspecified conditions.

Time behaviour / Performance

The degree to which a system or component accomplishes itsdesignated functions within given constraints regarding processingtime and throughput rate [after IEEE 610].

Resource utilization

The capability of the software product to use appropriate amounts

and types of resources, for example the amounts of main andsecondary memory used by the program and the sizes of requiredtemporary or overflow files, when the software performs its functionunder stated conditions [after ISO 9126].


23/80


Internal Quality Attribute: Maintainability (1)

Attributes that relate to the effort that is necessary to carry out softwarechanges.

Analyzability

The capability of the software product to be diagnosed for deficienciesor causes of failures in the software, or for the parts to be modified tobe identified [ISO 9126].

Changeability

The capability of the software product to enable specified modificationsto be implemented [ISO 9126].

(Further sub-attributes of the quality attribute maintainability follow)


24/80


Internal Quality Attribute: Maintainability (2)

(Further sub-attributes of the quality attribute maintainability)

Stability

The capability of the software product to avoid unexpected effects

from modifications in the software [ISO 9126].Testability

The capability of the software product to enable modified software tobe tested [ISO 9126].


25/80


Internal Quality Attribute Portability (1)

Attributes that relate to the ability of software to be transferred fromone environment to another.

Adaptability

The capability of the software product to be adapted for differentspecified environments without applying actions or means otherthan those provided for this purpose for the software considered[ISO 9126].

Installability

The capability of the software product to be installed in a specified

environment [ISO 9126].

(Further sub-attributes of the quality attribute portability follow)


26/80


Internal Quality Attribute: Portability(2)

(Further sub-attributes of the quality attribute portability)

Co-Existence The capability of the software product to co-exist with other

independent software in a common environment sharing commonresources [ISO 9126].

Replaceability

The capability of the software product to be used in place of

another specified software product for the same purpose in thesame environment [ISO 9126].


27/80


Quality Requirements

Quality requirements state what quality attributes the productshould have at what quality level.

The sum of all quality attributes and the specific variants

Not all quality attributes can be achieved equally well.

E.g. efficiency may come at the expense of maintainability Set priorities

In close consultation with clients and users. Quality requirements (with the exception of functionality) are

part of the non-functional requirements in the specifications.


28/80


Testing and Quality

Testing measures the quality e.g. based on the number of locatedfailures and defects.

Testing increases the quality indirectly, as defects are detectedwhich can be corrected prior to delivery.

Testing increases the process quality indirectly, as defects aredocumented which can then be analysed and help prevent similarerrors from taking place in the future.

Testing increases the confidence in the quality of the system whenfew or no failures and defects are found.


29/80


30/80

Chapter


1



Principles of Testing

Fundamental Test Process Test Process



Ethics of Testing }


31/80


Seven Principles of Testing (1)

In the previous 40 years, the following seven testing principles haveemerged and can serve as guidelines:

Principle 1: Testing shows presence of defects

Testing can show that defects are present, but cannot prove thatthere are no defects.Program testing can be used to show the presence of bugs,but never to show their absence! Edger W. Dijkstra, 1970

Principle 2: Exhaustive testing is impossible

Complete / exhaustive testing is not feasible except for trivialprograms (see slides in chapter 1 on exhaustive testing)

(continued)


32/80



Principle 3: Early testingTesting is not a phase in the software development that happens atthe end; it shall be started as early as possible. Through early testing(e.g. reviews) which take place parallel to the development activities,defects are detected earlier and thus cost less to fix.

Principle 4: Defect clustering

Testing effort shall be focused proportionally to the expected and theobserved defect density of modules. A small number of modules oftencontains most of the defects discovered during pre-release testing, oris responsible for most of the operational failures.(continued)


33/80



Principle 5: Repetitions have no effects - Pesticide ParadoxJust repeating tests brings no new insights. Test cases need to beregularly reviewed, revised and modified.

Principle 6: Testing is context dependent

Testing is depending on context. For example, safety-critical softwaresystems are tested differently (more intensely and using othertechniques) from commercial applications.

Principle 7: Absence-of-errors fallacy

A system without failures does not imply that the system will meet theusers expectations.


34/80


Testing Effort?

Testing is economically useful, as long as the costs of finding andfixing a defect in testing are lower than the costs that areassociated with the occurrence of a defect when used.

A good test is like a liability insurance: it costs big money, butallows the project manager and the customer to sleep peacefully. Agood sleep includes a good insurance policy that covers allpossible risks. The trust in a software product includes a good testthat fully covers the reality of production.

Successful testing (detection of failures) reduces costs

M. Pol, T. Koomen, A. Spillner:Management and Optimization of the Test Process(German). dpunkt-Verlag, 2. Edition, 2002

Siedersleben, J. (Hrsg):Software methodology: Practical Knowledge forSoftware Engineers, (German). 2. Edition, Hanser,2003


35/80


More regarding Testing (1)

Testing effort in practice: 25% to 50% of the development effort

Test intensity and scope depends on risks and the criticality of theproject, the system and the domain

Defects can result in huge costs: Estimated losses due to software defects in medium and large

companies in Germany are approximately 84.4 billion Euros perannum

Productivity losses due to computer failures caused by softwaredefects are approximately 2.6% of sales - 70 billion Euros perannum

Study by LOT Consulting, KarlsruheIT-Services 3/2001, P. 31 (German)


36/80


More regarding Testing (2)

Testing is always subject to limited resources(e.g. time, test personal)

Particularly important:

Select appropriate test procedures which are compatible withthe test objectives and quality objectives

Avoid unnecessary tests which provide no new information

Take positive and negative test conditions into account

It can also be useful to test for functionality that has not beenrequested


37/80


Prioritization of Tests

Criteria for priorit ization: Expected impact (the severity of the damage a defect would cause) Probability of occurrence of a failure

Combined consideration of impact and likelihood of occurrence( Risk = Likelihood * Impact) Perception of the failure Prioritization of requirements by the customer Importance of quality characteristics by the customer

Priority of test cases from the perspective of development (seriousconsequences and / or complex cases first) High project risk Where many defects are found, more defects are likely to be

discovered! A change in the priority must be possible!

Limited resources (time and personnel) require that the most importanttest cases are carried out first!


38/80

Chapter


1







Ethics of Testing }


39/80


SW-Development Models: Waterfall-Model - 1970

Royce, W. W.:Managing the Development of Large SoftwareSystems: Concepts and TechniquesProc. WESCON, IEEE Computer Society Press, Los

Alamitos, CA, 1970.Reprinted at the ICSE'87, Monterey, California, USA.March 30 - April 2, 1987


40/80


SW-Development Models: Waterfall-Model - 1981

Boehm, B.:Software Engineering EconomicsPrentice-Hall Inc., London, 1981


41/80


V-Model (B. Boehm, 1979)

Barry W. Boehm:Guidelines for Verifying and Validating SoftwareRequirements and Design Specification.EURO IFIP 79, P.A. Samet (eds.) North-Holland, IFIP1979, 711-719


42/80


General V-Model

FunctionalSystem Design

TechnicalSystem Design

Programming

Defini tion of Requirements

ComponentSpecification

System Testing

Integration Testing

Acceptance Testing

Component Testing

KeyTest cases based oncorresponding documents


43/80


Test Process

Is closely linked with software development Is, however, a separate, independent process

A test plan is necessary for every test level (level test plan)

Testing cannot be considerd as a single activity (e.g. test execution)

Many individual tasks are performed within testing

The test process represents these individual testing activities in a coherentprocess


44/80


Activities of the Fundamental Test Process

Test Planning and Control Test Analysis and Design

Test Implementation and Execution

Evaluating Exit Criteria and

Reporting Test Closure Activities

These activities may overlap or take

place concurrently. The Fundamental Test Process is to

be tailored to each test level (e.g.module test, system test)

Planning and

Analysis and Design

Implementation andExecution

Closure

Start

Finish

Control

Evaluation andReport


45/80


Testing Tasks: Test Planning and Control (1)

Specification of Test Plan Determine test objectives, amount and

risks of testing. Specify test approach (techniques*,

test objects, coverage, teams whoparticipate in testing, testware).

Details of Test plan Test resources (e.g. staff, test environment, PCs). How intensive should particular parts of the system be tested? Which test techniques should be used? Determine exit criteria. What level coverage should be reached?

Prioritizing of test (taking defect risks into consideration). Planning for tool support (use, acquisition, training). Test schedule and test strategy are to be recorded in the test plan

Planning and

Analysis andDesign

Implementationand Execution

Test Closure

Start

Finish

Control


*refer to chapters 4 and 5


46/80


Testing Tasks: Test Planning and Control (2)

Drawing up a rough schedule Schedule tasks for test analysis and test

specification. Schedule test implementation, test

execution and test evaluation.

Test Control Measuring and analysing results. Monitoring and recording of test progress,

achieved test coverage and exit criteria. Initiate corrective actions.

Adapting time and resource planning. Making decisions.

Refer to chapter 6 for more details

Planning and

Analysis andDesign


Test Closure

Start

Finish

Control



47/80


Testing Tasks: Test Analysis and Design (1)

General testing objectives are detailled to specifictest requirements and criteria.

Basis are all documents showing therequirements for the software (test basis).

Reviewing the test basis (requirements,architecture, design, interface, ...).

Evaluating testability of requirements and the system. Identifying test conditions / test requirements. Designing the test environment (infrastructure, tools, ...). Designing test cases using the chosen test design techniques. Distinction to be made between high level test cases and detailed

test cases. Creating bi-directional traceability between test basis and test

cases

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

S


48/80


Testing Tasks: Test Analysis and Design (2)

Test cases contain more that just test data! Criteria and constraints that have to be met

before execution (e.g. condition of testobject, data, network status)

Before test execution determine which resultor behavior is expected and the post-conditions,that have to be fulfilled after the test, e.g. conditionof the test object and the database.

Refer to the next section in this chapter for further details Test cases are partly linked together to create test suites (test

procedure specification) for test execution Each test case should only contain one or a few elementary

process steps or system calls Pay attention in the test design to the post-conditions of the

test cases. These are the pre-conditions of next test cases.

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

St t


49/80


Testing Tasks: Test Implementation

Specifying and prioritizing test cases. Creating test data and test scenarios.

Creating test suites.

Preparing test harnesses and (possibly) writingautomated test scripts.

Verifying that the test system and test environment has been setup correctly.

Before test execution: Verifying completeness of parts to be tested.

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

Start


50/80


Testing Tasks: Test Execution (1)

First check main functions. If any incidentsare found here, continuing with deeper testsmay not make much sense!

Execute test cases either manually orby using test execution tools according to thetest procedure specification

Record the test execution accurately and completely (versions oftest object, test tools and testware ) in the test log.

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

Start


51/80



Compare actual results with expected results. Report discrepancies as incidents and analyze

them. If a discrepancy of actual and expectedresult is detected, it may not always meana software defect has been detected.

Check if any of the following aspects is true: incorrect expected result test environment not set up properly the test case specification is incorrect

Determine defect severity and priority for fixing Repeat test activities as a result of action taken to resolve each

incident/defect ( Re-Testing ).

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

Start


52/80



Create a test log The test log must contain details on which

parts were tested, when, by whom,how intensive and with what result.

On the one hand the description of testexecution in the test log has to becomprehensive for those not directly involved(e.g. clients)

On the other hand it has to be traceable if the planned test strategyhas really been applied

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

Start


53/80


Testing Tasks: Evaluating Exit Criteria

Compare results of test execution with thedefined objectives of the test.

Checking test logs against the exit criteriaspecified in test planning.

Reached test end? Achieved coverage?

(Problem: unreachable program code) Deciding if more tests are needed or if the specified exit criteria

should be changed (only after consulting stakeholders!) Further effort justified?

Addition practical factors for ending the test: No more time No more money

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

Start


54/80


Testing Tasks: Test Reporting (1) Analysis and

Design


Test Closure

Start

Finish


Planning and

Control

Write a test summary report forstakeholders.

The test summary report may have twopurposes:

Communicate test status

Report on test completion Refer to chapter 6 for more details on test reporting

Start


55/80


Testing Tasks: Test Reporting (2)

Example for exit criteria:Found defects in order of failure severity

0

0,5

1

1,5

2

2,5

3

W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11

New / Testing Hour

low

medium

high

critical

Failure Severity

Wx: Calendar Week

Analysis andDesign


Test Closure

Start

Finish


Planning and

Control

Start


56/80


Testing Tasks: Test Reporting (3)

Example for exit criteria:Comparison of found failures and corrected defects

0

10

20

30

40

50

60

W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11

New failures

Corrected defects

In Process

Wx: Calendar Week

Analysis andDesign


Test Closure

Finish


Planning and

Control

Start


57/80


Testing Tasks: Test Closure Activities (1)

Collect data from completed test phases andconsolidate experience, testware, facts, metricsetc.

Check which planned deliverables have beendelivered

Close incident reports or raise change requestsfor any that remain open

Document the acceptance of the system

Analysis andDesign


Test Closure

Finish


Planning and

Control

Start


58/80


Testing Tasks: Test Closure Activities (2)

Finalize and archive testware, thetest environment and the test infrastructure

Preserve and hand over the testware to themaintenance organization everything shouldbe reusable during maintenance, so it has to be

portable and updatable Analyse and document any lessons learned

for later projects and to improve test maturity Evaluation of test process Assessment of test process

Recognizing potential for improvement Improve the test process by applying the

findings

Analysis andDesign


Test Closure

Finish


Planning and

Control


59/80

Chapter


1





Test Cases, Expected Results, Test Oracles ch\


Ethics of Testing }


60/80


Criteria for Test Cases

Test cases for verification of specified results and of test objectdelivered results and reactions Positive test; expected input / operation ( normal )).

Test cases that verify the specified handling of exceptionalsituations and defects also need to be considered Negative test; expected false input / operation ( exceptional ). Remark: It is often difficult to create the conditions necessary

for execution of negative test cases (e.g. the overload of anetwork connection).

Test cases for verification of reactions of the test object for invalidand unexpected inputs or constraints, for which there was noexception handling specified Negative test / robustness test; unexpected erroneous inputs /

operations ( catastrophic ).

Test Specification


61/80


Test Specification High Level and Specific Test Cases (1)

Example: A company has ordered a program that should calculate theChristmas bonus of the staff in relation to the time they have beenworking there. In the description of the requirements you find thefollowing text:

Staff who have been with the company for more that three years willreceive 50 % of the monthly salary as Christmas bonus. Staff whohave been with the company for more than five years will receive 75%. Staff who have been with the company for more than eight years

will receive 100 % of their monthly salary.

How do the test cases look like?

Test Specification


62/80



From the text you can set up the following relationship between theallowance of the bonus and the time working for the company:

years with the company


63/80



High Level (logical)Test Case

1 2 3 4

Input value x(years with the company)

x


64/80


Expected Results and Test Oracle

After each excecuted test case it must be decided whether there isa failure or not.

For this decision the monitored result (actual result/behavior) hasto be compared with the expected result (expectedresult/behavior).

The expected behavior of the test object has to be determined inadvance for each test case.

The tester must obtain this information from appropriate sourceswhen specifying a test case.

In this connection we also speak of an oracle or test oracle thatcan be consulted and that predicts the expected results.


65/80


The expected results are to be deducted of the specification.

Following three possibilities: The tester derives the expected date from the input date on the basis

of the specification of the test object. This is the common practise. If a formal specification is available, an executable prototype can be

created with tool support. The results of the prototype can be usedas a test oracle for testing the actual program.

Parallel designs of the software program may be created by differentdevelopment groups. Each version of the program will then be testedagainst another version using the same test data. If there aredifferent results in the two versions there must be a failure in one of

the versions. Only those failures that have the same effect in all theversions remain undetected. This procedure is called Back-to-backtest.

Expected Results and Test Oracle


66/80


Other Test Oracles

Making use of user manuals The system itself (create operational profile)

This is the worst possibility

(Tested!) previous versions

Programs with similar functionality Exact values cannot always be predicted or calculated. Determine

range of tolerance, verify plausibility

Experience is important


67/80

Chapter


1





Test cases, Expected Results, Test Oracle


Ethics of Testing }


68/80



Making mistakes is human- but who likes admitting that?

Development = constructiveTest = destructive?

Testing is an extremely creativeand intellectually challenging task

Is it possible for developer to test

his own program?

What to you think?

Myers, Glenford J.:Systematic Testing of ProgramsOldenbourg, 2001 (7. Edition)


69/80


Psychology of Testing Developer Test

Being blind to see own mistakes If the developer has implemented a basic design error, e.g.

because he has misunderdstood the task, he is not able todetect this through his own tests.

He will not be able to specify appropriate test cases.

No familiarisation The developer knows his test object


70/80


Psycholoy of Testing - Independent Test Team

An independent test team is unbiased It his not his/their product The possible assumptions and misunderstandings of the developer

are not necessarily the assumptions and misunderstandings of thetester.

Need for familiarization To create test cases the tester needs to gain knowledge about the test

object. This requires time.

Test Know-how The tester should have test know-how A developer probably does not have this or first has to gain it (often

with not enough time available)

Even better, the know-how has already been acquired at university


71/80


Psychology of Testing Possible Levels

Tests are carried out with different levels of independence By the developer himself By colleagues of the developer in the same project By persons of other departments

By persons of other organizations. Tools can be used to support all possibilities.

The selection depends on the product as well as the project.

It is important to get the right mix and balance between

independent testing and tests performed by the developer.


72/80


Psychology of Testing Communication of Defects

Communication of identified defects or incidents to the developerand/or the management requires

neutral, fact-focused and constructive way of communication Undisturbed, open communication.

Proving another persons error is not an easy task. It requires goodinterpersonal skills.

Reproduceability of defects is important! Recording the test environment Differences to the development environment

Clear requirements, precise specification Its not a bug, its a feature!

Mutual understanding between testers and developers Collaboration rather than battles!


73/80

Chapter


1





Test cases, Expected Results, Test Oracle


Ethics of Testing }


74/80


Ethics of Testing (1)

Software testers often have access to confidential and legally privilegedinformation. This must not be used inappropriately.

Recognizing the ACM and IEEE code of ethics for engineers, the ISTQB states the following code of ethics :

1. PUBLIC

Certified software testers shall act consistently with the public interest.

2. CLIENT AND EMPLOYERCertified software testers shall act in a manner that is in the best interestsof their client and employer, consistent with the public interest.


75/80



3. PRODUCTCertified software testers shall ensure that the deliverables they provide(on the products and systems they test) meet the highest professionalstandards possible.

4. JUDGMENTCertified software testers shall maintain integrity and independence intheir professional judgment.

5. MANAGEMENTCertified software test managers and leaders shall subscribe to andpromote an ethical approach to the management of software testing.


76/80



6. PROFESSIONCertified software testers shall advance the integrity and reputation ofthe profession consistent with the public interest.

7. COLLEAGUESCertified software testers shall be fair to and supportive of theircolleagues, and promote cooperation with software developers.

8. SELFCertified software testers shall participate in lifelong learning regardingthe practise of their profession and shall promote an ethical approach tothe practice of the profession.


77/80


Summary

Failure Not fulfilling the requirements! Describing this situation as failure cause: fault/defect in the software that has been caused by an error made by a person

Tests are important measures for securing the quality criteria

Functionality, reliability, usability, efficiency, maintainability undportability [according to ISO 9126]

Exhaustive testing is not possible. Tests are always randomchecks, therefore they can detect failures only with a certainprobability

The intensity and amount of testing depends on the expected riskof faulty behavior of the program

Tests should be prioritized


78/80


Summary

The fundamental test process consists of the main activities

Test Planning and Control Test Analysis and Design

Test Implementation and Execution

Evaluating Exit Criteria and Reporting

Test Closure Activities

A test case consists of

an input value and

an expected result as well as

preconditions and constraints for the execution of the test case, and

postconditions that have to be fulfilled after the test case.

While executing the test case, the test object shows its actual behavior. If there is a discrepancybetween expected and actual behavior there is a failure.

A test oracle determines the expected values for each test case before test execution

Humans make mistakes but they do not like admitting them!

By now, you should be able to answer the following


79/80

? ? ?


questions

Define the terms software failure, defect, error.

What is defect masking?

Explain the difference between testing and debugging.

Define the terms verification and validation.

Explain why tests cannot be exhaustive. Name the main software quality attributes according to ISO 9126.

Define the term reliability of a system.

Explain the main activities of the fundamental test process.

What is a test oracle?

Why should developers not test their own programs?


80/80

Finally

The process cannot be stopp ed. The process cannot be stopped.

Copy file:

Installation

Sort results

1 entry was found.Would you like to sort th e results?

Yes No

Cancel

Documents

CTFL Chapter 1 Fundamentals