Chapter 1:Fundamental of

TestingSystems Testing & Evaluation (MNN1063)

Fundamental of Testing

0 • Why is testing necessary?0 • What is testing? 0 • General testing principles 0 • Fundamental test process0 • The psychology of testing0 • Code of ethics

Why is testing necessary?

0Human make mistakes0Mistakes – unimportant, expensive or dangerous0Must check – some things can always go wrong0Mistakes may comes from:

0 Bad assumption0 Blind spot

Software Systems Context

0Software that did not work as expected?0 Error on a bill, delay on credit card processing, website

that did not load correctly etc.0Different software carry different level of risk0Problems – trivial, costly and damaging (loss of

money, time or business reputation, or results in injury or death

RISK: A factor that could result in future negative consequences; usually expressed as impact and likelihood

Causes of Software Defects

0People who design or build the software can make mistakes

0Mistakes means flaws in the software => defects/bugs/faults

0Defects may caused system to fail to do what it should do (or do something it shouldn’t) => causing failure

Error(Mistakes)

Defects(Bugs/Faults)

Failure

Do our mistakes matter?

0Think about the consequences of mistakes…0Human being, programmer, tester => can make error0Error => defects => defect code executed => systems

failure0Systems and project are complicated

Defects & Failures

0Errors in specifications, design and implementation0Errors in use of the systems0Environmental conditions0 Intentional damage0Potential consequences of earlier errors, intentional

damage, defects and failures

When do defects arise?Requirement 1

Correct requirement

Designed to meet requirement

Build to meet design

Product works as expected

Requirement 2

Correct requirement

Designed to meet requirement

Mistakes made in build

Product has bugs in it

When do defects arise?Requirement 3

Correct requirement

Mistake made in design

Built to meet design

Product has design flaws

Requirement 4

Mistakes made in requirement

Design to meet requirement

Built to meet design

Wrong product delivered

What is cost of defects?0The cost of finding and fixing the defects rises

considerably across the life cycle

Cost

Time

Cost of fi

nding and fixing defects in

creases

over time

Requirements Design Build Test Live use

Role of Testing in Software Development, Maintenance & Operations

0We have seen human errors can caused defect or fault at any stage of software development life-cycle

0Rigorous testing is necessary during the maintenance & development to identify defects => to reduce failures in operational environment

0To increase quality of operational systems0Executing tests helps us move towards improved

quality of product and services

Testing & Quality

0Testing helps us to measure quality of software in terms of:0 The number of defects found0 The number of tests run0 The system covered by the test

0Both functional and non-functional software requirements

0Testing gives us confidence in quality0Poor test may uncover few defects

What is Quality?

0Project aim to deliver software that meet customer needs & correct specification

0Validation – “ is this the right specification”0Verification – “is the system correct to specification”0Need to understand what customer understand by

quality and what their expectation are

What is root cause analysis?

0Detect failures => track them back to their root cause0Several ways & involving a group brainstorming ideas

and discussion0Assignment #1:

0 Find 3 ways to do Root Cause Analysis.0 Discuss how it works.

How much testing is enough?

0We have a choice:0 Test everything?0 Test nothing?0 Test some part?

0Example: One-digit field – how many test is enough?0Real example: one screen with 15 input fields0Pressures on project: time & budget0Need test approach which provides the right amount of

testing0Risk assessment – helps to decide how much testing

to do

What is testing?

0Recall the common objectives of testing0What is testing? – checking if the software is OK?0Lets discuss an analogy of: driving test0The format of the driving test and conduct of

examiner:0 The test is planned & prepared0 The test has known goals0 The test is therefore carried out to show that the driver

satisfied the requirement for driving & demonstrate that they are fit to drive

Defining software testing

0 ISTQB definition of software testing0Process – Testing is a process rather a single activity –

there are series of activities 0All life cycle activities – software development life

cycle (discuss this in Chapter 2)0 Requirements 0 Design (to prevent defects introduced into code) – test

via the test design

Defining software testing

0 Both static and dynamic 0 Dynamic testing – execute codes & demonstrate the results0 Static – reviewing documents including the source code

0 Planning – to manage the testing: plan, control, report testing progress, close testing

0 Preparation – selecting test conditions & design test cases0 Evaluation – check results and evaluate software under

test & completion criteria0 Software products and related work product – not only test

code, requirements & design specification

Defining software testing

0Objectives for testing0 Determine the software product satisfy specified

requirements0 Demonstrate that the software products are fit for

purpose0 Detect defects

0Objectives may be different in each stage

General testing principles

0Principle 1: Testing shows the presence of defects0Principle 2: Exhaustive testing is impossible0Principle 3: Early testing0Principle 4: Defect clustering0Principle 5: Pesticide paradox0Principle 6: Testing is context dependent0Principle 7: Absence-of-errors fallacy

Fundamental test process

0Fundamental test process:0 Planning and control0 Analysis and design0 Implementation and execution0 Evaluation exit criteria and reporting0 Test closure activities

Test Planning and control

0We make sure we understand the goal and objectives of the customer, stakeholders, the project and the risks

0Based on the understanding, we set the goal & objectives for the testing itself

0To help us, we may have:0 Test policies – rules for testing0 Test strategy – Overall high-level approach

Test planning and control

0 Test planning has the following major task:0 Determine the scope and risks; Identify the objectives of

testing0 Determine the test approach (techniques, test items,

coverage, identifying and interfacing the teams involve in testing, testware)

0 Implement the test policy and/or the test strategy0 Determine the required test resources (people, test

environment, PCs)0 Schedule test analysis and design tasks, test implementation,

execution and evaluation0 Determine the exit criteria

Test planning and control

0Test control is an on-going activity0We compare the actual progress with planned

progress0Test control has the following major tasks:

0 Measure & analyze the results of reviews and testing0 Monitor and document progress0 Provide information on testing0 Initiate corrective actions0 Make decisions

Test Analysis and Design

0Test analysis and design has the following major tasks:0 Review the test basis (product risk analysis,

requirements, architecture, design specification and interface)

0 Identify test conditions (behavior and structure)0 Design the test (Chapter 4)0 Evaluate testability of the requirements and systems0 Design the test environment set-up and identify any

required infrastructure and tools

Test implementation and Execution

0 Implementation:0 We take the test conditions and make them into test

cases and set up the test environment0 Having the high-level design – we now build them0 We set up the environment0 Build our test data

0Execution:0 Execute the test cases0 Log the outcome of test execution0 Compare actual result with expected result

Evaluating exit criteria and reporting

0 Is an activity where test execution is assessed against the defined objectives

0Evaluating exit criteria involve following major tasks:0 Check the test logs against the exit criteria0 Assess if more test needed0 Write a test summary report