Risk Based Testing

Ladislau Szilagyi

www.EuroQST.ro

-Why we need RBT ?

-Types of risks

-Managing risks

-Methods of evaluation & risk analysis

-Costs and benefits

Definitions (ISTQB glossary)

Risk = a factor that could result in future negative

consequences; usually expressed as impact and

likelihood.

Risk based testing = a testing strategy intended

to reduce the level of product risks and inform the

stakeholders of their status, starting in the initial

stages of a project. It involves the identification of

product risks and the use of risk levels to guide the

test process.

Safety = the capability of the software product to

achieve acceptable levels of risk of harm to

people, business, software, property or

environment in a specified context of use.

Risk outcome

Risk = negative outcome? … not always …

• Positive risks are opportunities, desired by both

the Project Manager and the stakeholders, and

may positively affect the project ( such as

increasing the ROI or finishing the project ahead

of time ).

• … But, positive risk may generate negative risks

( finishing a part of the project way before

schedule will create a lot of slack, as other

resources are not scheduled to work on the

project until much later ).

Risk types

• Product risk

– Functional

– Non-functional

• Project risk

– External

– Organizational

– Technical

Risk types

Project risk categories

• External:

– service provider related issues;

– client related issues;

…

Risk types

…

• Organizational: – skill and staff shortages;

– personal and training issues;

– political issues, such as: • problems with testers communicating their needs and

test results;

• failure to follow up on information found in testing and reviews (e.g. not improving development and testing practices).

– improper attitude toward or expectations of testing (e.g. not appreciating the value of finding defects during testing).

…

Risk types

…

• Technical:

– problems in defining the right requirements;

– the extent that requirements can be met given

existing constraints;

– the quality of the design, code and tests.

Risk dimensions

• Likelihood = the estimated probability that a risk

will become an actual outcome or event

• Impact = the damage that will be caused if the

risk become an actual outcome or event

Risk dimensions

Risk dimensions are depending on the context:

• Automotive: – Exposure (the relative expected frequency of the

operational conditions in which the damage can possibly happen)

– Control (the relative likelihood that the user can act to prevent the damage)

– Severity of the damage

• Avionics: – Threat

– Vulnerability

– Consequences

Risk factors

• Technical

– Complexity of technology and teams

– Personnel and training issues among the business analysts, designers, and programmers

– Conflict within the team

– Contractual problems with suppliers

– Geographical distribution of the development organization

– Legacy versus new approaches

– …

Risk factors

– …

– Tools and technology

– Bad managerial or technical leadership

– Time, resource and management pressure

– Lack of earlier quality assurance

– High change rates

– High earlier defect rates

– Interfacing and integration issues

Risk factors

• Business

– Frequency of use of the affected feature

– Damage to image

– Loss of business

– Potential financial, ecological or social losses or liability

– Civil or criminal legal sanctions

– Loss of license

– Lack of reasonable workarounds

– Visibility of failure leading to negative publicity

Risk options

• Ignore

• Assume

• Delegate

• Mitigate

• Contingency planning

RBT activities

• Establish the context

• Risk identification

• Risk analysis – Risk assessment

– Risks sorting

• Risk mitigation

• Risk contingency

• Risk reporting

Establishing the context

• Study the business domain

• Identify the stakeholders

• Define a framework

• Planning of the remaining activities

Product Risk Identification

• Risk statement

“Something may fail in some way due to some circumstances”

– Something – The component or feature where the

problem could occur

– Fail in some way – What potential failure

– Some circumstances – The reasons or vulnerabilities, why we are concerned

Product Risk Identification techniques

• Expert interviews

• Project matrix

• Independent assessment

• Use of risk templates

• Lessons learned

• Risk workshops

• Brainstorming

• Checklists

Product Risk Analysis techniques

• Informal – fast, cheap, but not accurate – Pragmatic Risk Analysis and Management (PRAM)

– Systematic Software Testing (SST)

– Product Risk Management (PRisMa)

Consequence

Pro

bab

ilit

y

Low

Low

High

High

1

2

3

4

Product Risk Analysis techniques

• Formal – accurate, but takes time, expensive – Cost-of-exposure

– Quality function deployment

– Fault tree analysis

– Hazard analysis

– Failure Mode Effect Analysis

FMEA - Example of formal Risk Analysis technique

• Failure Mode Effect Analysis – Define the System

– Identify Potential Failure Modes & Their Causes

– Evaluate the Effects on the System of Each Failure Mode

– Identify Failure Detection Methods

– Identify Corrective Measures for Failure Modes

3 factors:

– Severity = The criticality of the effects of bugs in this failure mode, should any exist, from 1 (most damaging) to 5 (least damaging),

– Likelihood = The probability of—and extent of impact associated with—bugs included in this failure mode, from 1 (most probable) to 5 (least probable).

– Priority = The importance of fixing bugs in this failure mode, should any exist, based primarily on the ability of the delivered system to meet customer needs, though also on logistical project issues, regulatory or standards compliance, or other business considerations, from 1 (most important to fix) to 5 (least important to fix).

Product Risk Analysis techniques

Product Risk Mitigation techniques

• Non-testing related

• Testing related

– Choosing an appropriate test design technique

– Reviews & inspection

– Reviews of test design

– Level of independence

– Most experienced person

– The way re-testing is performed

– Regression testing

RBT activities

Project Risk Mitigation techniques

• Non-testing related

• Testing related

– Early preparation of test ware

– Pre-testing test equipment

– Pre-testing earlier versions of the product

– Tougher entry criteria

– Requirements for testability

– Participation in reviews of earlier project results

– Participation in problem and change management

– Monitoring of the testing progress and quality

Questions?