SENG521 (Fall 2002)far@enel.ucalgary.ca1 SENG 521 Software Reliability & Testing Software...

SENG521 (Fall 2002) far@enel.ucalgary.ca 1

SENG 521SENG 521Software Reliability & Software Reliability & TestingTesting

Software Reliability ModelsSoftware Reliability Models

(Part 2a)(Part 2a)

Department of Electrical & Computer Engineering, University of Calgary

B.H. Far （ far@enel.ucalgary.ca ）http://www.enel.ucalgary.ca/~far/Lectures/SENG521/02a/

SENG521 (Fall 2002) far@enel.ucalgary.ca 2

ContentsContents Basic Features of the Software Reliability ModelsBasic Features of the Software Reliability Models Single Failure ModelSingle Failure Model Reliability Growth ModelReliability Growth Model Exponential Failure Class ModelsExponential Failure Class Models Weibull and Gamma Failure Class ModelsWeibull and Gamma Failure Class Models Infinite Failure Category ModelsInfinite Failure Category Models Bayesian ModelsBayesian Models Early Life-Cycle Prediction ModelsEarly Life-Cycle Prediction Models

SENG521 (Fall 2002) far@enel.ucalgary.ca 3

Section 1Section 1

Basic Features

SENG521 (Fall 2002) far@enel.ucalgary.ca 4

GoalGoal It is important to be able to

Predict probability of failure of a component or system

Estimate the mean time to the next failure Predict number of (remaining) failures

during the development. Such tasks are the target of the reliability

management models.

SENG521 (Fall 2002) far@enel.ucalgary.ca 5

Reliability ModelReliability Model

ReliabilityReliabilityModelModel

ReliabilityReliabilityModelModel

Fault introduction:

Characteristics of the product (e.g., program size)

Development process (e.g., SE tools and techniques, staff experiences, etc.)

Fault removal:

Failure discovery (e.g., extent of execution, operational profile)

Quality of repair activity

Environment

SENG521 (Fall 2002) far@enel.ucalgary.ca 6

Two Reliability QuestionsTwo Reliability Questions

Single failure specification: What is the probability of failure of a system

(or a component)?

Multiple failure specification: If a system (or a component) fails at time t1,

t2, …, ti-1, what is the probability of its failure at time ti?

SENG521 (Fall 2002) far@enel.ucalgary.ca 7

Model ClassificationModel Classification Time domain: Calendar or execution time Category: Number of failures is finite or infinite. Type: Distribution of the number of failures

experienced by the time specified. Class (only finite category): Functional form of

the failure intensity over time. Family (only infinite category): Functional form

of the failure intensity in terms of the expected number of failures experienced.

SENG521 (Fall 2002) far@enel.ucalgary.ca 8

Various Reliability Models Various Reliability Models /1/1 Exponential Failure Class ModelsExponential Failure Class Models

Jelinski-Moranda model (JM) Nonhomogeneous Poisson Process model

(NHPP) Schneidewind model Musa’s Basic Execution Time model (BET) Hyperexponential model (HE) Others

SENG521 (Fall 2002) far@enel.ucalgary.ca 9

Various Reliability Models Various Reliability Models /2/2 Weibull and Gamma Failure Class ModelsWeibull and Gamma Failure Class Models

Weibull model (WM) S-shaped Reliability Growth model (SRG)

Infinite Failure Category ModelsInfinite Failure Category Models Duane’s model Geometric model Musa-Okumoto Logarithmic Poisson model

SENG521 (Fall 2002) far@enel.ucalgary.ca 10

Various Reliability Models Various Reliability Models /3/3 Bayesian ModelsBayesian Models

Littlewood-Verrall Model

Early Life-Cycle Prediction ModelsEarly Life-Cycle Prediction Models Phase-based model

SENG521 (Fall 2002) far@enel.ucalgary.ca 13

Another Categorization /1Another Categorization /1 Time between failure modelsTime between failure models

Jelinski-Moranda (Exponential Failure Model) Musa-Basic (Exponential Failure ModelModel) NHPP (Exponential Failure ModelModel) Geometric (Infinite FailureInfinite Failure ModelModel) Musa-Okumoto (Infinite FailureInfinite Failure ModelModel) Littlewood-Verrall (Bayesian ModelBayesian Model)

SENG521 (Fall 2002) far@enel.ucalgary.ca 14

Another Categorization /2Another Categorization /2 Failure count modelsFailure count models

Generalized Poisson Shick-Wolverton Yamada S-shaped NHPP (Exponential Failure ModelModel) Schneidewind (Exponential Failure ModelModel)

SENG521 (Fall 2002) far@enel.ucalgary.ca 15

Section 2Section 2

Single Failure Model

SENG521 (Fall 2002) far@enel.ucalgary.ca 16

Hardware Reliability Hardware Reliability ModelsModels

Uniform model: Probability of failure is

fixed.

Exponential model: Probability of failure

changes exponentially over time

1

0

t Tf t

t T

f

t

T

0tf t e

f

t

T

0

SENG521 (Fall 2002) far@enel.ucalgary.ca 17

Single Failure Model /1Single Failure Model /1 Probability Density Function

(PDF): depicting changes of the probability of failure up to a given time t.

A common form of PDF is exponential distribution

Usually we want to know how long a component will behave correctly before it fails, i.e., the probability of failure from time 0 up to a given time t.

Exponential PDF:

0

0 0

te tf t

t

SENG521 (Fall 2002) far@enel.ucalgary.ca 18

Single Failure Model /2Single Failure Model /2 Cumulative Density

Function (CDF): depicting cumulative failures up to a given time t.

For exponential distribution, CDF is:

0

tF t f t

1 0

0 0

te tF t

t

SENG521 (Fall 2002) far@enel.ucalgary.ca 19

Single Failure Model /3Single Failure Model /3 Reliability function (R): depicting a component

functioning without failure until time t.

For exponential distribution, R is:

1R t F t

tR t e

SENG521 (Fall 2002) far@enel.ucalgary.ca 20

Single Failure Model /4Single Failure Model /4 What is the expected value of failure time T? It is the mean of the probability density function (PDF),

named mean time to failure (MTTF)

For exponential distribution, MTTF is:

0

E T t f t dt

1MTTF

SENG521 (Fall 2002) far@enel.ucalgary.ca 21

Single Failure Model /5Single Failure Model /5 Median time to failure (tm): a point in time that the

probability of failure before and after tm are equal.

Failure Rate z(t): Probability density function divided by reliability function.

For exponential distribution, z(t) is: λ

0

1 12 2

mt

mt f t dt or F t

f tz t

R t

SENG521 (Fall 2002) far@enel.ucalgary.ca 22

Single Failure Model /6Single Failure Model /6 System Reliability: is the multiplication of

the reliability of its components.

For exponential distribution:

1

n

system ii

R t R t

1 21 2

1

nn

n

ii

ttt tsystem

t

system

R t e e e e

R t e

SENG521 (Fall 2002) far@enel.ucalgary.ca 23

Single Failure Model /7Single Failure Model /7 System Cumulative Failure Rate: is the

sum of the failure rate of its components.

For exponential distribution:

1

n

system ii

z t z t

1 21

n

system n ii

z t

SENG521 (Fall 2002) far@enel.ucalgary.ca 24

Section 3Section 3

Reliability Growth Model

SENG521 (Fall 2002) far@enel.ucalgary.ca 25

Reliability Growth Reliability Growth Models Models /1/1

One can assume that the probability of failure (probability density function, PDF) for all failures are the same (e.g., replacing the faulty hardware component with an identical one).

In software, however, we want to “fix” the problem, i.e., have a lower probability of failure after a repair (or longer Δti = ti-ti-1). Therefore, we need a model for reliability growth (i.e., reliability change over time).

SENG521 (Fall 2002) far@enel.ucalgary.ca 26

Reliability Growth Models Reliability Growth Models /2/2 Common software reliability growth models

are: Basic Exponential model Logarithmic Poisson model

The basic exponential model assumes finite failures (ν0) in infinite time.

The logarithmic Poisson model assumes infinite failures.

SENG521 (Fall 2002) far@enel.ucalgary.ca 27

Validity of the ModelsValidity of the Models Software systems are

changed (updated) many times during their life cycle.

The models are good for one revision period rather than the whole life cycle.

RevisionPeriod 1

RevisionPeriod 4

SENG521 (Fall 2002) far@enel.ucalgary.ca 28

Reliability Growth Models Reliability Growth Models /3/3 Parameters involved in reliability growth

models:1) Failure intensity (λ): number of failures per

natural or time unit.

2) Execution time (τ): time since the program is running. Execution time may be different from calendar time.

3) Mean failures experienced (μ): mean failures experienced in a time interval.

SENG521 (Fall 2002) far@enel.ucalgary.ca 29

Reliability Growth Models Reliability Growth Models /4/4

Mean failures experienced (μ) for a given time period (e.g., 1 hour execution time) is calculated as:

No. of failures in interval (n)

Probability

(p)n x p

0 0.10 0

1 0.18 0.18

2 0.22 0.44

3 0.16 0.48

4 0.11 0.44

5 0.08 0.40

6 0.05 0.30

7 0.04 0.28

8 0.03 0.24

9 0.02 0.18

10 0.01 0.10

Mean failure (μ) 3.04

1

: probability of occurrence

: number of failures

n

ii

i

i p

p

i

SENG521 (Fall 2002) far@enel.ucalgary.ca 30

Reliability Growth Reliability Growth Models /5Models /5

Failure intensity (λ) versus execution time (τ)

0

0

0

0

0

(B)

(P)1

e

0

0

Initial failure intensity

Total failures

Decay parameter

SENG521 (Fall 2002) far@enel.ucalgary.ca 31

Reliability Growth Reliability Growth Models /6Models /6

Failure intensity (λ) versus mean failures experienced (μ)

00

0

1(B)

(P) e

SENG521 (Fall 2002) far@enel.ucalgary.ca 32

Reliability Growth Reliability Growth Models /7Models /7

Mean failures experienced (μ) versus execution time (τ)

0

0

0

0

1

1ln 1

(B)

(P)

e

SENG521 (Fall 2002) far@enel.ucalgary.ca 33

Reliability Growth Reliability Growth Models /8Models /8

Failure(s) in time period

Probability

Elapsed time

(1 hour)

Elapsed time

(5 hours)

0 0.10 0.01

1 0.18 0.02

2 0.22 0.03

3 0.16 0.04

4 0.11 0.05

5 0.08 0.07

6 0.05 0.09

7 0.04 0.12

8 0.03 0.16

9 0.02 0.13

10 0.01 0.10

11 0 0.07

12 0 0.05

13 0 0.03

14 0 0.02

15 0 0.01

Mean 3.04 7.77

SENG521 (Fall 2002) far@enel.ucalgary.ca 34

How to Use the Models?How to Use the Models? Release criteria: time

required to test the system to reach a target failure intensity:

0

0

( )

( )

ln

1 1 1

P

F

F P

B

P

: Present failure intensity

: Target failure intensityP

F

SENG521 (Fall 2002) far@enel.ucalgary.ca 35

Reliability MetricsReliability Metrics Mean time to failure (MTTF): Usually calculated by

dividing the total operating time of the units tested by the total number of failures encountered (assuming that the failure rate is constant).

Example: MTTF for Windows 2000 Professional is 2893 hours or 72 forty-

hour workweeks. MTTF for Windows NT Workstation is 919 hours or 23 workweeks. MTTF for Windows 98 is 216 hours or 5 workweeks.

Mean time to repair (MTTR): mean time to repair a (software) component.

Mean time between failures (MTBF): MTBF = MTTF + MTTR

SENG521 (Fall 2002) far@enel.ucalgary.ca 36

Reliability Metrics:Reliability Metrics: AvailabilityAvailability

Software System Availability (A):

λ is failure intensity

tm is downtime per failure Another definition of availability:

Example: If a product must be available 99% of time and downtime is 6 min, then λ is about 0.1 failure per hour (1 failure per 10 hours) and MTTF=594 min.

11

1 m m

A tA t or t

t t t A t

MTTF MTTFA

MTTF MTTR MTBF

SENG521 (Fall 2002) far@enel.ucalgary.ca 37

Reliability Metrics:Reliability Metrics: ReliabilityReliability

Software System Reliability (R):

λ is failure intensity

R is reliability

t is natural unit (time, etc.) Example: for λ=0.001 or 1 failure for 1000 hours,

reliability (R) is around 0.992 for 8 hours of operation.

ln 10.95

R t R tt or for R t

t t

SENG521 (Fall 2002) far@enel.ucalgary.ca 38