IEEE TRANSACTIONS ON RELIABILITY, VOL. 37, NO. 1,1988 APRIL 41

Optimum Number of Errors Corrected before Releasing a Software System

D. S. Bai, Senior Member ASQC Korea Advanced Institute of Science and Tech- nology, Seoul

Korea Advanced Institute of Science and Tech- nology, Seoul

w. Y. Yun

Key Words - Software system, Decreasing failure rate, Average gain

Raadn Ai& - Purpose: Widen state of art Special math needed for derivation: Probability Special math needed to use results: Same Results useful to: Software & Reliability engineers

Abstract - This paper considers software release problems based on Jelinski-Moranda and decreasing-failure-rate models. The following software release policy is considered: After a preassigned number, n, of errors are removed, the test is ter- minated and the software system is released to the operational phase. The average gain is used as a criterion; there exists a unique optimal value of n. Numerical examples indicate that the software release policy based on the number of errors detected can be a good alternative to the existing policy.

I. INTRODUCTION

Reliability models for a software system have been studied by many authors [7]. Most of these studies are con- cerned with stochastic models of errors resulting from bugs in a software system.

An important problem in the development of software system is when to stop testing and release the system. The longer the testing phase, the higher the reliability and the lower the operating cost. However, delays in software release increase the testing and other costs. Hence, the op- timum length of testing phase is a tradeoff between the two competing facets. A model for software reliability assess- ment during testing is called a software reliability growth model (SRGM). Optimal release problems based on an SRGM have been studied by some authors [l, 4, 6, 81. They used total average cost as a criterion.

In this paper, software release problems based on Jelinski-Moranda and decreasing failure rate (DFR) models are studied. The test of the software system is ter- minated after a preassigned number of errors, n, are removed. The average gain is used as a criterion. In the two SRGMs, the optimal value of n is shown to be unique. In section 2, a software release problem based on Jelinski- Moranda model [2] is considered. Section 3 shows that the proposed policy can be applied to all DFR models.

Basic Assumptions

1. Correction of an error takes only negligible time. 2. Each failure of software system is caused by one

3. An error detected is removed with certainty. 4. Planning horizon is infinite.

error.

Notation

M total number of errors in the software system fit), F(t), r(t) Cdf, Sf, hazard rate of the software

system

L(t) mean residual life: F(x) dx/F(t)

n

n* optimaln Tk x k Tk - Tk-i

Ek{Xk+lIx1, ..., Xk} posterior s-expectation of &+l,

c1 cost-rate of testing c2 c3

C c3 - c2 G( .) average gain

number of errors detected when the test is ter- minated

time at which error k is detected; To = 0

fk, Xk observed Values Of Tk, x k

givenxi, ..., xk cost of correcting an error during test phase cost of correcting an error during operational phase (c3 > c2)

Policy

The software system is released for operation as soon as n * errors are detected and corrected.

11. JELINSKI-MORANDA MODEL

Jelinski & Moranda [2] proposed a model for describ- ing failures of computer software in which the hazard rate is :

r,,(t) = B[M - (n - 1)J for Tn-l d t e T,,,

where 8 is a proportionality constant. We show that there exists a unique optimal value of n.

Case I . Mis known [SI.

The average gain is:

a n ) = cn * - c1 E{ T,,.}.

001 8-9529/88/04OO-O041$01 .00@ 1988 IEEE

42 IEEE TRANSACTIONS ON RELIABILITY, VOL. 37, NO. 1,1988 APRIL

Theorem 1. An optimum n* exists and is unique. (This theorem is proved in the appendix.)

Case 2. M is a random variable.

We assume that M follows a Poisson distribution with mean X. The construction of the optimal procedure is based on the principle of optimality [9]. Just after error n is detected, the gain in cn - clt,. At this stage, we may either stop testing or continue testing until another error is detected.

The average gain, G(xl, ..., x,), when the optimal procedure is cQntinued from the position (xl , ..., x,), is:

W i J xn) = max{cn - cite, E{G(xi, e..) x,, X,+d}}

(2)

Equiation (2) is, in general, difficult to solve. However, if fora l l j > Oandalln > 0-

E,{...{E,+i-i{X,+iJXi, ... 9 X, X n + i , .-.> Xn+j-i}}}

aE,{X,+l IxlJ *.a> X,}J (3)

then it follows that the optimal procedure is to look ahead only one stage from any given position. Inequality (3) holds if, for all n > 0, inequality

E~(E~+~{X~+ZIXI, .**, xn, X~+I}} a E~{X~+IIXI, xn}J

(4)

holds [9]. Therefore, if (4) holds, the optimal procedure is to stop testing at stage n if and only if - cn - clt, 2 ~ ( n + 1) - ci[t, + E,{X,+i 1x1, ..., x,}],

or, equivalently, if and only if -

E,{X,+i 1x1, ..., xn} aCi/c. (5)

Case 2A. X is known.

The joint distribution of X1, ..., X,, given M = m, is

AxlJ ..., x,lm) = [B"m!/(m - n)!] exp[- e[ml

+ (m - IN2 + ... + (m - n + 1)X"l.

Since M is Poisson distributed with mean A, the posterior distribution of M, given xl, ..., x, is:

f ( m Ixl, ..., x,) = pz-" exp( - p,)/(m - n)!, m > n,

p, = X exp(- et,)

t, = k xi. is 1

We assume that, if a error is not detected until a predeter- mined time T(T > 1 4 ) at any stage, the test stops and the software system is released. Hence,

E,{X,+llxl, ..., x,} = Pr{X = O}T

E,{E,+1{X,+21~1, ..., x,, X,+,}} = Pr{X = 0, orX = 1)T

where Xfollows a Poisson distribution with mean, Aexp( - et,). Since T > 1/0, and (1 - exp( - iOT))/(ie) is decreas- ing function of i, inequality (4) is obviously satisfied. Thus the optimal number of errors detected is the smallest in- teger satisfying (9, which is usually difficult to obtain. However, since exp(- i07) 4 1 for large T, we may use E{ l/(OX)} 2 cJc. The average value of X1 in a Poisson distribution may be obtained numerically [3].

Case 2B. X is unknown.

The gamma conjugate prior distribution with parameters CY and 0 is assumed:

Then, after some simplifications -

p = exp(- et,)/(p + 1).

Hence, E,(X,+l 1x1, ..., x,} and En{En+i{Xn+2 1x1, ..., x,, X,+,}} are equal to (6) and (7), where X follows a negative binomial distribution with parameters and exp( - et,)/@ + 1). Since T > 1/8 and (1 - exp( - ieT)/(iB) is decreas- ing in i , inequality (6) is satisfied. Thus the optimal number of errors detected is the smallest integer satisfying (5) . However, the exact optimal value is difficult to obtain. Since exp( - ieT) 4 1 for large T, we may use E{ l/eX} 2 cl/c. The average value of X-' is obtained numerically [3].

111. DFR MODEL The proposed policy is applied to another SRGM in

which the hazard rate is DFR.

Lemma 1. If the hazard rate of a system is decreasing E{ T,+l - T,} is increasing in n. (This lemma is proved in the appendix.)

Theorem 2. If the hazard rate of software system is decreasing; there is a unique optimal value of n.

BAI/YUN: OPTIMUM NUMBER OF ERRORS CORRECTED BEFORE RELEASING A SOFTWARE SYSTEM

-

43

Numerical Examples

Four examples in Koch & Kubat [4] are considered; they all correspond to case iii of the proof of theorem 1 in the appendix, viz, 1/0 > c/c, and l/(MB) < c/cl. Those authors claim that although the data in the examples are not taken from specific projects, they represent realistic costs emanating from their experience. Table 1 summarizes the results of computation. In all these examples, since T * is close to E{ T,.} and n * is close to the average number of errors detected, M(l - exp( - BT*)) the differences bet- ween Kock & Kubat policy and the proposed policy in average gain are small. The Koch & Kubat model is sum- marized in the appendix.

TABLE 1 Comparison of Maximum Values of Average Gain

Example #1 #2 #3 #4

21500 21500 21500 21500 0 0 u)oo 2000

3500 m 3500 m 30 50 30 50 1.2 1.6 1.2 1.6

Koch & Kubat policy

T + 1.41 1.94 0.11 1 .Ol M(1- exp( - T 'e)) 24.88 41.16 18.06 41.04 O(T3 55402 244829 10583 38451

Proposed policy

n* 25 48 19 42 E(Tn.1 1.43 1.81 0.81 1.11 G(n 9 56832 247698 1 1 029 39063

VI. CONCLUDING REMARKS In existing models [I , 4, 6, 81, the testing phase is ter-

minated at a specified time. In this paper, a software system is released for operational phase after a specified number of errors (equal to the number of system failures) are detected. The average gain is used as a criterion for finding the optimal value of n. The uniqueness of the op- timal n is shown for the two cases. Hence, the optimal solutions can be obtqined numerically by, for example, a bisection method. The example demonstrates that the pro- posed policy is a good alternative to the existing policy.

APPENDIX

Summary of Koch h Kubat Model

The average gain is -

G(T) = cM(1 - exp( - TO)) - clT, (A-1)

where Tis the software release time and M(I - exp( - Te)) is the average number of errors detected during testing.

The value of T satisfying dG(T)/dT = 0, is -

Since d2G(T)/dT2 < 0, G(T) is a convex function of T, and Tp is the maximum point of (A-1). Hence, if 1/M8 2 c/cl, the optimal release time is T * = 0. Otherwise T * = T9.

Proof of Theorem 1

A necessary condition for n to maximize (1) is

G(n - 1) < G(n) and G(n) 2 G(n + l), (A-2)

which is equivalent to

E{Tn+I - T,} 2 kandE{T, - T,-1} < k, (A-3)

where k = c/cl.

Therefore, E{ T, - T,,-l} is increasing in n. Hence -

i. If 1/8 < c/cl, then n* = M, that is, the software system is not released for operation until all errors are detected and corrected.

ii. If 1/(M#) 2 c/cl, then n* = 0, that is, the soft- ware system is released without test.

iii. If 1/0 > c/cl and 1/(M8) < c/cl, then 0 e n* < M. Since E{ T, - Tn-l} is an increasing function of n, n * is unique and can be obtained numerically by bisection method. QED

Proof of Lemma I

E{Tn+l - T,} = E{E(Tn+I - tnITn = tn}} = E{L(tJ}. Since DFR implies increasing mean residual lifetime, L(t,) is increasing in t,. Since t, > tn-l, L(tn) > Ut,-1) and E{L(G)} > E{L(h-d}. QED

Proof of Theorem 2

A necessary condition for n maximizing G(n) is the same as (A-I) which is equivalent to -

where k E c/cl. Therefore, if c/cl < E{ TI}, then n* = 0; that is, the software system is released for operation without test. If c/cl 2 lim {E{ T, - T,-l}}, then the soft-

ware system is not released for operation. Except for these n - -

44 IEEE TRANSACTIONS ON RELIABILITY, VOL. 37, NO. 1.1988 APRIL

two cases, the value of n satisfying condition (A-4) is unique by lemma 1 and can be obtained numerically by the

[9] M. C. K. Yang, D. D. Wackerly, A. Rosalsky, “Optimal stopping rule in proofreading”, J. Applied Probability, vol 19, 1982, pp

bisection method. QED 723-729.

REFERENCES

E. H. Forman, N. D. Singpurwalla, “Optimal time intervals for testing hypothesis on computere software errors”, IEEE Trans. Reliability, vol 28, 1979 Aug, pp 250-253. Z. Jelinski, P. Moranda, “Software reliability research”, in Statistical Computer Performance Evaluation, Academic Press, 1972, pp 465484. N. L. Johnson, S . Kotz, Discrete Univariate Distribution, Houghton Mifflin Co., 1970. H. S. Koch, P. Kubat, “Optimal release time of computer software, IEEE Trans. Software Eng., vol SE-9, 1983 May, pp 323-327. P. Kubat, ,H. S. Koch, “Pragmatic testing protocols to measure software reliability”, IEEE Trans. Reliability, vol 32, 1983 Oct, pp

K. Okumoto, A. L. Goel, “Optimal releases time for software system based on reliability and cost criteria”, J. System Software,

C. V. Ramamoorthy, F. B. Bastani, “Software reliability status and perspective”, IEEE Trans. Software Eng.. vol SE-8, 1982 May, pp

S. Yamada, H. Narihisa, S. Osaki, “Optimal software policies with a scheduled software delivery time”, Int. J. Systems Science, vol 15,

338-341.

VOI 1, 1980, pp 315-318.

354-371.

1984, pp 904-915.

AUTHORS

Dr. D. S. Bai; Department of Industrial Engineering; Korea Advanced Institute of Science and Technology (KAIST); POBox 150, Chongyangni; Seoul, KOREA.

D. S. Bd was born in Haeju, Korea, 1937 October 11. He received a PhD in statistics in 1971 from the Ohio State University. He taught statistics at the Ohio State University in 1971-1972 and 1981-1982 and at the State University of New York at Oneonta in 1972-1974. Since 1974 he has been a Professor at KAIST. His major interests lie in statistical in- ference in stochastic processes, quality control, and reliability theory.

W. Y. Yun; Department of Industiral Engineering; KAIST; POBox 150, Chongyangni; Seoul, KOREA.

W. Y. Yun: For biography see IEEE Trans. Reliability, vol35, 1986 Oct, p 454.

Manuscript TR86-092 received 1986 July 25; revised 1987 May 4; revised 1987 July 15.

IEEE Log Number 17962 4 T R W

INTERNATIONAL - RELIABILITY AVAILABILITY MAINTAINABILITY ntwram Conference for the Electric Power Industry

“RAM for Utility and Component Supplier Optimization”

Theme: Emphasizes practical application of RAM techniques in the design, manufacturing, and support of products and services for the utility industry.

Program :

Plenary session, tutorials, presentation of 90

Publication of conference proceedings. Industrial, technical, educational exhibits related to RAM in generation, transmission, and distribution of electric power.

papers.

Endorsed by: IEEE, CEA, ASQC, IIE, ASME, SRE, SLE, NERC, CNS and other technical and profes- sional societies involved in the electric power industry.

Hosted by: Bonneville Power Administration

Date: 1988 June 14-17

Location: Portland, Oregon USA

Contact:

Ray Vanderzanden, Inter-RAM Conference Chair ’n Bonneville Power Administration

Portland Oregon 97208 USA POBOX 3621 - EMQ2

Phone: 503-230-51 43 4 TR D