A new heuristic for test suite reduction

ELSEVIER Information and Software Technology 40 (1998) 347-354

A new heuristic for test suite reduction’

T.Y. Chen*, M.F. Lau

Department of Computer Science, The University of Melbourne, Parkville 3052, Australia

Received 14 November 1997; received in revised form 7 January 1998; accepted 12 March 1998

Abstract

A testing objective has to be defined in testing a program. A test suite is then constructed to satisfy the testing objective. The constructed test suite contains redundancy when some of its proper subsets can still satisfy the same testing objective. Since the costs of executing test cases and maintaining a test suite for regression testing may be expensive, the problem of test suite reduction arises. This paper proposes a heuristic towards the optimization of a test suite. 0 1998 Elsevier Science B.V.

Keywords: Set covering; Software testing; Test suite reduction

1. Introduction

A testing objective has to be defined before a program is

tested [13]. Generally speaking, a testing objective can be considered as a set of testing requirements. For example, if the testing objective is that every branch of the program be executed at least once, each branch can be considered as a testing requirement. Obviously, different testing criteria have different sets of testing requirements. In this paper,

we simply use the term requirements instead of testing requirements whenever it is clear in the context.

Once a set of requirements is determined, test cases are designed to satisfy the requirements. A set of test cases that can collectively satisfy all requirements is said to be a test suite. A typical method of constructing a test suite, which has been employed by most automated test case generators [7,8,11,14,16], is to construct a test case for each requirement. A requirement is said to be feasible if there is a test case from the input domain of the program that can satisfy the requirement. The above process cannot be fully automated because the problem of determining whether a requirement is feasible is undecidable [17]. In other words, there is, in gene&, no algorithm that can generate

a test case for any given requirement. Usually, the infeasible requirements can be identified by human intervention. Thus, we assume in our study that all requirements are feasible

* Corresponding author. Tel: +61 3 9287 9101; fax: +61 3 9348 1184; e-mail [email protected]

Some results of this paper were presented in the Third International Conference on Software- Quality Management, 1995 [2].

0950-5849/98/$19.00 8 1998 Elsevier Science B.V. All rights reserved PII SO950-5849(98)00050-O

and the infeasible requirements can be identified and dis- carded.

Test cases designed specifically for a requirement may satisfy other requirements. As a result, the constructed test suite may contain redundancy because some of its proper subsets may still satisfy the set of requirements.

A test case in a test suite is said to be redundant if the same testing objective can still be satisfied by other test cases. On the other hand, some test cases in the test suite cannot be removed. Otherwise, some requirements can never be satisfied by the remaining test cases. Such test cases are said to be essential. For example, suppose that a program has five branches, namely bl, bz, b3, b4 and b5. Let ti be the test case which satisfies bt for i = 1,2,3,4,5. If tr and t3 coincidentally satisfy b2 and b5, respectively, the test suite ( tl,t2,t3,t4,t5) contains redundancy because ( tl,t3,t4} can still satisfy bl, bl, b3, bq, and bS. In this situation, t2 and t5 are the redundant test cases. If b4 can only be satisfied by t4, t4 is an essential test case.

The problem of test suite reduction arises [ 121 for three main reasons: (1) the constructed test suite usually contains redundancy; (2) the cost of executing test cases may be expensive; and (3) the cost of maintaining a test suite for regression testing may also be expensive. Test suite reduction aims at finding a subset of the test suite that can still satisfy all requirements. Such a subset is known as a representative set [9].

To software practitioners, the most interesting question is whether a smaller sized representative set will detect less faults than the original test suite. Intuitively speaking, the fault detecting ability of a test suite would decrease with the

348 T. Y. Chen, M.F. Luu/lnformation and Sojlware Technology 40 (1998) 347-354

removal of redundancy. Some researches [ 18,191 show that there is little or no significant reduction in the fault detecting ability of a representative set when compared to the original test suite. On the other hand, some researches (e.g. [15]) show that redundancy has been applied successfully to increase the fault detecting ability of a test suite. Thus, test suite reduction is not always cost-effective. There are such conflicting conclusions because different studies have different assumptions. Obviously, we should remove redundancy in situations similar to those studies in [ 18,191. How- ever, for situations similar to Ref. [ 151, we should take into consideration whether or not the savings in testing costs can outweigh the reduction in fault detecting ability.

The problem of finding an optimal representative set (that is, a representative set having the least number of elements) is well known to be NP-complete [9,12]. In a general context, finding the optimal representative set is equivalent to solving the set covering problem. The greedy heuristic G has been proposed to solve the set covering problem. The ratio of the size of the solution by the greedy heuristic to the size of an optimal solution is bound by O(lnn) [lo].

Recently, Harrold et al. [9] have proposed a heuristic algorithm (hereafter, referred to as H) to reduce the size of a test suite. Their rationale is to categorize test cases in the test suite according to different degree of ‘essentialness’ and select test cases in the order of most ‘essential’ to least ‘essential’. Based on a different rationale, we propose another heuristic for test suite reduction in this paper.

The rest of this paper is organized as follows. Section 2 introduces some of the basic concepts and result for test suite reduction. Section 3 discusses the greedy heuristic G and heuristic H. In Section 4, we propose a new heuristic and discuss its intuition. Section 5 compares our heuristic with the heuristic H. Moreover, we suggest further possible extension of the work in Section 6.

2. Preliminaries

Let us present some basic definitions and notation used in the paper. For any set A, IA 1 is used to denote the cardinality of set A. For any two sets of sets A and B, A @ B denotes the set {C u D: &A, DEB}.

We use R and T to denote the set of all testing requirements and the test suite, respectively. In this study, we assume that both R and Tare non-empty and finite. Further- more, we assume that for every requirement rER, there is always a test case tin the input domain which satisfies r. Let m and n denote the cardinality of R and T, respectively.

The satisjkzbility relation between test case tcT and requirement reR can be represented by a binary relation S(T,R) from T to R. The satisfiability relation is defined as the set ((t,r) ET X R: t satisfies r}. Whenever there is no ambiguity, we use S instead of S(T,R). For any tcT and any T’( # 4) G T, we use Req(t) and Req(T’) to denote the set of all requirements that are satisfied by the test case t and those

test cases in T’, respectively. Similarly, for any reR and any R’( f 4) C R, we use Test(r) and Test(R’) to denote the set of all test cases that can satisfy the requirement r and the requirements in R’, respectively.

A subset TI of the test suite T is said to be a representative set of S if Req(T,) = R. The problem of test suite reduction is to find a representative set of S. A representative set T1 of S is said to be optimal if ITI 1 I IT21 for any representative set T2 of S. We use REP(S) and OPT-REP(S) to denote the sets of all representative sets and optimal representative sets of S, respectively. Clearly, T is a representative set of S, and optimal representative sets of S always exist. Although optimal representative sets may not be unique, they have the same cardinality.

There are two different kinds of test cases in a test suite, namely the essential test cases and the redundant test cases. A test case is said to be an essential test case of S if Req(ZI( t)) f R. Let Ess(S) denote the set of all essential test cases of S. Whenever there is no ambiguity, we use Ess instead of Ess(S).

Contrary to the concept of essential test cases is the concept of redundant test cases. A test case is said to be a redundant test case of S if Req(YI{t)) = R. Obviously, if &T is redundant, ZJ{ t) # 4. If t is redundant, an optimal representative set of S(ZJ{ t),R) may not be optimal with respect to S. There is a special kind of redundant test cases known as the l-to-l redundant test cases. A test case t,_,cT is said to be l-to-l redundant if there exists a test case t( f t,-&T such that Req(tl_1) G Req(t). For any l-to-l redundant test case t 1_1, an optimal representative set of S(ZJ{ t l.I),R) is still optimal with respect to S.

3. Related work

Given a collection C(A) of n subsets of a set A having m elements, the set covering problem is to find the smallest number of subsets of A from C(A) such that their union is A. The set covering problem is well known to be NP-complete [6]. Greedy heuristic, referred to as G, in Fig. 1 has long been used to solve the general set covering problem. It repeatedly selects the set that contains the maximum number of uncovered elements. The worst case time complexity of the greedy heuristic G is O(mn.min(m,n)) [6]. The ratio of the number of elements of C(A) in the solution given by the greedy heuristic to the number of elements in an optimal solution is bound by 1 + Inn [lo].

In the following, we briefly describe the heuristic H proposed by Harrold et al. [9] for test suite reduction. Let Rk denote the set of all requirements in R that are satisfied by exactly k test cases in T. Moreover, we use d to denote the maximum number of test cases that a requirement can be satisfied (that is, d = max( ITest(r)heR

Intuitively speaking, test cases in Test(Ri) are considered to be more ‘essential’ than those in Test(R$ for ii < j. Clearly, test cases that satisfy requirements in Rr are

T. Y. Chen, M. F. Lmhfomation and Sofnoare Technology 00 (I 998) OOO-000

Greedy heuristic G input A : a set of m elements

C(A) : collection of n subsets of A OUtQUt c : collection of se.lected elements of C(A) declare U : setofal.luucoveredelementsinA

X : selected elements of C(A) begin

/* initialization */ U=A; c = 0; while (V # 0)

/* select a subset that has the maximum mm&r of uncovered elements */ select X E C(A) such that IX n UI is maximum; lJ=u\x; C=CU{X};

end-while end G

Fig. 1. Greedy heuristic G.

349

essential. The intuition of heuristic H is to select test cases according to their degree of ‘essentialness’.

Heuristic H first considers the group of requirements in RI. It selects test cases in Test(R,) ( = Ess) and removes those requirements satisfied by the selected test cases. It then considers the group of unsatisfied requirements in R2. It selects the test case t in Tesr(Rz) which can satisfy a maximum number of unsatisfied requirements in Rz. When- ever there is a tie situation among several test cases, it then selects the test case which satisfies a maximum number of unsatisfied requirements in RJ amongst those test cases which cause the tie situation in Rz. If a tie situation occurs again in R3, it selects the test case which satisfies the maximum number of unsatisfied requirements in R4. If a decision cannot be made, it continues on the next group and so on. Eventually, if a decision cannot be made when the process reaches Rh an arbitrary choice is then made. The above process continues until all unsatisfied requirements in R2 are satisfied. The heuristic then considers the group of unsatisfied requirements in Rs,.._,Rd (one at a time) and selects test cases in a similar manner. The worst case time complexity of H is O(m(m + n).d) [9].

4. Heuristic GRE

In this section, we present our heuristic GRE which is based on the following three strategies:

l the essentials strategy-the strategy of selecting all essential test cases;

l the l-to-l redundancy strategy-the strategy of removing l-to-l redundant test cases;

. the greedy strategy- the strategy of selecting test cases that meet a maximum number of requirements that are not yet satisfied.

The intuition of the greedy strategy is to achieve the

global optimal representative set through a series of local optima. If t is selected by the greedy strategy, the reduced

satisfiability relation is S(ZJ( t}, RWeq(t)). Clearly, the greedy heuristic G is the repeated application of the greedy strategy.

Intuitively speaking, the essentials strategy selects all essential test cases. After applying the essentials strategy, the original problem S(T,R) is reduced to S(N?ss, RiReq(Ess)). Chen and Lau [3] have proved that OPT_REP(S) = OPT_REP(S(I\Ess, RU?eq(Ess))) 0 {Ess} where Em is the set of all essential test cases. This means that: (1) the union of an optimal representative set of S(mss, RiReq(Ess)) and Ess forms an optimal representative set of S(T,R); and (2) every optimal representative set of S(T,R) can be constructed in this way. Hence, the essentials strategy should be applied as early as possible.

Since there are no essential test cases in the reduced satisfiability relation S(I\Ess, RUeq(Ess)), the essential strategy alone is not sufficient to yield a representative set unless all essential test cases can collectively satisfy all requirements in R. Since the situation that RWeq(Ess)) = 4 rarely occurs, the essentials strategy should be incorporated with other strategies.

A natural partner is the greedy strategy. Since essential test cases must appear in every representative set, the greedy strategy will eventually select all essential test cases. How- ever, the later the essential test cases are selected, the higher the chance for test cases selected at earlier stages become redundant in the end. Therefore, an enhanced version of the greedy heuristic, referred to as GE in the sequel, is to select all essential test cases first and then apply the greedy heuristic G.

Let us recall that a test case f is essential if Req(Y’i( t)) # R. In other words, a test case r is essential if and only if there exists a requirement reR such that r can only be satisfied by r (that is, Test(r) = {r}). Hence, the worst case time complexity for identifying all essential test cases is O(mn), where m and n denote the number of elements in the sets R and T, respectively. Since the time complexity of the greedy heuristic G is O(mnmin(m,n)), the worst case time complexity of GE is also O(mnmin(m,n)).

350 T. Y. Chen, M.F. L.m/lnformation and Sofnvare Technology 40 (1998) 347-354

algorithm GRE input R : set of 811 requirements, {rl, . . . , rm)

T : set of all test casea, {tl, . . . , tn} Z : setoftestcasesaatisfyingr(,i=l,...,m Ri : set of requirements satisfied by tf, i = 1, , n

output Selected : resulting set of test c8ses declare UI’IS8tidkd~ : set of all unsatisfied requirements

ASelected : selected set of teat casea in each step Test : set of unselected and non l-to-l redundant test cases M8X : maJcimumcardinalityofR1, . . ..R., ma0 : returns the maximum of a set of numbers

begin /* initialization */ Selected = 0; UnsatisfiedReq = R; Test = T; /* select essential teat cases */ Selected = IJiTi such that ri E R 8nd ITil = 1; Unsatisfied-&q = R \ (lJr Rj, tj E Selected); Test = Test \ Selected; for-each ti E Selected do & = 0; for-each ti E 2’ \ Selected do & = & \ (U, Rj, tj E Selected); while (UnsatisfiedReq # 0)

ASelected = 0; mergesort(Teat); /* sort test cases in Teat in descending order of l&l */ removexedundant(‘kst, RI, . . . . R,,, Tl, . . . . T,); if (there are some Ti such that Pi E Unsatisfied&q and ITfl = 1)

/* select essential test cases */ ASelected = lJi Tt S&I that ri E UnS8tisfiedm 8nd lTi[ = 1;

elfie /* select test case that satisfies the max. number of requirements */ Max = max({l&l : i = 1, . . . , n}); S&Ct 8 test case tj E Test SUCh that IRjl = Mm; ASelected = { tj }

end-if Selected = Selected U ASelected; Test = Test \ ASelected; UnS8ti&d_&J = Un&isfiedReq \ (Ur Rj, tj E ASelected); for-each ti E UnsatisfiedReq do Ti = Ti \ Adelected; for-each rf $ UnsatisWReq do Tf = 0; for-each ti E ASelected do & = 0; for-each ti E Test do & = & \ (Q Rj, tj E ASelected);

end-while end GRE

Fig. 2. Algorithm GRE.

The 1 -to- 1 redundancy strategy is defined as the technique of reducing the satisfiability relation S(T,R) by the removal of a l-to-l redundant test case in T. After the application of the l-to-l redundancy strategy, the original problem S(T,R)

is reduced to S(~(tl.l),R) where tl.l is l-to-l redundant. It should be noted that l-to-l redundant test cases should be removed one at a time. Otherwise, some requirements may be left unsatisfied. For example, consider the situation that a requirement reR can only be satisfied by tl and t2, and both tl and t2 satisfy the same set of requirements (that is, Req(t,) = Req(t2)). Obviously, both tl and t2 are l-to-l redundant. However, we can only remove either f1 or t2, but not both Otherwise, r cannot be satisfied.

The l-to-l redundancy strategy should be repeatedly applied until there are no l-to-l redundant test cases, because the more l-to-l redundant test cases are removed, the smaller the reduced problem.

For the l-to-l redundancy strategy, Chen and Lau [31 have proved that OPT_REP(S(n( t I_, ) ,R)) C_ OPT-REP(S)

where tl_l is a l-to-l redundant test case. In other words, any optimal representative set of S(ZJ{ tl.,} ,R) is optimal with respect to S(T,R). However, optimal representative sets of S(T,R) are not necessarily optimal representative sets of S(71{tl.,},R). Recently, Chen and Lau [4] have found out how to construct OPT_REP(S(T,R)) from OPT_REP(S(Ti(t,.l),R)).

Since the 1 -to-l redundancy strategy only aims at reducing the satisfiability relation rather than giving a representative set, it should be incorporated with other strategies. We need not consider combining the l-to- 1 redundancy strategy with the greedy strategy because the greedy strategy effectively ignores the l-to-l redundant test cases. The reason is listed as follows. Clearly, if a l-to-l redundant test case does not satisfy the maximum number of requirements, it will not be selected by the greedy strategy. On the other hand, if a l-to-l redundant test case fl_I satisfies the maximum number of requirements in S(T,R), there is another test case t2 such that Req(t,_l) = Req(t2). As a result,

T. Y. Chen, M. F. LaJInfotmation and Software Technology 00 (1998) 000400

function renwe~edundant(Test, &, . . . . R.,,, Tl, . . . . T,) input Test : wt of test cam arranged in descending order of I&l.

For simplicity, let S3t = (4, t;, . . .}. & :setof~~entssatisfiedbyti,i=l,...,n T( :setoftestcaaee8st~r‘,i=l,...,m

output Test : set of non l-t*1 redundant teat cases Ti : %t of non l-to-l redundant test COW satisfying ri, i = 1,. . . , m

declare ARedundant : set of l-to-l redundant test cases size : number of test cases in Test at start

begin /* initialization */ ARedundant = B; size = ITe6tl; /* select test cases for checking 14~1 redundancy */ fori=ltosize-ldo

if (t: E Test) forj=i+ltosizedo

if (ti E Test) /* check l-to-l redundancy, let tk = t: and tl = 5 */

if& E Rk)

ARedundant = AFkdundant IJ {t$}; Test = Teat \ {t;};

end-if end-if

end-for end-if

end-for /* removal of l-&l redundant teat casea */ for-each Ti do Ti = Ti \ ARedundant;

end removexedundent

Fig. 3. function rcmove_rcdundant ().

351

t2 is also one of the potential test cases in S(n(t,.,)J?). Since tl_l and t2 can be regarded as equivalent, the potential test cases of S(T,R) and s(n( t i-1 ) P) for the greedy strategy are effectively the same.

An interesting property of the 1 -to- 1 redundancy strategy is that some test cases may then become essential with respect to the reduced problem after applying the l-to- 1 redundancy strategy because the number of test cases that satisfy a requirement, say rip may decrease to 1. These ‘newly emerged’ essential test cases should then be selected because any representative set with respect to the reduced problem must include them. On the other hand, some test cases may become l-to-l redundant after the essentials strategy is applied. Thus, the essentials and l-to-l redundancy strategies are complementary to each other and should be alternately applied. Since only one reduced problem is to be solved after applying the essentials strategy or the l-to-l redundancy strategy, application of these two strategies will not result in an exponential growth of the reduced problems.

Based on the observation of the essentials strategy and the 1 -to- I redundancy strategy, we propose the heuristic GRE in Fig. 2. Heuristic GRE alternately applies: (1) the essentials strategy; and (2) the repeated application of l-to-l redundancy strategy. The greedy strategy is applied when, and only when, both the essentials strategy and the 1 -to- 1 redundancy strategy cannot be applied. In fact, it is easy to imple- ment a flag in GRE to indicate whether the greedy strategy has been applied. If the greedy strategy has not been

applied, GRE guarantees to deliver an optimal representative set. This gives a distinctive feature to GRE because other heuristics mentioned in this paper cannot provide this extra information. Moreover, GRE may still deliver an optimal representative set even if the greedy strategy is applied.

We analyze the worst-case time complexity of GRE. The time complexity for the selection of all essential test cases is at most O(mn) where m and n denote the number of elements in the sets R and T, respectively. Since there are at most n essential test cases, the time complexity for removing those requirements satisfied by all essential test cases is at most O(nk) where k is the maximum number of requirements that a test case can satisfy (that is, k = maxi 5 i c n(IReq(ti)I)). Moreover, GRE loops at most min(m,n) times until all requirements are satisfied or all test cases are selected. During each loop, the time complexity for removing l-to-l redundant test cases is at most O(n’k). If there are some essential test cases, the time complexity for selecting the essential test cases is at most O(m). Otherwise, the time complexity for selecting the test case which satisfies the maximum number of requirements is O(1) time because the test cases have already been sorted, as stated in the algorithm. The worst case time complexity of the sorting is O(nlogn). Finally, the time complexity for removing those requirements satisfied by the selected test cases is at most O(A). Therefore, the worst case time complexity of GRE is O(mn -I- nk + min(m,n>(m + n2k)) which is O(min(m,n)(m + n2k)). Fig. 3

352 T. Y. Chen, M. F. L.m/lnformation and Sojiware Technology 40 (I 998) 347-354

Table 1 Test cases for pushdown

Test case A[i].key” First Last

tl 21 1 2

t2 2134 1 4

t3 51342 1 5

t4 3214567 1 7

t5 12 1 2

t6 123 1 3

t7 1432 2 4

fs 21345 1 5

t9 321456 1 6

t10 4132 1 4

TII 1234 2 4

tI2 621453 1 6

sistatt~from 1. Fort,,A[l].key = 2andAi2l.key = 1.

5. An example

In this section, we illustrate with examples that neither GRE nor H always deliver a smaller sized representative set. Fig. 4 is the control flow graph of the pushdown module taken from Ref. [ 11 (p. 273). The module pushdown accepts

n START

anarrayA[l],..., A[n] and two integersfirst and Zast indicat- ing the first and last entries of the array A being considered by the module. It pushes the element Atfirst] down through its descendants by a sequence of swaps to its proper position in the tree until A satisfies the partially ordered tree prop-

erty. An array A[l],..., A[n] are said to obey the partially ordered tree property if A[i].key 5 A[2 X i].key and A[i].key 5 A[2 X i + l&key for 1 5 i 5 ln/2].

Suppose the testing objective used is branch coverage. Then, each requirement corresponds to a branch of the program. There are 19 branches in the control flow graph, namely, b,,..., b 19. All test cases used are listed in Table 1. For example, for tl, the values of$rst and last are 1 and 2, respectively. Moreover, the values of A[ l&key and A[2].key are 2 and 1, respectively. We assume that all test cases have the same execution cost. Hence, the performance of each heuristic is solely judged by the size of the representative set.

Suppose the test suite is { tl,tz,t3,t4,ts,ts,t7) whose satisfiability relation is given in Table 2. In this case, GRE delivers (t2,t4,tl} (test cases are listed according to the order of selection) or [tz,t4rt7} as its representative set. H delivers (t3,tl(t7),t4(b6),t2(t5)} (any test case enclosed within the

b14

r=Zr

bl3

,wrp(N&A[~l)

Y# bll

b18

r=2r+l

b17

rmp(AIrJAIZr+ll)

Y+ b15

’ b10

Fig. 4, Control flow diagram of pushdown.

353 T.Y. Chen, M.F. lmdlnfomation and Sofiware Technology 00 (1998) OOO-000

Table 2 GRE performs better than H

b, bz b3 b4 bs b6 bl bs b9 bm bil blz bu b14 b I5 bt6 bi7 bls b19

tt x x x x x X x x

12 x x x x x x X x x x x

t3 x x x x X x x x x x x x

14 x x x x X X x x x x x

t5 x x x x x X X

16 x x x x X X X X

f7 x x x x x X x x X

Table 3 H performs better than GRE

bl bz b3 b4 bs b6 b7 bs b9 blo bll bu b b14 13 bl5 b16 b,7 618 b19

tl x x x x x X x x

t2 x x x x x x X x x x x


t4 x x x x X X x x x x x

t8 x x x x X x x x x X X

t9 x x x x x x X X X X x x

Table 4

GE performs better than GRE

bl bz b3 b4 bs b6 b7 bs b9 blo b,, bi2 bu b 14 bl5 bt6 b 17 bl8 b 19

II x x x x x X x x


t4 x x x x X X x x x x x

15 x x x x x X X

16 x x x x X X X X

t8 x x x x X x x x x X X

210 x x x x x x x x x x x x

Ill x x x x x X X

t12 x x x x x x x x x X X x x

parentheses can be used to replace the corresponding test case to form another representative set) as one of the many representative sets. All representative sets given by GRE and H have exactly three and four test cases, respectively. Thus, GRE pedorms better than H.

Now, consider the case that the test suite is (r I,f2,r3,f4,f8,t9} whose satisfiability relation is given in Table 3. In this case, H delivers either (fl,f4,r2} or (fl,rs,rg) as its representative set whereas GRE gives (rl,r3.f2(r~),f~(fs)}. Thus, H performs better than GRE.

As mentioned in Section 4, an enhanced version of the greedy heuristic, referred to as heuristic GE, can be formed by combining the essentials strategy and the greedy strategy. The heuristic GE selects all essential test cases first and then repeatedly applies the greedy strategy to find a representative set. Intuitively speaking, the performance of heuristic GRE should be better than that of GE because GRE makes use of an extra strategy, the l-to-l redundancy strategy. For example, for the satisfiability relation defined in

Table 2, GE gives (f~,fl(f7), fq(f~),f~(f~)) as a representative set whereas GRE gives ( f2,r4,fl(f7)).

However, there are situations that GE performs better than GRE. For example, suppose the test suite is ( f~,f~,f~,rs,r~,rg,t~O,tll,tlZJ with the satisfiability relation given in Table 4. GE gives (fl~,f~,f~(fll)} whereas GRE gives ~fs(f~~),r~,f~~(f~~),f~(f~)}. Hence, GE performs better than GRE. Moreover, in this case, GE performs better thanHbecauseHgives (f(f )f r(r f )f(r f)) .On 5 11~3~1 10112r4 68

the other hand, there are situations that H performs better than GE. For the satisfiability relation given in Table 3, GE gives (fl,f3,f2(fg),f4(f8)} whereas H gives either (r~,r4,f~] or

{fl,t8tf9).

6. Discussion

In this paper, we discuss three strategies for test suite reduction, namely, the essentials strategy, the Z-to-Z

354 T. Y. Chen, M.F. LauHnfonnation and So&are Technology 40 (1998) 347-354

redundancy strategy and the greedy strategy. We have the following observations. First, the essentials strategy should be applied before any other strategy because: (1) every representative set contains all essential test cases; and (2) the union of an optimal representative set of the reduced problem and the set of all essential test cases forms an optimal representative set of the original problem. Second, after applying the l-to-l redundancy strategy, an optimal representative set of the reduced problem is still optimal with respect to the original problem. Thus, the 1 -to- 1 redundancy strategy should be repeatedly applied whenever possible because the more l-to-l redundant test cases are removed, the simpler the reduced problem is. Third, if the application of the essentials strategy and the l-to-l redundancy strategy is sufficient to deliver a representative set, the representative set is optimal with respect to the original problem.

We propose heuristic GRE as an approach to test suite reduction. The heuristic GRE is basically the alternate application of: (1) the essentials strategy; and (2) the repeated application of the l-to-l redundancy strategy. The greedy strategy is applied only when both strategies cannot be applied. A distinctive feature of GRE is that we know the representative set is optimal, if the greedy strategy has never been applied. Finally, we illustrate with the module pushdown and branch coverage testing as an example that none of the three heuristics (GRE, GE and H) delivers a smaller sized representative set at any time.

In order to gain understanding of the performance of the heuristics, GRE, GE and H, a simulation study is being conducted. The aim of this simulation study is to provide general guidelines for testing practitioners in selecting a ‘better’ heuristic for test suite reduction. The results of the study [5] can be summarized as follows. First, all three heuristics GE, GRE and H deliver representative sets of nearly the same size when the ratio of overlapping is less than 3. The ratio of overlapping is defined as the quotient of the total count of requirements satisfied by all test cases to the total number of requirements. Second, for ratio of overlapping lies between 3 and 7, the representative sets found by GRE is smaller than those of GE and H. Third, when the ratio of overlapping is greater than 7, the performance of the heuristics GE and GRE are nearly the same, and the size of the representative sets produced by either GE or GRE is less than that produced by heuristic H.

References

[l] A.V. Aho, J.E. Hopcroft, J. D. Ullman, Data Structures and Algo- rithms, Addison-Wesley, Reading, MA, 1983.

[2] T.Y. Chen, M. F. Lau, Heuristics towards the optimization of the size of a test suite, in: Proceedings of the Third International Conference on Software Quality Management, 1995, vol. 2, pp. 415-424.

[3] T.Y. Chen, M.F. Lau, Dividing strategies for the optimization of a test suite, Information Processing Letters 60 (3) (1996) 135- 141.

[4] T. Y. Chen, M. F. Lau, On the completeness of l-to-l redundancy dividing strategy (submitted for publication).

[5] T. Y. Chen and M. F. Lau, A simulation study on some heuristics for test suite reduction (submitted for publication).

[6] T. H. Cormen, C. E. Leiserson, R. L. Rivest, Introduction to Algo- rithms, MIT Press, Cambridge, MA, 1990.

[7] R.A. DeMillo, A.J. Offutt, Constraint-based automatic test data generation, IEEE Transactions on Software Engineering 17 (9) (1991) 900-910.

[8] R. Gupta, M.L. Soffa, Employing static information in the generation of test cases, Software Testing, Verification and Reliability 3 (1993) 29-48.

[Q] M.J. Harrold, R. Gupta, M.L. Soffa, A methodology for controlling the size of a test suite, ACM Transactions on Software Engineering and Methodology 2 (3) (1993) 270-285.

[lo] D.S. Johnson, Approximation algorithms for combinatorial problems, Journal of Computer and System Sciences 9 (3) (1974) 256-278.

[l l] B. Korel, Automated software test data generation, IEEE Transactions on Software Engineering 16 (8) (1990) 870-879.

1121 H. K. N. Leung, L. White, Insights into regression testing, in: Proceed- ings of International Conference on Software Maintenance, 1989, pp. 60-69.

[ 131 G.J. Myers, Software Reliability: Principles and Practice, Wiley, New York, 1976.

[14] C.V. Ramamoorthy, On the automated generation of program test data, IEEE Transactions on Software Engineering 2 (4) (1976) 293-300.

1151 P. Thtvenod-Fosse, H. Waeselynck, Y. Crouzet, Software structural testing: an evaluation of the efficiency of deterministic and random test data, Technical report LAAS-91.389, Centre National De La Recherche Scientifique, Laboratoire D' Analyse et D' Architecture Des Systems, 1991.

[16] W.T. Tsai, D. Volovik, T.F. Keefe, Automated test case generation for programs specified by relational algebra queries, IEEE Transactions on Software Engineering 16 (3) (1990) 316-324.

[17] E.J. Weyuker, Translatability and decidability questions for restricted classes of program schemas, SIAM Journal on Computing 8 (4) (1979) 587-598.

[18] W.E. Wong, J.R. Horgan, S. London, A.P. Mathur, Effect of test set minimization on fault detection effectiveness, in: Proceedings of the 17th International Conference on Software Engineering, 1995, pp. 41-50.

[19] W.E. Wang, J.R. Horgan, A.P. Mathur, A. Pasquini, Test set size minimization and fault detection effectiveness: a case study in a space application, in: Proceedings of COMPSAC 97: the 21st Annual International Computer Software and Application Conference, 1997, pp. 522-528.

Documents

A new heuristic for test suite reduction