A To Be Optimal Or Not in Test-Case Prioritizationthe latter signiﬁcantly outperforms the former in terms of either fault detection or execution time. As the optimal technique schedules

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007 1

To Be Optimal Or Not in Test-Case PrioritizationDan Hao, Lu Zhang, Lei Zang, Yanbo Wang, Xingxia Wu, Tao Xie

Abstract—Software testing aims to assure the quality of software under test. To improve the efficiency of softwaretesting, especially regression testing, test-case prioritization is proposed to schedule the execution order of test casesin software testing. Among various test-case prioritization techniques, the simple additional coverage-based technique,which is a greedy strategy, achieves surprisingly competitive empirical results. To investigate how much difference thereis between the order produced by the additional technique and the optimal order in terms of coverage, we conducta study on various empirical properties of optimal coverage-based test-case prioritization. To enable us to achieve theoptimal order in acceptable time for our object programs, we formulate optimal coverage-based test-case prioritization asan integer linear programming (ILP) problem. Then we conduct an empirical study for comparing the optimal techniquewith the simple additional coverage-based technique. From this empirical study, the optimal technique can only slightlyoutperform the additional coverage-based technique with no statistically significant difference in terms of coverage, andthe latter significantly outperforms the former in terms of either fault detection or execution time. As the optimal techniqueschedules the execution order of test cases based on their structural coverage rather than detected faults, we furtherimplement the ideal optimal test-case prioritization technique, which schedules the execution order of test cases basedon their detected faults. Taking this ideal technique as the upper bound of test-case prioritization, we conduct anotherempirical study for comparing the optimal technique and the simple additional technique with this ideal technique. Fromthis empirical study, both the optimal technique and the additional technique significantly outperform the ideal techniquein terms of coverage, but the latter significantly outperforms the former two techniques in terms of fault detection. Ourfindings indicate that researchers may need take cautions in pursuing the optimal techniques in test-case prioritizationwith intermediate goals.

Index Terms—Test-Case Prioritization, Integer Linear Programming, Greedy Algorithm, Empirical Study.

F

1 INTRODUCTION

Regression testing is an expensive task in soft-ware maintenance. For example, the industrialcollaborators of Elbaum et al. [1], [2] reportedthat it costs seven weeks to execute the entiretest suite of one of their products. Test-caseprioritization [3], [4], [5], [6], [7], [8], whose aimis to maximize a certain goal of regression test-ing via re-ordering test cases, is an intensivelyinvestigated approach for reducing the cost ofregression testing.

As the main goal of regression testing isto assure the quality of the software under

• Dan Hao, Lei Zang, Yanbo Wang, Lu Zhang, and XingxiaWu are with the Key Laboratory of High Confidence SoftwareTechnologies, Ministry of Education and with the Instituteof Software, School of Electronics Engineering and ComputerScience, Peking University, Beijing, 100871, P. R. China; Tao Xieis with the Department of Computer Science, University of Illinoisat Urbana-Champaign. E-mail:{haodan, zhanglucs, zanglei,wangyanbo}@pku.edu.cn;[email protected];[email protected]

regression testing, most techniques for test-case prioritization aim at maximizing the fault-detection capability of the prioritized set oftest cases. As whether a test case can detect afault is unknown before running the softwareunder test, the fault-detection capability canhardly be used to guide scheduling the exe-cution order of the test cases directly. Since atest case cannot detect a fault if the test casedoes not execute (or cover) the correspondingfaulty structural unit, most techniques for test-case prioritization (e.g., [9], [10], [6], [11]) usestructural coverage as the substitutive goal (i.e.,intermediate goal) for test-case prioritization.However, these techniques can be only sub-optimal if measured based on the intermediategoal. Moreover, the simple additional coverage-


based technique1, which is proposed in an earlypaper on test-case prioritization by Rothermelet al. [6], remains very competitive among allthe existing techniques for test-case prioriti-zation [4], [10]. In particular, with regard tothe intermediate goal, the additional coverage-based technique is by far the most effectivetechnique [4].

Considering the surprisingly good empiricalresults of the additional coverage-based tech-nique, we are curious about how much dif-ference there would be between the order oftest cases achieved by the additional coverage-based technique and the order of test cases withthe optimal value on the intermediate goal.Furthermore, as it may be very costly to guar-antee optimality due to the NP-hardness of thetest-case prioritization problem, it is also inter-esting to investigate other empirical propertiesof the optimal order to understand whether itis cost-effective to achieve the optimal order.

To learn the cost and effectiveness differ-ence between the additional coverage-basedtechnique and the optimal order, we conductan empirical study on ten non-trivial objectprojects, systematically investigating empiricalproperties of optimal coverage-based test-caseprioritization in comparison with the additionalcoverage-based test-case prioritization. To en-able our study, we model optimal coverage-based test-case prioritization2 as an IntegerLinear Programming (ILP) [12] problem andthus are able to achieve the optimal order interms of coverage in acceptable time usingan existing ILP solver for many non-trivialprograms. In particular, our empirical studyevaluates the effectiveness and efficiency of thetwo techniques using three metrics. Consid-ering the impact of coverage granularity, ourempirical study further considers two types ofcoverage for both the optimal technique and

1. The additional coverage-based technique is a greedy al-gorithm that always orders the test case covering the moststructural units (e.g., statements or methods) not yet coveredby previously executed test cases before any other previouslyunexecuted test cases.

2. Rothermel et al. [6] used an optimal technique as a controltechnique to evaluate the effectiveness of some techniques fortest-case prioritization. However, their optimal technique is atechnique with the knowledge of which test case detects whichfault but still using the additional strategy to order test cases.

the additional technique: statement coverageand method (function) coverage.

Furthermore, to learn the upper bound oftest-case prioritization, we also implement theideal optimal test-case prioritization technique,which schedules the execution order of testcases based on the number of detected faults.Although this ideal optimal technique is notpractical, it may serve as a control technique.Then we conduct an empirical study on an-other five non-trivial object projects, system-atically investigating the effectiveness of theoptimal technique and the additional techniquecompared with the ideal technique.

According to our empirical results, the op-timal technique is slightly better than theadditional technique with ignorable differencefor achieving optimal coverage. However, theoptimal technique is significantly worse thanthe additional technique for most target pro-grams in terms of fault detection. Moreover,although both the optimal technique and theadditional technique significantly outperformthe ideal technique in terms of coverage, thelatter significantly outperforms the former twotechniques in terms of fault detection. There-fore, in test-case prioritization, it is not worth-while to pursue optimality by taking the cov-erage as an intermediate goal.

This article makes the following main contri-butions:

• To our knowledge, the first empirical studyon the optimal coverage-based test-caseprioritization, demonstrating that the op-timal technique may be inferior to the ad-ditional technique in practice.

• A formulation of optimal coverage-basedtest-case prioritization as an integer linearprogramming problem, which enables usto obtain the execution order of test casesto achieve optimal coverage.

The remaining of this article is organized asfollows. Section 2 presents some backgroundof test-case prioritization. Section 3 presentsa formulation of optimal coverage-based test-case prioritization as an ILP problem. Section 4and Section 5 present the design, results, andanalysis of our two empirical studies. Section 6discusses some issues in this article. Section 7


briefly presents related work and Section 8concludes this article.

2 BACKGROUND

In this section, we present background on test-case prioritization along with optimal test-caseprioritization.

2.1 Test-Case PrioritizationTest-case prioritization [3], [4], [5], [6], [7], [8]is a typical software-engineering task in regres-sion testing, which is formally defined [1] asfollows. Given a test suite T and its set ofpermutation on the test cases denoted PT , test-case prioritization aims to find a permutationPS in PT such that for any permutation T ′

of PT , f(PS) ≥ f(T ′), where f is a functionfrom PT to a real number that represents thefault-detection capability. That is, the ultimategoal of test-case prioritization is to maximizethe early fault-detection capability of the prior-itized list of test cases.

Researchers proposed APFD3 [6], which isthe abbreviation of average percentage of faultsdetected, to measure the effectiveness of theprioritized list of test cases on detecting faults.Formula 1 gives the definition of APFD.

APFD = 1−∑m

j=1 TFj

nm+

1

2n(1)

In Formula 1, m denotes the number of faultsdetected by the test suite T , n denotes thenumber of test cases in T , and TFj denotes thefirst test case in the T ′ (which is a permutationof T ) that exposes the fault j. As n and mare fixed for any given test suite and faultyprogram, higher APFD values imply higherfault-detection rates.

For ease of representation, we use an exam-ple program with 5 faults and a test suite with 5test cases to explain test-case prioritization. Ta-ble 1 shows the fault-detection capability (fromthe second column to the sixth column) andstatement coverage (from the seventh column

3. Researchers also extended the APFD metric to considerother concerns (e.g., fault-severity and cost) in software testing,and proposed metrics such as APFDc [13] and NormalizedAPFD [14].

TABLE 1Example Test Suite

Test Fault Statement1 2 3 4 5 1 2 3 4 5 6 7 8 9

A ∗ ∗ • • • • •B ∗ ∗ ∗ ∗ • • • •C ∗ ∗ ∗ • • •D ∗ • • • • • •E ∗ • • • • •

to the last column) of these test cases. In partic-ular, we use ∗ to represent that the correspond-ing fault is detected by the corresponding testcase and • to represent that the correspondingstatement is covered by the corresponding testcase. If the test suite is executed in the orderA-B-C-D-E, the corresponding APFD value is0.74.

2.2 Optimal Test-Case PrioritizationAs whether a test case can detect a fault isunknown before running the software undertest, the fault-detection capability (measuredby APFD), which can be viewed as an ul-timate goal of test-case prioritization, cannotbe used to guide scheduling the execution or-der of the test cases directly. Therefore, mosttechniques for test-case prioritization (e.g., [9],[10], [6], [11]) use structural coverage as theintermediate goal for test-case prioritization.Taking structural coverage as an intermediategoal, analogous to the APFD metric, Li etal. [4] proposed three coverage-based metrics,which are average percentage of branch cov-erage (APBC), average percentage of decisioncoverage (APDC), and average percentage ofstatement coverage (APSC), to measure the ef-fectiveness of the prioritized set of test cases toachieve structural coverage. For the simplicityof presentation, we use a generic term (i.e.,APxC) to represent the three coverage-basedmetrics and any other coverage-based metrics.Formula 2 gives the definition of APxC.

APxC = 1−∑u

j=1 TSj

nu+

1

2n(2)

In Formula 2, u denotes the number of struc-tural units (e.g., branches, statements, or meth-ods), n denotes the number of test cases in T ,and TSj denotes the first test case in T ′ (which


is a permutation of T ) that covers the structuralunit j. Note that given a different type ofcoverage, APxC denotes a different metric. Forexample, APxC denotes APSC for statementcoverage but APMC for method (function) cov-erage4.

In regression testing, a test case has been ex-ecuted on the previous version of the softwareunder test, and thus the structural coverageof a test case is known and can be used todirectly guide scheduling the execution orderof the test cases on the current version ofthe software under test. In particular, the sim-ple additional coverage-based technique is pro-posed [6], which schedules the execution orderof test cases based on the number of struc-tural units uncovered by previously selectedtest cases. Given a test suite T , we use T ′′ torepresent the set of selected test cases and thenT −T ′′ represent the set of unselected test casesin the process of test-case prioritization. Inthis process, each time the additional coverage-based technique selects a test case t from T−T ′′

so as to maximize f(T ′′ ∪ t), which representsthe number of structural units covered by T ′′

and t. Applying the additional coverage-basedtechnique to the test suite in Table 1, we getanother execution order of this test suite D-A-E-B-C, D-A-E-C-B, D-E-A-B-C, D-E-A-C-B, D-E-C-A-B, or D-E-C-B-A.

Furthermore, taking the structure coverageas a goal, it is possible to produce an optimalorder of test cases for maximizing the earlystructural coverage (e.g., branch coverage, de-cision coverage, or statement coverage). In thisarticle, we view this process as optimal test-case prioritization, which schedules the execu-tion order of test cases in order to maximizeAPxC rather than APFD. Li et al. [4] havedemonstrated that the problem of achievingoptimal APxC values for test-case prioritizationis NP-hard. In other words, it may be verycostly to achieve optimality.

Applying this optimal technique to the testsuite in Table 1, we find that test cases A and Eare the first two test cases to be executed in theoptimal order because only these two test cases

4. Method coverage is for object-oriented programs, whereasfunction coverage is for procedural programs. However, bothcoverage criteria are very similar in nature.

can achieve 100% statement coverage. That is,in all the permutations of the five test cases,such an execution order achieves the maxi-mized APSC, which is 0.77. However, suchoptimality does not refer to optimal APFD.Using this optimal order, executing the firsttwo test cases A and E, only three faults aredetected. Suppose that test cases B and C areexecuted as the first two test cases, and thenall the faults can be detected. That is, using thetest cases B and C as the first two test casesin the execution order may produce largerAPFD values. In summary, optimal test-caseprioritization mentioned in this article doesnot refer to the traditional optimal test-caseprioritization [15], which prioritizes test casesbased on the number of faults that they actuallydetect given that such information is known. Toavoid misunderstanding, we use ideal optimaltest-case prioritization to refer to traditionaloptimal test-case prioritization in this article.Note that ideal optimal test-case prioritizationis not a practical technique because the numberof faults that each test case detects is unknownbefore testing.

3 OPTIMAL COVERAGE-BASED TEST-CASE PRIORITIZATION USING ILPAs our study focuses on empirical properties ofoptimal coverage-based test-case prioritization,a precondition is to implement a technique foroptimal coverage-based test-case prioritizationto enable us to achieve optimal APxC val-ues for at least some middle-sized programs.Inspired by our previous work [11] on test-case prioritization, we model optimal coverage-based test-case prioritization by an integer lin-ear programming model. Using this model, weare able to achieve an optimal order of testcases for all the programs used in our study.

For ease of presentation, we present the opti-mal coverage-based technique in terms of state-ment coverage. Analogously, the technique canbe easily extended to other coverage criteria(e.g., method coverage, block coverage, anddecision coverage) in test-case prioritization.

Let us consider a program containingm statements (denoted as ST ={st1, st2, st3, ...stm}) and a test suite containing


n test cases (denoted as T = {t1, t2, t3, ...tn}).Let T ′ be a permutation of T . In the following,Sections 3.1, 3.2, and 3.3 present decisionvariables, constraints, and the objectivefunction in the ILP model, respectively.Section 3.4 presents the optimization to furtherreduce the size of the ILP model.

3.1 Decision VariablesThe ILP model uses two groups of decisionvariables. Section 3.1.1 presents the first groupof variables representing in what order the testcases are executed. Section 3.1.2 presents thesecond group of variables representing whichstatements are covered after the execution ofeach test case.

3.1.1 Variables for Execution OrderTo represent in what position each test case isexecuted, the ILP model uses n ∗ n Booleandecision variables (denoted as xij , where 1 ≤i, j ≤ n). Formally, xij is defined in Formula 3as follows.

xij =

{1, if the j-th test case in T ′ is ti (1 ≤ i, j ≤ n);0, otherwise.

(3)Note that Section 3.2.1 presents a group of

constraints to ensure that the set of values forxij corresponds to a permutation of T .

3.1.2 Variables for Statement CoverageTo represent which statements are covered af-ter executing each test case (i.e., accumula-tive statement coverage), the ILP model usesn ∗ m Boolean decision variables (denoted asyjk, where 1 ≤ j ≤ n and 1 ≤ k ≤ m). Formally,yjk is defined in Formula 4 as follows.

yjk =

1, if the first j test cases in T ′ covers stk

(1 ≤ j ≤ n and 1 ≤ k ≤ m);0, otherwise.

(4)Similarly, a group of constraints (presented

in Section 3.2.2) are needed to ensure that thecoverage information represented by yjk (1 ≤j ≤ n and 1 ≤ k ≤ m) is in accordance with theexecution order represented by xij (1 ≤ i, j ≤n).

3.2 Constraints

The ILP model uses two groups of constraints.Section 3.2.1 presents the first group of con-straints to ensure that values of variables inSection 3.1.1 represent a possible execution or-der of test cases. Constraints in Section 3.2.1ensure that values of variables in Section 3.1.2are consistent with values of variables in Sec-tion 3.1.1.

3.2.1 Constraints for Execution OrderThere are two sets of constraints in the ILPmodel to ensure that values of xij (1 ≤ i, j ≤ n)represent a possible execution order of testcases. First, the constraints in Formula 5 canensure that each position in the permutation ofT holds one and only one test case. Second, theconstraints in Formula 6 can ensure that eachtest case appears in the permutation of T onceand only once.

n∑i=1

xij = 1 (1 ≤ j ≤ n) (5)

n∑j=1

xij = 1 (1 ≤ i ≤ n) (6)

3.2.2 Constraints for Statement CoverageThere are also a group of constraints in theILP model to ensure that the values of yjk(1 ≤ j ≤ n and 1 ≤ k ≤ m) are in accordancewith the values of xij (1 ≤ i, j ≤ n). Thedefinition of these constraints is involved withthe coverage information of the test cases. Inparticular, for a test suite (denoted as T ={t1, t2, ...tn}) and a set of statements (denotedas ST = {st1, st2, ...stm}), Formula 7 is usedto represent whether a test case covers a state-ment.

cik =

{1, if ti covers stk;0, otherwise. (7)

Note that cik (1 ≤ i ≤ n and 1 ≤ k ≤ m)are not variables and their values can be ob-tained before test-case prioritization. Based onthe coverage information, the ILP model usesthe constraints in Formula 8 to ensure that thevalues of y1k (1 ≤ k ≤ m) are in accordance


with the statements covered by the first testcase in T ′.

n∑i=1

cik ∗ xi1 = y1k (1 ≤ k ≤ m) (8)

First, in Formula 8,∑n

i=1 cik ∗ xi1 (1 ≤ k ≤ m)represents whether stk is covered by the firsttest case in the permutation of T . The reasonis as follows. Only one test case appears asthe first test case in T ′, and thus only onevariable among xi1 (1 ≤ i ≤ n) is 1 accordingto Formula 5. As a result, for any stk (1 ≤k ≤ m), if the first test case in T ′ covers stk,∑n

i=1 cik ∗ xi1 is 1; otherwise,∑n

i=1 cik ∗ xi1 is0. Second, based on the preceding meaning of∑n

i=1 cik ∗ xi1 (1 ≤ k ≤ m), Formula 8 ensuresthat, for any 1 ≤ k ≤ m, y1k is 1 if and only ifthe first test case in T ′ covers stk.

Similarly,∑n

i=1 cik ∗ xij (2 ≤ j ≤ n, 1 ≤ k ≤m) represents whether the j-th test case in T ′

covers stk. Thus, Formulae 9, 10, and 11 ensurethat, for any 2 ≤ j ≤ n and any 1 ≤ k ≤ m, yjkis 1 if and only if the j-th test case in T ′ coversstk or at least one test case before the j-th testcase in T ′ covers stk.

yjk ≥n∑

i=1

cik ∗ xij (2 ≤ j ≤ n, 1 ≤ k ≤ m) (9)

yjk ≥ yj−1,k (2 ≤ j ≤ n, 1 ≤ k ≤ m) (10)

n∑i=1

cik ∗xij + yj−1,k ≥ yjk (2 ≤ j ≤ n, 1 ≤ k ≤ m)

(11)That is to say, the constraints in Formulae 8,

9, 10, and 11 ensure that the values of yjk (1 ≤j ≤ n and 1 ≤ k ≤ m) are in accordance withthe accumulated statement coverage for all then test cases in T ′. Note that the variable yjk isa boolean variable whose value is either 0 or 1.

3.3 Objective FunctionThe objective function is to maximize the APSCmetric, which is an instance of APxC (definedin Formula 2). In fact, the definition of APSCusing Formula 2 is equivalent to that in For-mula 12, in which m denotes the number ofstatements, n denotes the number of test cases

in T , and Ni denotes the number of statementscovered by at least one test case among the firsti test cases in T ′ (which is a permutation of T ).

APSC =

∑n−1i=1 Ni

nm+

1

2n(12)

Based on this definition, maximizing theAPSC value is thus equivalent to maximizing∑n−1

i=1 Ni. As yjk (1 ≤ j ≤ n and 1 ≤ k ≤ m)denotes whether stk is covered after executingthe first j test cases in T ′,

∑mk=1 yjk (1 ≤ j ≤ n)

is then the number of statements covered by atleast one test case among the first j test cases inT ′. Thus, Formula 13 can serve as the objectivefunction in the ILP model for optimal coverage-based test-case prioritization.

maximizen−1∑j=1

m∑k=1

yjk (13)

3.4 OptimizationIn the preceding ILP model, the number ofdecision variables is |T | ∗ |T | + |ST | ∗ |T | andthe number of constraints is 3 ∗ |ST | ∗ |T |+ 2 ∗|T | − 2 ∗ |ST |, where |T | denotes the numberof test cases to be prioritized and |ST | denotesthe number of statements of the program. As itis usually costly to solve ILP problems, it maybe necessary to further reduce the size of theILP model through optimization.

The basic idea of the optimization is to re-duce statements covered by exactly the sameset of test cases into one super statement. Forany given test suite T and a program P , asuper statement is formally defined as a set ofstatements that are executed by the same testcases in T . For any statement si and sj , if thereexists a test case in T that executes one but onlyone of these two statements, the two statementsmust belong to different super statements.Based on the concept of super statements, thestatements of P may be divided into severalsets of statements, each of which is a superstatement. Moreover, these super statementshave no common statements. Let us use theexample in Table 1 to explain the concept ofsuper statements. The five test cases (includingA, B, C, D, and E) divide the nine statementsinto the following eight super statements: {s1},{s2}, {s3,s6}, {s4},{s5},{s7},{s8}, and {s9}.


Note that the notion of super statements issimilar to that of blocks. However, statementsbelonging to one block also belong to onesuper statement, but statements from one superstatement may belong to more than one block.In particular, one super statement (whichrepresents the set of statements executed bythe same set of test cases) can be denoted asa tuple (sstk, numk), where sstk (1 ≤ k ≤ m)denotes the k-th super statement and numk

(1 ≤ numk ≤ m) denotes that the k-th superstatement actually contains numk statements.Thus, program ST can be denoted as a setof super statements: ST = {(sst1, num1),(sst2, num2), . . . , (sstm′ , numm′)}, where1 ≤ m′ ≤ m and

∑m′

k=1 numk = m. Thatis, the eight super statements in Table 1can be denoted as ST = {(sst1, 1), (sst2, 1),(sst3, 2),(sst4, 1),(sst5, 1),(sst6, 1),(sst7, 1),(sst8, 1)}.

Based on the preceding representation ofthe program ST , decision variables in For-mula 4 can be redefined on the basis of superstatements. That is to say, yjk (1 ≤ j ≤ nand 1 ≤ k ≤ m′) is 1 if and only if the firstj test cases in T ′ covers the super statement(sstk, numk). Furthermore, m in Formulae 7,8, 9, 10, and 11 should be replaced by m′.The object function should be redefined asFormula 14, as the number of statements ineach super statement also impacts the APSCvalue.

maximizen−1∑j=1

m′∑k=1

yjk ∗ numk (14)

Utilizing the notion of super statementsrather than statements, the number of variablesand the number of constraints decrease so thatit becomes less costly to solve the ILP model foroptimal test-case prioritization. On the otherside, the optimization process does not changethe functionality of the ILP model presentedfrom Section 3.1 to Section 3.3 because superstatements can be viewed as another way torepresent statements. That is, the optimizationintroduced in this subsection changes only rep-resentation of the preceding ILP model. Forexample, using the statement coverage directly,we have the following 45 variables c11, c12,

. . ., c19, . . ., c51, c52, . . ., c59, each of whosevalues represents whether the correspondingtest case in Table 1 covers the correspondingstatement. For statements s3 and s6, they arecovered by the same test cases (i.e., A, B,and D), and thus ci3 is always equal to ci6where i = 1 . . . 5. Based on the observation thatci3 = ci5, we transform the ILP model by usingthe notion of super statement. In particular, weuse the following 40 variables for statementcoverage, which are c11, c12, . . ., c18, . . ., c51,c52, . . ., c58, each of whose values representswhether the corresponding test case covers thecorresponding super statement and statementss3 and s6 belong to the same super statement.That is, using the notion of super statements,some statements are combined into one (whichis actually super statement) because test casesin T cannot distinguish them. Moreover, theoptimal technique based on other structuralcoverage (e.g., method/function coverage) canbe optimized in the same way. Furthermore,this notion can also be used to improve existingtest-case prioritization techniques.

4 STUDY IIn this study, we investigate empirical proper-ties of optimal coverage-based test-case prior-itization by answering the following researchquestions.

• RQ1: How do optimal coverage-basedtest-case prioritization and additionalcoverage-based test-case prioritizationperform differently in terms of structuralcoverage (using the APxC metric)?

• RQ2: How do optimal coverage-basedtest-case prioritization and additionalcoverage-based test-case prioritizationperform differently in terms of faultdetection (using the APFD metric)?

• RQ3: What is the difference in exe-cution time between optimal coverage-based test-case prioritization and addi-tional coverage-based test-case prioritiza-tion?

The first research question intends to inves-tigate whether the optimal technique can besuperior to the additional technique in terms of


TABLE 2Studied Programs

Abb. Program LOC #Method #Groups of #Testmutants

pt print tokens 203 20 2,356 4,072pt2 print tokens2 203 21 2,036 4,057rep replace 289 23 760 5,542sch schedule 162 20 829 2,627

sch2 schedule2 144 18 1,310 2,683tca tcas 67 11 987 1,592tot tot info 136 9 1,790 1,026sp space 6,218 136 9,657 13,585

jtopas-v0 1,831 267 56 95jt jtopas-v1 1,871 269 63 126

jtopas-v2 1,912 274 84 128siena-v0 2,225 196 50 567siena-v1 2,206 187 53 567siena-v2 2,207 187 53 567

sie siena-v3 2,242 197 54 567siena-v4 2,255 198 57 567siena-v5 2,255 198 57 567siena-v6 2,246 198 57 567siena-v7 2,232 194 59 567

structural coverage. The second research ques-tion intends to investigate whether the optimaltechnique can be superior to the additionaltechnique in terms of fault detection. The thirdresearch question intends to investigate howmuch more cost in terms of execution timethe optimal technique has than the additionaltechnique.

In the remaining of this section, we presentthe independent and dependent variables inSections 4.1 and 4.2, the target programs, faults,and test suites used in our experiment in Sec-tion 4.3, the experimental procedure in Sec-tion 4.4, the threats to validity in Section 4.5,the results in Section 4.6, and the findings ofour empirical study in Section 4.7.

4.1 Independent VariablesThis empirical study focuses on comparingoptimal coverage-based test-case prioritizationand additional coverage-based test-case prior-itization. To control the impact of coveragecriteria, we consider two intensively investi-gated coverage criteria: statement coverage andmethod (function) coverage. Therefore, our em-pirical study uses the following four indepen-dent variables: (1) the optimal test-case pri-oritization technique based on statement cov-erage, (2) the optimal test-case prioritizationtechnique based on method (function) cover-age, (3) the additional test-case prioritization

technique based on statement coverage, and (4)the additional test-case prioritization techniquebased on method (function) coverage.

Note that Section 3 presents only the optimaltest-case prioritization technique using state-ment coverage. However, it is straightforwardto extend this technique for test-case prioriti-zation using method (function) coverage. Thatis, the optimal test-case prioritization techniquebased on either statement coverage or method(function) coverage is implemented followingthe same ways as described in Section 3, whosedifference lies in only the coverage granularity.

Furthermore, in the test-case prioritizationprocess of the additional technique, it is pos-sible that some earlier selected test cases of thetest suite T have achieved the same statementcoverage as the whole test suite. In such sce-nario, we repeat the prioritization strategy byassuming that no test cases have been selectedand scheduling the remaining unselected testcases. That is, we implement the additionaltest-case prioritization technique by applyingthe additional strategy again to the remainingunselected test cases. Note that we repeat theadditional greedy strategy more than once be-cause prior work on test case prioritization [2],[16], [17], [15] usually implements the addi-tional technique in the same way5.

4.2 Dependent VariablesAccording to our research questions, we con-sider the following three dependent variables.

• Values for the APxC metric. The APxCmetric (defined in Formula 2) is to mea-sure the effectiveness of the preceding fourtechniques for structural coverage. Notethat the APxC metric becomes APSC forstatement coverage and APMC for method(function) coverage.

• Values for the APFD metric. The APFDmetric (defined in Formula 1) is to mea-sure the effectiveness of the preceding fourtechniques for fault detection.

• Execution time. The execution time is formeasuring the time efficiency of the pre-ceding four techniques.

5. Whether the additional greedy strategy repeats more thanonce in the implementation of the additional technique, doesnot affect APxC values, but may affect APFD values.


4.3 Programs, Faults, and Test Suites

In this study, we used eight C programsand eleven versions of two Java programsthat have been widely used in previous stud-ies of test-case prioritization [2], [4]. Thesetarget programs are all available from theSoftware-Infrastructure Repository (SIR6). Ta-ble 2 presents the statistics of these target pro-grams, where the last two columns present thenumber of mutant groups (each mutant groupconsists of five mutants) injected into the targetprograms and the total number of test cases inthe test pool. The former eight programs arewritten in C, whereas the latter programs arewritten in Java.

As SIR provides a large number of test casesin the test pool for each C program, in ourempirical study, we created a series of testsuites for each C program based on randomselection using a strategy similar to Rothermelet al. [6], Elbaum et al. [1], [2], and Li et al. [4].In particular, for each program, we created 100test suites each containing 10 test cases, 100test suites each containing 20 test cases, 100test suites each containing 30 test cases, and100 test suites each containing 40 test cases.When creating a test suite, we employed oneor more rounds. The aim of using more thanone round in our study was to control thesizes of created test suites. In one round, weadopted the same strategy used by Rothermelet al. [6], Elbaum et al. [1], [2], and Li etal. [4], which is to repeatedly select one test caserandomly that can increase statement coverageof previously selected test cases in this round.Given a number n (n=10, 20, 30, or 40), whilethe total number of selected test cases is smallerthan n, we continuously employed anotherround. When the total number of selected testcases is larger than n, we stopped and usedthe first n selected test cases to form the testsuite. As the Java programs in our empiricalstudy do not have a large number of test casesavailable at SIR considering the scale of theseJava programs, similar to prior studies on theseJava programs, we used all the test cases ofeach Java program as a test suite.

6. http://sir.unl.edu/portal/index.php.

As these target programs have only a smallnumber of manually injected faults availablein SIR, we generated faulty versions for eachprogram using program mutation, followingthe procedure similar to prior research [18]. Inparticular, for each program, we used mutationtesting tools (i.e., MuJava [19] for Java pro-grams, and Proteum [20] for C programs usingtheir default setting) to generate all mutants,randomly selected five unselected mutants andconstructed a faulty version by grouping thesemutants. Following this procedure, we con-structed a lot of mutant groups, each of whichis viewed as a program with multiple faults.As some mutants cannot be killed7 by any testcases of a test suite, we view these mutants asunqualified mutants8. As each mutant groupconsists of five randomly selected mutants,each group in our empirical study consists of 1-5 qualified mutants by removing those unqual-ified mutants. Furthermore, we excluded suchpair of a test suite (denoted as T ) and a mutantgroup (denoted as M ) that any test case in Tcannot kill any mutant in M , because such apair would always produce 0 APFD values forany execution order of the test cases and cannotbe used in the comparison of any test-caseprioritization techniques. Table 3 summarizesthe number of used mutant groups containingvarious numbers of qualified mutants. Accord-ing to this table, for each program, many of itsmutant groups consist of more than one fault.

4.4 Experimental Procedure

For each program, we executed the programwith a constructed test suite, recording the cov-erage information (i.e., which statements andmethods have been executed by each test case).Based on the coverage information, we appliedthe optimal test-case prioritization techniquesand the additional test-case prioritization tech-niques, recording the prioritized test cases. Inparticular, we used IBM’s integer linear pro-

7. If a test case reveals the fault injected by mutation, the testcase is said to kill the corresponding mutant [21].

8. Among these unqualified mutants, some are equivalentmutants that always behave as the original program, whereassome are not equivalent mutants but they behave as theoriginal program for the given test suite.


TABLE 3Statistics on Faults

Program Number of qualified mutantsOne Two Three Four Five

print tokens 1,9680 88,680 259,432 373,140 199,804print tokens2 4,928 33,806 140,900 311,854 322,296

replace 51,756 228,981 539,970 691,612 386,804schedule 737 9,043 41,081 116,861 163,866

schedule2 15,008 68,013 172,410 185,415 82,140tcas 30,661 81,495 122,487 112,896 42,546

tot info 1,813 15,623 99,226 284,285 315,052space 547 7,253 55,432 296,833 3,502,718

jtopas-v0 18 16 13 2 1jtopas-v1 13 20 17 11 1jtopas-v2 10 29 17 18 6siena-v0 0 1 3 16 30siena-v1 0 0 5 25 23siena-v2 0 1 4 28 20siena-v3 0 1 10 20 23siena-v4 0 1 7 24 25siena-v5 0 0 9 21 27siena-v6 1 0 10 19 27siena-v7 0 1 7 26 25

gramming tool ILOG CPLEX9 in implementingthe optimal test-case prioritization techniques.Moreover, to speedup test-case prioritizationand reduce the cost in solving the ILP model,we use the notion of super statements in im-plementing the optimal test-case prioritizationtechniques and the additional test-case prioriti-zation techniques. That is, for test-case prioriti-zation based on statement coverage, includingthe optimal techniques and the additional tech-niques, we use the notion of super statementsto represent statement coverage and similarsuper methods to represent method coverage.

Then we calculated the APSC values for theprioritized test suite of each test-case prioriti-zation technique based on statement coverage,and the APMC values for the prioritized testsuite of each test-case prioritization techniquebased on method (function) coverage. Further-more, we calculated the APFD value of eachprioritized test suite for each mutant group.

All the first study was conducted on a PCwith an Intel E5530 16-Core Processor 2.40GHz,and the operating system is Ubuntu 9.10.

4.5 Threats to ValidityThe threat to internal validity lies in the im-plementation of the optimal test-case priori-tization technique. To reduce this threat, we

9. http://www.ibm.com/software/webphere/ilog.

used IBM’s ILOG CPLEX to solve the ILPmodel and reviewed all the code of the test-case prioritization techniques before conduct-ing the experimental study. Furthermore, in theempirical study we implemented two prioriti-zation techniques on two types of widely usedcoverage and tended to generalize the con-clusions to prioritization techniques based onother types of coverage (e.g., branch coverageand MC/DC coverage). To reduce this threat,we will conduct more experiments on othertypes of coverage in future. The threats to exter-nal validity mainly lie in the target programs,faults, and test suites. All the programs usedin our study are intensively used in previousstudies on test-case prioritization [15], althoughthe programs may not be sufficiently represen-tative. This threat can be further reduced byusing more programs in various programminglanguages (e.g., C, C++, Java and C#). Similarto prior research [18], the faults are generatedby using some mutation tool, not real faultsfrom practice. However, according to the ex-perimental study conducted by Do and Rother-mel [21], mutation-generated faults are suitableto be used in studies of test-case prioritizationto replace hand-seeded faults. To reduce thethreat from test suites, we constructed a largenumber of test suites based on the collected testcases in the test pool. Further reduction of thisthreat also relies on using more real test suites.

4.6 ResultsTo answer the three research questions, we usethree subsections (each corresponding to oneresearch question) to present the results of thefirst study10.

4.6.1 RQ1: Results on APxC MetricFigures 1 and 2 present the column graphsof the mean APxC values (including APMCand APSC) of the optimal coverage-based tech-niques and the additional coverage-based tech-niques using method (function) coverage andstatement coverage. Table 5 shows the standard

10. The complete experimental data (including the imple-mentation of the optimal technique and the additional tech-nique) are accessible at https://github.com/ybwang1989/On-Optimal-Coverage-Based-Test-Case-Prioritization/wiki.


Fig. 1. Mean of APMC Results for Techniqueson Method Coverage

Fig. 2. Mean of APSC Results for Techniqueson Statement Coverage

deviations of the APxC values, where the opti-mal technique is abbreviated as “Opt.” and theadditional technique is abbreviated as “Add.”.To allow us to perform statistical analysis, sim-ilar to previous work [15], we group the resultsof the three versions of jtopas. That is, althoughjtopas has three versions jtopas−v0, jtopas−v1,and jtopas − v2, we put all the APxC resultsof the three versions into one group withoutconsidering the difference between these ver-sions and take all these APxC results as theresults for jtopas. Similarly, we do this kindof grouping when presenting other results (i.e.,APFD and execution time) of jtopas and siena.From Figures 1 and 2, we make the followingobservations.

First, for each program, the mean APxCvalue of the additional technique is no largerthan that of the optimal technique, consideringeither statement coverage or method (function)coverage. This observation is as expected sincethe optimal technique is designed to maximizethe APxC values. However, except for a fewprograms (e.g., space and jtopas), the mean ofAPxC values of the additional technique areactually the same as those of the optimal tech-

Fig. 3. Comparison on APMC Results betweenthe Optimal Technique and the Additional Tech-nique on Method Coverage

nique. Moreover, even when the optimal tech-nique outperforms the additional technique,their differences on the mean APxC resultsare ignorable. Second, the APSC values onstatement coverage are smaller than the APMCvalues on method coverage. This observationis also as expected because it is easier to pro-duce a prioritized test suite with high coarse-granularity coverage (e.g., method coverage)than with high fine-granularity coverage (e.g.,statement coverage). Third, we compare thestandard deviations of the APxC values andfind that there are no significant differences be-tween the two techniques for either statementcoverage or method (function) coverage.

Furthermore, for each program, we calcu-lated the percentage of cases on comparing theAPxC values of the optimal technique and theAPxC values of the additional technique. Inparticular, Opt. > Add. denotes the cases thatthe optimal technique outperforms the addi-tional technique, Opt. = Add. denotes the casesthat the optimal technique and the additionaltechnique produce exactly the same APxC re-sults, and Opt. < Add. denotes the cases thatthe additional technique outperforms the opti-mal technique. Figures 3 and 4 summary theseresults.

From Figures 3 and 4, the number of casesfor Opt. < Add. is always 0, and the numberof cases for Opt. = Add. is typically largerthan that for Opt. > Add. (except for space andjtopas). This observation again demonstratesthat the differences in APxC values betweenthe two techniques are ignorable.


Fig. 4. Comparison on APSC Results betweenthe Optimal Technique and the Additional Tech-nique on Statement Coverage

We further performed a one-way ANOVA11

analysis on the APxC values. The analysis re-sults are shown by Table 4, where the signifi-cance level α is set to be 0.05. Moreover, thelast row shows whether the hypothesis thatthere is no significant difference between thecompared two techniques is rejected (denotedas R) or not rejected (denoted as N). Accordingto this table, for either program and eithercoverage, the p-values are close to 1 and thusthere are no significant differences betweenthe two techniques. Furthermore, the F-valueson method (function) coverage are no largerthan those on statement coverage and the p-values on method coverage are no smaller thanthose on method coverage, demonstrating thatthe differences on method (function) coveragemay be even smaller than those on statementcoverage.

Note that for jtopas the optimal techniquesseem better than the additional techniquesfrom Figure 3, Figure 4, and Table 5. However,from Table 4 there is no significant differencebetween the two techniques because jtopashas only three APSC results and three APMCresults.

4.6.2 RQ2: Results on APFD MetricFigures 5 and 6 present the column graphs ofthe mean of the APFD values of the optimalcoverage-based technique and the additionalcoverage-based technique using method (func-tion) coverage and statement coverage. Table 6

11. The one-way ANOVA analysis was performed usingSPSS 15.0, which is an analytical software and accessible athttp://www.spss.com.

Fig. 5. Mean of APFD Results for Techniqueson Method Coverage

Fig. 6. Mean of APFD Results for Techniqueson Statement Coverage

presents the standard deviations of these APFDvalues. Unlike the APxC values given by Fig-ure 1, Figure 2, and Table 5, Figure 5, Figure 6,and Table 6 demonstrate that, for each pro-gram, the mean APFD value of the additionaltechnique is larger than that of the optimaltechnique, whereas the standard deviations onthe APFD values also indicate the superiorityof the additional technique.

Figures 7 and 8 present the statistics on caseswhere the additional technique outperformsthe optimal technique, is equal to the optimaltechnique, and is inferior to the optimal tech-nique, respectively. These two figures can con-firm the superiority of the additional techniquein general.

To check whether the differences are sig-nificant, we also perform a one-way ANOVAanalysis for the APFD value, where significancelevel is also set to be 0.05. The analysis re-sults are shown by Table 7. From this table,the differences between the two techniques areusually significant, except for the two Javaprograms. Note that the number of pairs oftest suites and mutant groups for the Javaprograms are much smaller than that for the


TABLE 4ANOVA Analysis on APxC Results between the Optimal Technique and the Additional Technique

Test-case prioritization on method coverage Test-case prioritization on statement coveragept pt2 rep sch sch2 tca tot sp jt sie pt pt2 rep sch sch2 tca tot sp jt sie

F-value 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.01 0.00 0.00 0.01 0.02 0.00 0.00 0.00 0.00 0.04 0.01 0.00p-value 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.87 0.94 1.00 0.96 0.92 0.88 0.98 1.00 1.00 0.95 0.84 0.94 1.00Result N N N N N N N N N N N N N N N N N N N N

TABLE 5Standard Deviations of APxC Results


Opt. 0.02 0.02 0.01 0.01 0.01 0.01 0.01 0.05 0.05 0.00 0.04 0.04 0.04 0.02 0.02 0.02 0.02 0.06 0.00 0.00Add. 0.02 0.02 0.01 0.01 0.01 0.01 0.01 0.05 0.05 0.00 0.05 0.04 0.04 0.02 0.02 0.02 0.02 0.06 0.00 0.00

TABLE 6Standard Deviations of APFD Results


Opt. 0.10 0.11 0.13 0.08 0.08 0.18 0.08 0.07 0.09 0.02 0.06 0.07 0.11 0.07 0.07 0.15 0.07 0.07 0.06 0.01Add. 0.09 0.09 0.10 0.08 0.06 0.15 0.06 0.07 0.10 0.02 0.05 0.06 0.10 0.05 0.05 0.13 0.05 0.07 0.07 0.01

TABLE 7ANOVA Analysis on APFD Results between the Optimal Technique and the Additional Technique

Test-case prioritization on method coveragept pt2 rep sch sch2 tca tot sp jt sie

F-value 15023.15 34628.46 89936.39 2165.38 11823.49 22870.44 41503.49 595.42 0.61 0.07p-value 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.44 0.80Result R R R R R R R R N N

Test-case prioritization on statement coveragept pt2 rep sch sch2 tca tot sp jt sie

F-value 1236.95 410.25 2366.23 4184.84 6559.60 487.55 3624.29 386.60 0.08 1.94p-value 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.77 0.16Result R R R R R R R R N N

Fig. 7. Comparison on APFD Results betweenthe Optimal Technique and the Additional Tech-nique on Method Coverage

C programs. The insignificance may be due tothe insufficient sampling space.

Fig. 8. Comparison on APFD Results betweenthe Optimal Technique and the Additional Tech-nique on Statement Coverage

4.6.3 RQ3: Results on Execution Time

Table 8 presents the average execution time(in seconds) of the optimal technique and theadditional technique for each program using ei-ther method (function) coverage and statement


TABLE 8Average Execution Time (second)

Tech. Method Coverage Statement Coverage10 20 30 40 10 20 30 40

pt Opt. 3.42 3.90 3.73 3.71 3.35 4.29 5.02 5.51Add. 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.01

pt2 Opt. 3.29 3.74 3.53 3.68 3.38 4.39 5.03 6.91Add. 0.00 0.00 0.00 0.01 0.00 0.01 0.01 0.02

rep Opt. 3.28 3.68 3.64 3.74 3.58 4.67 5.03 7.55Add. 0.00 0.00 0.00 0.01 0.00 0.00 0.01 0.01

sch Opt. 3.29 3.62 3.71 3.94 3.24 3.75 4.25 4.22Add. 0.00 0.00 0.00 0.01 0.00 0.01 0.02 0.04

sch2 Opt. 3.25 3.46 3.83 3.94 3.55 3.81 3.80 4.22Add. 0.00 0.00 0.01 0.01 0.00 0.01 0.02 0.03

tca Opt. 3.29 3.50 3.48 3.79 4.35 4.18 4.40 4.88Add. 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.01

tot Opt. 3.27 3.39 3.80 3.76 3.62 3.53 4.41 5.75Add. 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.02

sp Opt. 3.50 4.90 5.46 6.55 3.67 6.49 9.57 19.84Add. 0.00 0.00 0.00 0.01 0.01 0.03 0.05 0.07

jt Opt. 206.22 163.39Add. 0.19 0.75

sie Opt. 3,344.45 2,418.80Add. 2.82 6.61

coverage12. As the execution time consists ofonly the time used in test-case prioritization,the execution time is dependent on the numberof prioritized test cases and thus for each pro-gram we list the prioritization time for the testsuites of the same size. In particular, we use 10,20, 30, and 40 to represent test suites with thecorresponding sizes. As different versions ofeach Java program are of similar sizes and havetest suites of similar sizes, we group resultson these versions into one row. Note that eachversion of Java programs has only one testsuite.

From Table 8, we can observe that the pri-oritization time of the optimal technique istypically tens of or even hundreds of moretimes than that of the additional technique,although the prioritization time of the optimaltechnique is still acceptable.

4.7 SummaryWe summarize the main findings of the firstempirical study as follows. First, the optimal

12. The execution time of the compared techniques as wellas the analysis on the execution time is specific to our imple-mentation of the compared techniques in this empirical study.Therefore, the optimal technique and the additional techniquemay be more faster in other implementations. However, as thisarticle does not target at speedup these techniques, these tech-niques are implemented in a standard way without consideringoptimization.

coverage-based technique is ignorably betterthan the additional coverage-based techniquefor coverage (i.e., in terms of APxC). Second,the optimal coverage-based technique is sig-nificantly worse than the additional coverage-based technique for fault detection (i.e., interms of APFD). Third, the optimal coverage-based technique is much less time-efficient thanthe additional coverage-based technique.

As the optimal technique is designed to guar-antee the maximum APxC values, it is expectedto be more effective on the APxC metric thanthe additional technique, which is a simplegreedy algorithm to produce sub-optimal so-lutions. It might be somewhat surprising thatthe difference on the APxC metric betweenthe two techniques is so small. However, thisfinding confirms the observation made by Li etal. [4] that optimization techniques can achieveAPxC values similar to the additional tech-nique but are difficult to outperform the ad-ditional technique. It is also interesting to seethat the additional technique can outperformthe optimal technique on the APFD metric. Aprobable reason is that covering a structuralunit may not guarantee the detection of faultsin the unit. In fact, our previous work [22]has demonstrated that some techniques lesscompetitive than the additional technique forcoverage can in turn outperform the additionalstrategy for fault detection. Furthermore, it isof no surprise to see that the additional tech-nique is more time-efficient than the optimaltechnique. Therefore, we may draw a generalconclusion that additional coverage-based test-case prioritization should actually be superiorto optimal coverage-based test-case prioritiza-tion in practice.

5 STUDY II

As optimal coverage-based test-case prioritiza-tion takes structural coverage as the interme-diate goal and cannot guarantee to achieve theoptimal APFD results, it is interesting to learnhow good this technique is by comparing withthe ideal order of test cases, which has thelargest APFD results. Therefore, we conductanother study to investigate how good optimal


TABLE 10Statistics on Faults

Program Number of qualified mutantsOne Two Three Four Five

gzip-v0 0 0 0 0 1,997gzip-v1 0 0 0 0 2,004gzip-v2 0 0 0 0 2,276grep-v0 173 138 179 180 193grep-v1 166 150 168 183 202grep-v2 178 144 178 194 211

xmlsecurity-v0 160 118 119 127 286xmlsecurity-v1 148 132 119 138 273xmlsecurity-v2 147 131 134 135 276

time&money-v0 5 17 28 59 80time&money-v1 10 26 40 73 129

jgrapht-v0 15 30 36 58 55jgrapht-v1 19 44 64 86 79

coverage-based test-case prioritization and ad-ditional coverage-based test-case prioritizationare by comparing with the ideal optimal test-case prioritization.

To answer this research question, we imple-mented the ideal optimal test-case prioritiza-tion technique, which schedules the executionorder of test cases based on their detectedfaults. This ideal optimal test-case prioritiza-tion technique serves as the control techniqueto show the upper bound of test-case prioriti-zation, because this technique assumes that it isknown which faults each test case detects andis not applicable in practice.

5.1 SetupIn this study, we used six versions of twoC projects and seven versions of three Javaprojects that have also been used in previousstudies of test-case prioritization [23]. Table 9presents the statistics of these programs. Thefirst six programs are written in C, whereasthe latter seven programs are written in Java.Following the same procedure as Section 4,we constructed faulty versions for C programsby grouping mutants generated by a muta-tion testing tool MutGen [24] and constructedfaulty versions for Java programs by groupingmutants generated by a mutation testing toolJavalanche [25]. Table 10 summarizes the num-ber of used mutant groups containing variousnumbers of qualified mutants.

Similar to the first study in Section 4, foreach program, we constructed test suites for Cprograms and used the existing test suites for

Java programs. Based on the coverage informa-tion of each test suite, we applied the optimaltest-case prioritization technique implementedusing GUROBI13 and the additional test-caseprioritization technique, recording the priori-tized test cases.

To learn the upper bound of test-case pri-oritization, we implemented the ideal optimaltest-case prioritization technique as a controltechnique. To generate the optimal order of testcases, it is necessary to consider all possibleorders, and thus the worst-case runtime of theideal optimal algorithm may be exponential intest suite size [6]. Therefore, Rothermel et al. [6]proposed to use an additional greedy strategyinstead and called it an optimal technique,although the additional greedy strategy maynot always produce the optimal order. In thisarticle, we follow this work and implement theideal optimal technique in the same way [6].In particular, for each program with multiplefaults (i.e., a mutant group), we prioritized testcases based on their number of detected faultsthat have been not detected by existing selectedtest cases. This ideal technique produces anideal execution order of test cases. However,this ideal optimal technique is not practicalbecause whether a fault is detected by a testcase is not known before testing.

Then we calculated the APSC values for theprioritized test suite of each test-case prioriti-zation technique based on statement coverage,and the APMC values for the prioritized testsuite of each test-case prioritization techniquebased on method (function) coverage. Further-more, we calculated the APFD value of eachprioritized test suite for each mutant group. Allthis study was conducted on a PC with a Corei7-3770 Processor 3.4GHz whose operating sys-tem is Windows 7.

5.2 ResultsThe optimal technique proposed in this arti-cle takes the intermediate goal (i.e., structuralcoverage) as a goal in scheduling test cases,

13. http://www.gurobi.com/. Note that although Study Iand Study II use different tools in implementing the optimalcoverage based test-case prioritization technique, they are im-plemented following the same way as described in Section 3.


TABLE 9Studied Programs

Abb. Program LOC #Method #Groups of #Testmutants

gz gzip-v0 7,050 89 1,997 214gzip-v1 7,266 88 2,004 214gzip-v2 7,975 104 2,276 214

gp grep-v0 10,929 133 11,114 470grep-v1 12,555 149 12,472 470grep-v2 13,128 155 12,959 470

xml xmlsecurity-v0 35,579 1,231 810 97xmlsecurity-v1 35,622 1,234 810 97xmlsecurity-v2 33,523 1,146 823 95

tm time&money-v0 1,589 212 189 104time&money-v1 2,124 212 278 146

jg jgrapht-v0 8,605 310 194 48jgrapht-v1 11,251 377 292 63

whereas the ideal optimal technique takes theultimate goal (i.e., detected faults) as a goalin scheduling test cases. Thus, the ideal opti-mal technique is expected to produce an idealexecution order of test cases, although theideal technique is not practical. To learn thedifference between the optimal technique andthe ideal technique, we perform a one-wayANOVA analysis for the APxC results and theAPFD results between the optimal techniqueand the ideal optimal technique, whose analy-sis results are shown by Table 11 and Table 12.As there are a huge number of APFD results(over 1,000,000) for each C subject, we cannotconduct the one-way ANOVA analysis usingSPSS on our PC. Therefore, for each C program(i.e., gzip-v0, gzip-v1, gzip-v2, grep-v0, grep-v1, and grep-v2), we conducted the one-wayANOVA analysis on the results of all test suitesconsisting of the same number of test casesseparately, and recorded their analysis results.Then for each C subject (i.e., gzip and grep)we listed the range of their analysis results inTable 12. From this table, the optimal techniquesignificantly outperforms the ideal techniquein terms of APxC results, but the latter sig-nificantly outperforms the former in terms ofAPFD results.

Similarly, we perform a one-way ANOVAanalysis for the APxC results and the APFDresults between the additional technique andthe ideal optimal technique, whose analysis re-sults are shown by Table 13 and Table 14. Fromthis table, we got the similar conclusion asthe comparison between the optimal technique

TABLE 11ANOVA Analysis on APxC Results between the

Optimal Technique and the Ideal Technique

Test-case prioritization on method coveragegz gp xml tm jg

F-value 12151.32 9721.26 19.81 456.15 7.51p-value 0.00 0.00 0.00 0.00 0.01Result R R R R R

Test-case prioritization on statement coveragegz gp xml tm jg


TABLE 12ANOVA Analysis on APFD Results between the

Optimal Technique and the Ideal Technique


F-value 82.89-1111065.00

23747.13-31228.53

598.97 300.95 376.66

p-value 0.00 0.00 0.00 0.00 0.00Result R R R R R


F-value 22.28-1066038.00

13998.26-30333.62

516.58 339.68 391.00


and the ideal technique. That is, the additionaltechnique significantly outperforms the idealtechnique in terms of APxC results, but thelatter significantly outperforms the former interms of APFD results.

From this study, the ideal optimal test-caseprioritization achieves the best APFD resultsbut the worst APxC results. This observationalso consolidates our findings in the first study.That is, it is not worthwhile to purse optimality


TABLE 13ANOVA Analysis on APxC Results between theAdditional Technique and the Ideal Technique





TABLE 14ANOVA Analysis on APFD Results between theAdditional Technique and the Ideal Technique


F-value 68.63-17492.00

24618.58-30140.31

600.42 253.92 438.16



F-value 26.28-20885.99

13818.13-28622.88

525.88 263.44 382.03


by taking the coverage as an intermediate goalin test-case prioritization.

6 DISCUSSION

In this section, we discuss some related issues.First, due to the difference between the ul-

timate goal (i.e., APFD) and the intermediategoal (i.e., APxC), when pursing the maximumAPxC values, the optimal coverage-based test-case prioritization technique generates somescheduled test suites that overfit for the APxCvalues and have smaller APFD values thanthe simple additional coverage-based test-caseprioritization technique. As the fault-detectioncapability of test cases is unknown withoutrunning test cases, it is impossible to use theultimate goal to guide test-case prioritizationand thus it is necessary to use an intermediategoal instead. To bridge the gap between theultimate goal and the intermediate goal andovercome the overfitting problem in optimaltest-case prioritization, researchers may tunethe intermediate goal slightly. For example, in

our previous work [22], we unified the two typ-ical greedy test-case prioritization techniques(i.e., the total test-case prioritization and theadditional test-case prioritization techniques)by using a parameter p on APxC.

Second, although our empirical study is ontest-case prioritization, its conclusions may begeneralized to other software-testing tasks [26],[27] (e.g., test-suite reduction [28]). Test-case prioritization is only one of the typicalsoftware-testing tasks with the characteristicsthat pursing the intermediate-goal optimal-ity may become overfit for the intermediategoal and thus become less effective than someheuristic or greedy techniques for the ultimategoal. Other software-testing tasks share thesame characteristics. Test-suite reduction is pro-posed in regression testing, whose aim is tofind the minimal set of the existing test suiteso that the minimal subset has the same fault-detection capability as the whole test suite.As it is impossible to know the fault-detectioncapability of test cases without running testcases, researchers in test-suite reduction alsotend to use intermediate goals (e.g., structuralcoverage) to replace the ultimate goal (i.e.,fault-detection capability). When pursing theoptimality, existing test-suite reduction basedon ultimate goals may also become less ef-fective when measured on the ultimate goal.In this article, we have investigated this prob-lem based on the empirical study of test-case prioritization due to effort limit; a similarproblem occurs in many other software-testingtasks and the conclusions from the empiricalstudy may be generalized. In other words, re-searchers should take precautions in pursuingoptimal solutions using the intermediate goal.

7 RELATED WORK

Test-case prioritization is an optimization prob-lem, similar to test-suite reduction, which isconceptually related to the set cover problemand the hitting set problem. For ease of presen-tation, we classify existing research on test-caseprioritization into four groups.

The first group focuses on investigatingvarious coverage criteria for test-caseprioritization [29]. Besides statement


coverage, researchers have also investigatedfunction/method coverage [1], branchcoverage [30], definition-use associationcoverage, modified condition/decisioncoverage [31], specification coverage [32],the ordered sequence of program entities [33].Furthermore, Korel et al. [3], [34] proposedmodel-based test-case prioritization, whichfirst maps the change on the source code tomodel elements and then schedules the orderof test cases based on their relevance to thesemodel elements. As our research focuses onprioritization strategies, we adopt two typicalcoverage criteria in our empirical study:statement coverage and function/methodcoverage. Different from these coverage-basedtechniques, Ledru et al. [35] do not assumethe existence of code or specification and thusproposed a prioritization technique based onthe string distances on the text of test cases,which may be useful for initial testing.

The second group focuses on strategies fortest-case prioritization. Besides the additionalstrategy, researchers have also widely investi-gated the random strategy and the total strat-egy. Li et al. [4] investigated some optimizationtechniques based on genetic algorithms andhill climbing for test-case prioritization. TheirAPxC results show that the additional strategyoutperforms both the strategy based on a ge-netic algorithm and the strategy based on hillclimbing, but the differences are insignificant.Different from their work, in this article, weformalize optimal coverage-based test-case pri-oritization as an ILP problem. Furthermore, Liet al. [36] conducted a simulation experiment tostudy five search algorithms for test-case prior-itization, including the total greedy algorithm,the additional greedy algorithm, the 2-optimalgreedy algorithm, the hill climbing algorithm,and the genetic algorithm. Their results showthat the additional greedy algorithm and the2-optimal greedy algorithm are both preferableand outperform the others. Jiang et al. [10] in-vestigated a random adaptive strategy for test-case prioritization. According to their results,the random adaptive strategy is less effectivein terms of APFD but also less costly than theadditional strategy. Hao et al. [37] combinedthe output of test cases to improve test-case pri-

oritization and this technique sometimes out-performs the total and the additional strategiesin terms of APFD. Zhang et al. [22], [23] in-vestigated strategies with mixed flavor of theadditional strategy and the total strategy. Theirresults demonstrate that there are mixed strate-gies that can significantly outperform both theadditional strategy and the total strategy interms of APFD.

Our research focuses on investigating empir-ical properties of the optimal strategy in termsof APxC. To our knowledge, our research is thefirst study on the optimal strategy.

The third group focuses on practical con-straints (e.g., fault severity [38] and time bud-get [11]) in test-case prioritization. For example,as time constraints may strongly affect the be-havior of test-case prioritization, Suri and Sing-hal [39] proposed an ant colony optimizationbased technique to prioritize test cases withtime constraints. Do et al. [40] conducted aseries of experiments to evaluate the effectsof time constraints on the costs and benefitsof test-case prioritization. Marijan et al. [41]proposed to use the test execution time as aconstraint and scheduled the execution order oftest cases based on historical failure data, testexecution time, and domain-specific heuristicsfor industrial conferencing software in continu-ous regression testing. As the cost of test casesand severity of faults vary, Huang et al. [42]proposed a cost-cognizant technique using his-torical records from the latest regression test-ing, which determines the execution order witha genetic algorithm. Zhang et al. [43] proposedto prioritize test cases based on varying testrequirement priorities and test case costs byadapting existing test-case prioritization tech-niques. Note that the ILP model proposed byZhang et al. [11] is for selecting test cases underthe time constraint, but not a prioritizationstrategy like the ILP model presented in thisarticle. Our research reported in this articlefocuses on investigating the optimal strategy inthe context without these practical constraints.However, it should be interesting to further in-vestigate the optimal strategy in contexts withone or more practical constraints.

The fourth group focuses on empirical stud-ies on test-case prioritization [1], [2], [6]. Exist-


ing empirical studies investigate various fac-tors (e.g., programming languages [44], cover-age granularity [2], fault type [18], [21]) thatmay influence the effectiveness of existing test-case prioritization techniques, whereas our em-pirical study investigates empirical differencesbetween the additional strategy and the opti-mal strategy. However, it may also be interest-ing to further investigate the influencing factorsfor the optimal strategy.

Furthermore, some researchers apply test-case prioritization to solve other software-engineering problems, e.g., mutation testing,fault localization. Zhang et al. [45] proposedto prioritize test cases based on the order thata test case killing the mutant is run earlier toreduce the cost of mutation testing. Sanchezet al. [46] proposed to apply test-case priori-tization to testing software product lines. Yooet al. [47] proposed a cluster-based test-caseprioritization technique, which first clusterstest cases using the agglomerative hierarchicalclustering technique and then prioritizes theseclusters to reduce human involvement in test-case prioritization.

8 CONCLUSION

As the additional strategy for coverage-basedtest-case prioritization is surprisingly compet-itive according to some prior work, we arecurious about how the optimal strategy forcoverage-based test-case prioritization wouldperform differently from the additional strat-egy for coverage-based test-case prioritization.Therefore, we have conducted an empiricalstudy on comparing these two strategies (us-ing APxC, APFD, and execution time) withstatement coverage and method coverage. Ac-cording to our empirical results, the optimalstrategy is only slightly more effective than theadditional strategy with regard to APxC, butsignificantly less effective than the additionalstrategy with regard to both APFD and theexecution time, demonstrating the inferiority ofthe optimal strategy in practice. Furthermore,we have implemented the ideal optimal strat-egy. Taking this strategy as the upper boundof test-case prioritization, we have conducted

another empirical study on comparing the ef-fectiveness of the optimal and the additionalstrategies. From this empirical study, the idealstrategy significantly outperforms the optimaland additional strategies in terms of APFD, butsignificantly worse in terms of APxC.

Therefore, when pursing the optimality intest-case prioritization using the intermediategoal (i.e., APxC), the produced solutions areoverfit for this goal and less effective thansolutions produced by the greedy techniquein terms of the ultimate goal (i.e., APFD).This conclusion indicates that it may not beworthwhile to pursue the optimality in test-case prioritization by taking coverage as anintermediate goal considering the efforts. Infuture work, we plan to further investigate theoptimality issue in other software-testing tasks(e.g., test-case generation or test-suite reduc-tion).

ACKNOWLEDGMENTS

This work is supported by the National 973Program of China No. 2015CB352201, the Na-tional Natural Science Foundation of Chinaunder Grants No.61421091, 61225007, 61529201,61522201, and 61272157. This work is alsosupported by the National Science Foun-dation of the United States under grantsNo. CCF-1349666, CCF-1409423, CNS-1434582,CCF-1434590, CCF-1434596, CNS-1439481, andCNS-1513939. Besides, we would like to thankShanshan Hou, Xinyi Wan, and Chao Guo.They contributed to the early discussion of thiswork.

REFERENCES

[1] S. Elbaum, A. Malishevsky, and G. Rothermel, “Prioritiz-ing test cases for regression testing,” in Proceedings of theInternational Symposium on Software Testing and Analysis,2000, pp. 102–112.

[2] S. Elbaum, A. G. Malishevsky, and G. Rothermel, “Testcase prioritization: A family of empirical studies,” IEEETransactions on Software Engineering, vol. 28, no. 2, pp. 159–182, 2002.

[3] B. Korel, L. H. Tahat, and M. Harman, “Test prioritizationusing system models,” in Proceedings of the 21st IEEEInternational Conference on Software Maintenance, 2005, pp.559–568.

[4] Z. Li, M. Harman, and R. Hierons, “Search algorithmsfor regression test case prioritization,” IEEE Transactionson Software Engineering, vol. 33, no. 4, pp. 225–237, 2007.


[5] C. D. Nguyen, A. Marchetto, and P. Tonella, “Test caseprioritization for audit testing of evolving Web servicesusing information retrieval techniques,” in Proceedings ofthe 9th International Conference on Web Services, 2011, pp.636–643.

[6] G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold,“Prioritizing test cases for regression testing,” IEEE Trans-actions on Software Engineering, vol. 27, no. 10, pp. 929–948,2001.

[7] W. Wong, J. Horgan, S. London, and H. Agrawal, “Astudy of effective regression testing in practice,” in Pro-ceedings of the International Symposium on Software Reliabil-ity Engineering, 1997, pp. 230–238.

[8] S. Yoo and M. Harman, “Regression testing minimisation,selection and prioritisation: A survey,” Software Testing,Verification and Reliability, vol. 22, no. 2, pp. 67–120, 2012.

[9] S. Hou, L. Zhang, T. Xie, and J. Sun, “Quota-constrainedtest-case prioritization for regression testing of service-centric systems,” in Proceedings of the International Confer-ence on Software Maintenance, 2008, pp. 257–266.

[10] B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse, “Adaptiverandom test case prioritization,” in Proceedings of the 24thIEEE/ACM International Conference on Automated SoftwareEngineering, 2009, pp. 257–266.

[11] L. Zhang, S. Hou, C. Guo, T. Xie, and H. Mei, “Time-aware test-case prioritization using integer linear pro-gramming,” in Proceedings of the International Symposiumon Software Testing and Analysis, 2009, pp. 213–224.

[12] H. Williams, Model Building in Mathematical Programming.New York: John Wiley, 1993.

[13] S. Elbaum, A. Malishevsky, and G. Rothermel, “Incor-porating varying test costs and fault severities into testcase prioritization,” in Proceedings of the 23rd InternationalConference on Software Engineering, 2001, pp. 329–338.

[14] X. Qu, M. B. Cohen, and K. M. Woolf, “Combinatorialinteraction regression testing: A study of test case gener-ation and prioritization,” in Proceedings of the 23rd IEEEInternational Conference on Software Maintenance, 2007, pp.255–264.

[15] H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, andG. Rothermel, “A static approach to prioritizing JUnit testcases,” IEEE Transactions on Software Engineering, vol. 38,no. 6, pp. 1258–1275, 2012.

[16] B. Jiang and W. K. Chan, “Input-based adaptive ran-domized test case prioritization: A local beam searchapproach,” Journal of Systems and Software, vol. 105, pp.91–106, 2015.

[17] L. Mei, Y. Cai, C. Jia, B. Jiang, W. K. Chan, Z. Zhang,and T. H. Tse, “A subsumption hierarchy of test caseprioritization for composite services,” IEEE Transactionson Services Computing, vol. 8, no. 5, pp. 658–673, 2015.

[18] H. Do and G. Rothermel, “A controlled experiment as-sessing test case prioritization techniques via mutationfaults,” in Proceedings of the 21st International Conferenceon Software Maintenance, 2005, pp. 411–420.

[19] Y.-S. Ma, J. Offutt, and Y. R. Kwon, “MuJava: An auto-mated class mutation system,” Software Testing, Verificationand Reliability, vol. 15, no. 2, pp. 97–133, 2005.

[20] M.E.Delamaro and J. C. Maldonado, “Proteum - A toolfor the assessment of test adequacy for C programs,” inProceedings of the International Conference on Performabilityin Computing Systems, 1996, pp. 79–95.

[21] H. Do and G. Rothermel, “On the use of mutation faultsin empirical assessments of test case prioritization tech-

niques,” IEEE Transactions on Software Engineering, vol. 32,no. 9, pp. 733–752, 2006.

[22] L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei,“Bridging the gap between the total and additional test-case prioritization strategies,” in Proceedings of the 35thInternational Conference on Software Engineering, 2013, pp.192–201.

[23] D. Hao, L. Zhang, L. Zhang, G. Rothermel, and H. Mei,“A unified test case prioritization approach,” ACM Trans-actions on Software Engineering and Methodology, vol. 24,no. 2, p. 10, 2014.

[24] J. Andrews, L. Briand, and Y. Labiche, “An empirical com-parison of test suite reduction techniques for user-session-based testing of Web applications,” in Proceedings of the27th IEEE International Conference on Software Engineering,2005, pp. 402–411.

[25] D. Schuler and A. Zeller, “Javalanche: Efficient mutationtesting for Java,” in Proceedings of the ACM Symposium onFoundations of Software Engineering, 2009, pp. 297–298.

[26] T. Y. Chen, F.-C. Kuo, D. Towey, and Z. Zhou, “A revisitof the tree studies related to random testing,” SCIENCECHINA Information Sciences, vol. 58, no. 5, 2015.

[27] W. TSAI, X. Bai, and Y. Huang, “Software-as-a-service(SaaS): Perspectives and challenges,” SCIENCE CHINAInformation Sciences, vol. 57, no. 5, 2014.

[28] D. Hao, L. Zhang, X. Wu, H. Mei, and G. Rothermel,“On-demand test suite reduction,” in Proceedings of 34thInternational Conference on Software Engineering, 2012, pp.738–748.

[29] C. Fang, Z. Chen, and B. Xu, “Comparing logic cover-age criteria on test case prioritization,” SCIENCE CHINAInformation Sciences, vol. 55, no. 12, 2012.

[30] M. Baluda, P. Braione, G. Denaro, and M. Pezz, “En-hancing structural software coverage by incrementallycomputing branch executability,” Software Quality Journal,vol. 19, no. 4, pp. 725–751, 2011.

[31] J. A. Jones and M. J. Harrold, “Test-suite reduction andprioritization for modified condition/decision coverage,”IEEE Transactions on Software Engineering, vol. 29, no. 3,pp. 195–209, 2003.

[32] G. Kovacs, G. A. Nemeth, M. Subramaniam, and Z. Pap,“Optimal string edit distance based test suite reductionfor SDL specifications,” in Proceedings of the 14th Interna-tional SDL Conference on Design for Motes and Mobiles, 2009,pp. 82–97.

[33] C. Fang, Z. Chen, K. Wu, and Z. Zhao, “Similarity-based test case prioritization using ordered sequences ofprogram entities,” Software Quality Journal, vol. 22, pp.335–361, 2014.

[34] B. Korel, G. Koutsogiannakis, and L. H. Tahat, “Applica-tion of system models in regression test suite prioritiza-tion,” in Proceedings of the IEEE International Conference onSoftware Maintenance, 2008, pp. 247–256.

[35] Y. Ledru, A. Petrenko, S. Boroday, and N. Mandran,“Prioritizing test cases with string distances,” AutomatedSoftware Engineering, vol. 19, no. 1, pp. 65–95, 2012.

[36] S. Li, N. Bian, Z. Chen, D. You, and Y. He, “A simula-tion study on some search algorithms for regression testcase prioritization,” in Proceedings of the 10th InternationalConference on Quality Software, 2010, pp. 72–81.

[37] D. Hao, X. Zhao, and L. Zhang, “Adaptive test-caseprioritization guided by output inspectation,” in Proceed-ings of the 37th Annual International Computer Software andApplications Conference, 2013, pp. 169–179.


[38] K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S.Roos, “Time-aware test suite prioritization,” in Proceed-ings of the International Symposium on Software Testing andAnalysis, 2006, pp. 1–11.

[39] B. Suri and S. Singhal, “Analyzing test case selection& prioritization using ACO,” ACM SIGSOFT SoftwareEngineering Notes, vol. 36, no. 6, pp. 1–5, 2011.

[40] H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, “Theeffects of time constraints on test case prioritization: Aseries of controlled experiments,” IEEE Transactions onSoftware Engineering, vol. 36, no. 5, pp. 593–617, 2010.

[41] D. Marijan, A. Gotlieb, and S. Sen, “Test case prioriti-zation for continuous regression testing: An industrialcase study,” in Proceedings of the 29th IEEE InternationalConference on Software Maintenance, 2013, pp. 540–543.

[42] Y.-C. Huang, K.-L. Peng, and C.-Y. Huang, “A history-based cost-cognizant test case prioritization technique inregression testing,” Journal of Systems and Software, vol. 85,no. 3, pp. 626–637, 2012.

[43] X. Zhang, C. Nie, B. Xu, and B. Qu, “Test case prioritiza-tion based on varying test requirement priorities and testcase costs,” in Proceedings of the 7th International Conferenceon Quality Software, 2007, pp. 15–24.

[44] H. Do, G. Rothermel, and A. Kinneer, “Prioritizing JU-nit test cases: An empirical assessment and cost-benefitsanalysis,” Empirical Software Engineering, vol. 11, no. 1, pp.33–70, 2006.

[45] L. Zhang, D. Marinov, and S. Khurshid, “Faster mutationtesting inspired by test prioritization and reduction,” inProceedings of the 2013 International Symposium on SoftwareTesting and Analysis, 2013, pp. 235–245.

[46] S. S. Ana B. Sanchez and A. Ruiz-Cortes, “A comparisonof test case prioritization criteria for software productlines,” in Proceedings of the 7th IEEE International Conferenceon Software Testing, 2014, pp. 41–50.

[47] S. Yoo, M. Harman, P. Tonella, and A. Susi, “Clusteringtest cases to achieve effective & scalable prioritisationincorporating expert knowledge,” in Proceedings of the 18thInternational Symposium on Software Testing and Analysis,2009, pp. 201–211.

Dan Hao is an Associate professor atthe School of Electronics Engineering andComputer Science, Peking University, P.R.China. She received her Ph.D. in ComputerScience from Peking University in 2008,and the B.S. in Computer Science from theHarbin Institute of Technology in 2002. Hercurrent research interests include softwaretesting and debugging. She is a senior

member of ACM.

Lu Zhang is a professor at the Schoolof Electronics Engineering and ComputerScience, Peking University, P.R. China. Hereceived both PhD and BSc in ComputerScience from Peking University in 2000and 1995 respectively. He was a postdoc-toral researcher in Oxford Brookes Uni-versity and University of Liverpool, UK.He served on the program committees of

many prestigious conferences, such FSE, ISSTA, and ASE. Hewas a program co-chair of SCAM2008 and will be a programco-chair of ICSM17. He has been on the editorial boards ofJournal of Software Maintenance and Evolution: Research andPractice and Software Testing, Verification and Reliability. Hiscurrent research interests include software testing and analysis,program comprehension, software maintenance and evolution,software reuse and component-based software development,and service computing.

Lei Zang is a third-year Computer ScienceMaster student from Software EngineeringInstitute, Peking University. He will gradu-ate in the July of 2016 and become a soft-ware engineer. His current research inter-ests include software testing and analysis.

Yanbo Wang is a PhD student in the De-partment of Regional Economics, Schoolof Government, Peking University. He re-ceived his B.S. in software engineeringfrom Huazhong University of Science andTechnology, Wuhan, China, in 2012. Hismajor interests are in the areas of complexsystem, complex dynamics and economiccomputation.

Xingxia Wu is a software engineer at a re-gional bank, China. She received both theBS and MS degrees in computer sciencefrom Peking University. Her research inter-ests include software testing and analysis.


Tao Xie is an Associate Professor in theDepartment of Computer Science at theUniversity of Illinois at Urbana-Champaign,USA. His research interests are softwaretesting, program analysis, software analyt-ics, software security, and educational soft-ware engineering. He is a senior memberof IEEE.

Documents

A To Be Optimal Or Not in Test-Case Prioritizationthe latter signiﬁcantly outperforms the former in terms of either fault detection or execution time. As the optimal technique schedules