Int J Softw Tools Technol Transfer (2012) 14:461–475DOI 10.1007/s10009-012-0226-1

ICTSS 2010

Multi-objective optimization algorithms applied to the classintegration and test order problem

Silvia Regina Vergilio · Aurora Pozo · João Carlos Garcia Árias ·Rafael da Veiga Cabral · Tiago Nobre

Published online: 1 March 2012© Springer-Verlag 2012

Abstract In the context of object-oriented software, acommon problem is the determination of test orders for theintegration test of classes, known as the class integration andtest order (CITO) problem. The existing approaches, basedon graphs, usually generate solutions that are sub-optimal,and do not consider the different factors and measures thatcan affect the construction of stubs. To overcome this limi-tation, solutions based on genetic algorithms (GA) have pre-sented promising results. However, the determination of acost function, which is able to generate the best solution,is not always a trivial task, mainly for complex systems.Therefore, to better represent the CITO problem, we intro-duce, in this paper, a multi-objective optimization approach,to generate a set of good solutions that achieve a balancedcompromise between the different measures (objectives).Three different multi-objective optimization algorithms(MOA) were implemented: Pareto ant colony, multi-objectiveTabu search and non-dominated sorting GA. The approachis applied to real programs and the obtained results allowcomparison with the simple GA approach and evaluation ofthe different MOA.

Keywords Multi-objective optimization · Integration test ·Object-oriented software

1 Introduction

Test is generally conducted in an incremental way [27]. Forexample, in the context of object-oriented (OO) software, atest strategy includes different levels [3,16]: method, class,

S. R. Vergilio (B) · A. Pozo · J. C. G. Árias · R. da Veiga Cabral ·T. NobreComputer Science Department, Federal University of Paraná,DInf-UFPR, CP 19081, Curitiba 19031-970, Brazile-mail: silvia@inf.ufpr.br

cluster and system levels. In general terms, the base code(or component) is developed, unit tested, integrated andtested.

In the integration test of OO software, an order to inte-grate and test classes is required. The determination of suchorder is an important task, since it affects [28]: the orderin which classes are developed; the design of test cases; theorder in which inter-class faults are detected; and the numberof created stubs for classes.

A stub is necessary when a class A requires another class Bthat is not available during the test of A, i.e., A depends on B.The stub is then constructed to emulate the behavior of B. Thecreation of stubs is an expensive and error-prone task and,consequently, the determination of the best inter-class order,which minimizes stubbing efforts, is an important problem tobe solved, called the class integration and test order (CITO)problem [1].

When there are no dependency cycles, the CITO problemcan be solved by a simple reverse topological ordering ofclasses considering their dependencies [22]. However, recentstudies with real Java systems show that the presence of com-plex cycles is very common [24], and solutions to find anoptimal order to minimize the stubbing effort are essential.

In the literature, different kinds of approaches were pro-posed to the CITO problem. They generally use graph-basedalgorithms [1,7,22,28,29] and present some limitations: inmany cases, the solutions found are sub-optimal; and theapproaches do not consider other measures that may influ-ence the stubbing process cost [6], such as number of attri-butes of a class, number of calls or distinct methods invoked,and other aspects related, for example, to organizational orcontractual reasons. For that reason, a solution based on agenetic algorithm (GA) was proposed in [6]. The authors usecoupling measures to determine the stubbing complexity. Theimplemented GA was evaluated with real programs by using

123

462 S. R. Vergilio et al.

four fitness functions: number of broken dependencies, num-ber of attributes, number of methods, and a geometric averageof attributes and methods [5]. The GA solution allows the useof different factors to establish the test orders. However, thechoice of the more adequate evaluation function for the GAis not always a trivial task. It can be very expensive and laborintensive for complex cases. This makes difficult the practicaluse of the GA solution based on the aggregation function.

To overcome this limitation and to obtain more adequatesolutions to the CITO problem, in a previous work [8], weintroduced a multi-objective optimization algorithm (MOA)based on ant colony optimization (ACO) [11,12]. This algo-rithm is a meta-heuristic and has been successfully appliedto solve many combinatorial optimization problems, such astraveling salesman problem, sequential ordering problem, setcovering problem, etc. To apply ACO, the optimization prob-lem is transformed into the problem of finding the best pathon a weighted graph.

With the introduced multi-objective approach, real con-straints and diverse factors that may influence the CITO prob-lem can be considered. However, to implement and evaluatethe introduced approach, we can find in the literature manyother MOA and it is impossible to determine a priori whichone is the best choice for a given problem. To answer thisquestion, this paper extends the previous work by applyingtwo other MOA: NSGA-II (non-dominated sorting GA [10])and MTabu (multi-objective optimization using Tabu search[15]). These algorithms were selected due to the followingreasons. NSGA-II is one of the most known and used MOA[17]. On the other hand, MTabu is not evolutionary but basedon a Tabu search algorithm, one of the most applied algorithmfrom the operations research field [14].

The three algorithms were executed with the same bench-mark used by Briand et al. [5]. This benchmark is composedof real systems with varying complexities, which permitsgood insights about the use of both approaches. The solu-tions returned by the MOA are evaluated according to thePareto dominance concept [25] and represent a good trade-offbetween the coupling measures: number of methods and attri-butes. The results were compared with a GA approach, namedhere single objective and genetic algorithm (SOGA)-basedapproach and point out that the multi-objective approachpresents a variety of good solutions even for complex cases.Furthermore, the MOA were compared. In the comparison,NSGA-II presents better results than PACO and MTabu for allsystems. NSGA-II returns a more diversified set of non-dom-inated solutions and allows the use of different alternativesin the integration test plans.

The paper is organized as follows. Section 2 presents areview of works related to the CITO problem including theSOGA approach. Section 3 introduces our multi-objectiveapproach and describes the implemented algorithms: PACO,NSGA-II and MTabu. Section 4 discusses the experimental

results obtained and compares the MOA. Section 5 concludesthe paper and contains future research works.

2 The class and integration test order problem

In the integration test phase, the goal is to check the structureof the software, established in the design phase, looking forfaults associated with the interfaces between modules, andtesting whether the integrated system components work asdesired. In the object-oriented (OO) context, the modules areclasses.

The CITO problem [1] can be defined as the identifica-tion of a priority order for integrating the classes. Such orderis related to the number of created stubs, which should bereduced to the possible minimum, since stubs represent addi-tional costs to the project. Hence, the minimization of thenumber of stubs created during the integration testing is animportant problem to be solved. It is also important to reducecost and effort spent in the stubbing process.

Most approaches to the CITO problem are based ongraphs: Example of such graphs are: test dependency graphs(TDG [29]) and object relation diagrams (ORD [22]). Inthe TDG, there are two types of nodes representing eithera class or a method, and different kind of edges to repre-sent dependencies between classes, between methods andbetween methods and classes. The ORD is a directed graphwhere the nodes represent classes and the edges representtheir relationships. When there are no dependency cyclesbetween the classes, the CITO problem can be solved bya simple reverse topological ordering of classes consideringtheir dependencies [22]. However, this is not always the case,since most systems contain cycles [24].

The approach proposed by Kung et al. [22] was the firstone to address this problem. The idea of this work is to iden-tify strongly connected components (SCCs) in the ORD, andto remove associations until no cycles remain. When thereare two or more candidate associations for cycle breaking, arandom selection is performed.

A solution was proposed by Tai and Daniels [28] basedon two class levels. Major-level numbers are assigned toclasses based only on inheritance and aggregation depen-dencies. Then within each major level, minor-level num-bers are assigned, based on association dependencies. SCCsare identified in the major level and each edge of the SCCsreceives a weight based on the related incoming and outgoingdependencies. To break cycles, edges with higher weights areselected considering they are likely involved in more cycles.However, according to Briand et al. [7], this assumption isnot always true. There are cases where class associationsare not involved in cycles, and the obtained solution is sub-optimal in terms of the required number of test stubs.

The work of Traon et al. [29] identifies the SCCs by usingTarjan’s algorithm. The weights are assigned by the sum of

123

MOA applied to the CITO problem 463

incoming and outgoing frond dependencies for a given classwithin the SCC. The frond dependency is a kind of edge,which is defined as going from a vertex (class) to one of itsancestors (a vertex that is traversed before it in a depth-firstsearch, i.e., the corresponding class depends on it, directlyor indirectly). For each nontrivial SCC (with more than onevertex), the procedure above is then called recursively. Theapproach is non-deterministic because different sets of edgescan be labeled as frond depending on the starting node, andwhen two or more nodes have the same weight, the selectionis arbitrary.

Briand et al. [7] proposed an approach that does not breakinheritance and aggregation edges and computes the weightsin a more precise way. The association edges in the SCCs areassigned with weights corresponding to the estimated num-ber of involved cycles. This number is calculated accord-ing to the number of incoming and outgoing dependencies.The edge with the highest weight is removed. The process isrepeated until no SCCs remain.

Abdurazik and Offutt [1] consider more information in theminimization of the stubbing effort. The weights are derivedfrom quantitative analysis of nine introduced kinds of cou-plings and are assigned to the edges and nodes. The cou-pling measures use the number of parameters, the number ofreturn value types, the number of variables and the numberof methods. The node weight is related to the estimated costof removing it. If a class is used by multiple classes, then allor part of the same stub for that class may be shared amongall classes that use it, thus reducing the stubbing cost. Theweight of a node is at least as high as the maximal weightof all incoming edges (assuming total sharing of the stub),and no higher than the sum of the weights of all incomingedges (assuming no sharing of the stub). The evaluation ofthis approach presented positive results when compared withthe approaches described before [1].

Kraft et al. [21] introduce and implement an approach,named class and order system (COS), which uses the ORDand six types of dependency edges: inheritance, composition,association, dependency, polymorphism and owned element.The weights of the edges are assigned considering their typesand a trial and error method. The edges with lowest weightare first removed to break the cycles. The number of stubsand the cost to construct them are considered.

Mao et al. [23] use an extension of the ORD and con-sider three heuristics: cycling weight, direction factors ofedges and association intensity to remove edges. The algo-rithm, named AICTO, first removes pair cycles (cycles withlength two) and then other kind of cycles are broken. To breakcycles, edges with highest weights are first selected.

Differently from the works that consider the type of rela-tionships between the classes, the work of Jaroenpibookiet al. [18,19] uses a TDG and class slicing techniques. Thestubs are constructed considering isolated parts from the clas-

ses. First, the SCCs are identified and classified according tothe cycle type. After then, classes in SCCs are sliced andsorted to generate an order.

All the works mentioned above use models based ongraphs to represent the dependencies between classes, andgraph-based algorithms are used to break the cycles. Mostof them recursively identify SCCs and in each SCC removeone dependency that maximizes the number of broken cycles[6]. They optimize the decision without determining the con-sequences on the ultimate results. There might be situationswhere breaking two dependencies has a lower cost than break-ing only one that would make the graph acyclic in one step. Inmany cases, the obtained solutions are sub-optimal. Anotherdisadvantage pointed out by Briand et al. [6] is that the cost ofconstructing a stub may depend on many factors and cannotbe completely measured or estimated. To adapt most graph-based algorithms to consider these factors seems difficult oreven impossible.

To allow the use of different kinds of constraints and cou-pling measures, the approach based on genetic algorithms[6] has presented the most promising results. The authorsused a fitness cost function based on different coupling mea-sures. Number of attributes and methods necessary for thestubbing procedure are considered besides the dependencyfactor. Briand et al. [5] conducted an experiment with a setof real programs and studied four different fitness functions:(1) number of dependencies; (2) method coupling; (3) attri-butes coupling; and (4) an aggregation of attribute and methodcoupling given by a geometric average. For each function,the GA was executed ten times. Thus, they have generated40 solutions for each of the mentioned system.

The solutions reported by Briand et al. [5] are presented inTable 1. Considering the ATM system, we can observe thatthe single objective GA (SOGA) approach with the aggrega-tion fitness function presented the best results, i.e., solutionswith the lowest number of attributes and methods (solutionwith 39 attributes and 13 methods, highlighted entries in thetable). This kind of evaluation function is an alternative todeal with a problem that is in fact multi-objective. However,it presents some restrictions related to its use in practice.The main difficulty is the determination of the appropriateweights. Briand et al. [5,6] suggest a set of steps to selectthe best weights to be used. These steps involve subjec-tive aspects, such as the choice of minimal values for themeasures.

In complex cases, when a large number of measures shouldbe analyzed or a large number of dependencies exist, theapplication of the proposed procedure can be very difficult,since tests with the weights are required, and the best weightscould never be reached. According to multi-objective resear-ches [9], the linear weights combination technique is a popu-lar scalarizing approach probably due to its simplicity.However, this approach usually does not find all Pareto front

123

464 S. R. Vergilio et al.

Table 1 Solutions found by SOGA approach

Functions 1 2 3 4 5 6 7 8 9 10

ATM system

Dependencies (52,19) (54,19) (67,13) (67,19) (52,19) (46,19) (46,19) (45,19) (59,19) (47,13)

Attributes (39,13) (39,19) (39,13) (39,19) (39,19) (39,19) (39,13) (39,13) (39,19) (39,13)

Methods (39,13) (67,13) (59,13) (67,13) (46,13) (46,13) (60,13) (61,13) (39,13) (67,13)

Average (39,13) (39,13) (39,13) (39,13) (39,13) (39,13) (39,13) (39,13) (39,13) (39,13)

ANT system

Dependencies (187,26) (187,26) (157,26) (187,26) (187,26) (213,22) (187,26) (157,26) (157,26) (157,26)

Attributes (131,33) (131,33) (131,33) (131,33) (131,33) (131,33) (131,33) (131,33) (131,33) (131,33)

Methods (178,19) (184,19) (184,19) (197,22) (227,22) (227,22) (197,22) (229,22) (226,22) (197,22)

Average (136,29) (136,29) (136,29) (136,29) (136,29) (136,29) (136,29) (136,29) (136,29) (136,29)

BCEL system

Dependencies (128,73) (128,70) (125,72) (125,75) (128,73) (125,72) (127,73) (127,73) (125,72) (125,72)

Attributes (47,86) (47,87) (47,85) (46,84) (46,85) (46,84) (46,76) (46,83) (46,85) (46,84)

Methods (131,67) (125,70) (131,70) (131,70) (131,67) (134,69) (133,69) (134,69) (138,70)

Average (56,70) (55,72) (58,72) (48,73) (54,73) (53,73) (59,73) (47,74) (59,73) (50,74)

DNS system

Dependencies (22,11) (28,11) (28,11) (19,11) (22,11) (19,11) (22,11) (28,11) (28,11) (19,11)

Attributes (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11)

Methods (28,11) (28,11) (19,11) (19,11) (19,11) (19,11) (22,11) (28,11) (28,11) (22,11)

Average (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11) (19,11)

SPM system

Dependencies (149,28) (149,28) (149,28) (149,28) (146,27) (146,27) (149,28) (149,28) (149,28) (149,28)

Attributes (146,27) (146,27) (146,27) (146,27) (146,27) (146,27) (146,27) (146,27) (146,27) (146,27)

Methods (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26)

Average (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26) (148,26)

(formed by non-dominated solutions) points of interest.Therefore, we introduce in the next section a multi-objectiveapproach, which obtains a set of good solutions achievingbalanced compromises between the measures. Adjustmentsin the evaluation function are not necessary.

3 A multi-objective approach

As mentioned before, the CITO problem is a constrainedmulti-objective optimization problem and may involve anumber of different objectives; different coupling measurescan be used associated to diverse factors that may influencethe stubbing process. To better deal with this problem, amulti-objective approach, which is introduced in this section,is more suitable. This approach includes the representationfor the solutions, objectives to be evaluated, and MOA, whichare described next.

3.1 Multi-objective optimization

Optimization problems including two or more objective func-tions are called multi-objective. Such problems do not have a

single solution because the objectives are usually in conflict.In this way, the goal is to find solutions that represent a goodtrade-off among them.

The general multi-objective maximization1 problem (withno restrictions) can be stated as to maximize Eq. 1.

−→f (

−→x ) = ( f1(−→x ), . . . , fk(

−→x )) (1)

subjected to −→x ∈ �, where: −→x is a vector of decision vari-ables, � is a finite set of feasible solutions, and k is thenumber of objectives.

Let −→x ∈ � and −→y ∈ � be two solutions. For a maximi-zation problem, the solution −→x dominates −→y if:

∀ fi ∈ −→f , i = 1 . . . Q, fi (

−→x ) ≥ fi (−→y ), and

∃ fi ∈ −→f , fi (

−→x ) > fi (−→y )

−→x is a non-dominated solution if there is no solution −→y thatdominates −→x .

The goal is to discover solutions that are non-dominatedby any other in the objective space. A set of non-dominated

1 Similarly to a minimization problem

123

MOA applied to the CITO problem 465

objective vectors is called Pareto optimal and the set of allnon-dominated vectors is called Pareto front.

The Pareto optimal set is helpful for real problems, forexample, engineering problems. It provides valuable infor-mation about the underlying problem [20], i.e., the analysisof such set can show some characteristics of the good solu-tions for the problem. In most applications, the search for thePareto optimal is NP-hard [20], then the optimization prob-lem focuses on finding an approximation set (PFApprox), asclose as possible to the Pareto optimal.

The main weakness of Pareto approach is the algorithmto check for non-dominance solutions. There is no efficientalgorithm to check for non-dominance in a set of feasiblesolutions (the conventional process is O(k M2) for each gen-eration, where k is the number of objectives and M is thepopulation size). Moreover, a mechanism to ensure diversityon the set of solutions is needed. Other possible disadvantageis related to the practical use of the solutions. The numberof solutions found can be very large, and depending on thecase, to select the desired solution may consume some effort.Despite such disadvantages, Pareto technique remains themost popular selection scheme adopted by multi-objectivealgorithms, because of the several advantages that it providesover (linear) aggregating functions [9].

3.2 Representing the CITO problem as multi-objective

As we mentioned before, the CITO problem is defined asan NP-hard problem and there is no algorithm that solves itin polynomial time. In addition to this, it is multi-objective.Many possible factors may influence the stubbing process:fault-proneness, contractual aspects, constraints associatedwith development process, etc. For example, the modular-ity and the way the software is structured can influence thecost. If a distributed development is adopted, a set of classesmay be required to be tested and integrated together. Some ofthese factors are associated with different measures that canserve as objectives for the algorithms. Modularity is associ-ated with coupling and cohesion measures that can be quan-tified by the number of attributes of a class, number of callsor distinct methods invoked, etc. To emulate the behavior ofa class (stub) with a large number of methods (50) (or attri-butes), in general, is more labor intensive and costly than toemulate the behavior of two or three classes, each one withtwo or three methods.

To deal properly with all these factors, in this section wepresent how the CITO problem can be represented and solvedby MOA. For this, two main points are discussed: the repre-sentation for the individuals in the population to be evolved,and how they are evaluated according to the objectives.

The individuals are the problem solutions. For the CITOproblem, the chosen representation is simple. The solution is

represented by a vector whose positions assume an integernumber in the interval [1, N ], N being the number of classesand each class being represented by a number. For instance,a valid solution for a problem with N = 5 classes is theorder o = (2, 1, 3, 5, 4). In this example, the first class tobe tested and integrated would be the class represented bynumber “2”. A solution for the CITO problem is a permu-tation of the classes where all classes must occur once, andonly once in the test order. The number of different solutionsis (N − 1)!/2.

To evaluate the solutions, we use two functions based onthe same coupling measures used in the works of Briand et al.[5,6]. This allows comparison with the related work basedon a simple GA (SOGA). The stubbing cost of an order o isbased on its attribute and method coupling. Two complexitymeasures are then calculated in the following way:

– ACplx(o) (attribute complexity): The attribute complex-ity counts the maximum number of attributes that wouldhave to be available in the stub created if the dependenciescorresponding to the order o were broken (the number ofattributes locally declared in the target class when refer-ences/pointers to the target class appear in the argumentlist of some methods in the source class). A matrix A isused to compute the attribute complexity. In this matrix,rows and columns are classes, and the element A(i, j)corresponds to the number of attributes required to breakthe corresponding dependency, where i depends on j .Then, for a given test order o, the attribute complexityACplx is calculated according to Eq. 2.

ACplx(o) =N∑

i=1

N∑

j=1

A(i, j); j �= l (2)

where N is the total number of classes and l is any classincluded before class i , in test order o.

For example, considering again the order o= (2, 1, 3, 5,4)

and the elements A(3, 1) = 2 and A(3, 4) = 1. This meansthat the class represented by number 3 depends on classes1 and 4. Since class 1 is before class 3 in o, the value 2 isnot counted, because class 1 is available. On the other hand,the number of attributes of class 4 is counted, since class 4is after class 3 in o and a stub for class 4 is required.

– MCplx(o) (method complexity): The method complex-ity counts the number of methods that would have to beemulated in the stub if the dependencies corresponding tothe order o were broken (the number of methods locallydeclared in the target class which are invoked by thesource class methods). The method complexity is com-puted, similarly to ACplx, using a matrix M(i, j). Then,

123

466 S. R. Vergilio et al.

for a given test order, the method complexity MCplx iscomputed as defined by Eq. 3.

MCplx(o) =N∑

i=1

N∑

j=1

M(i, j); j �= l (3)

where N is the total number of classes and l is any classincluded before class i , in test order o.

In this work, following Briand et al.’s work [5], inher-itance, and composition dependencies are not broken. Thismeans that the base/container classes must precede child/con-tained classes in any test order o. The dependencies that can-not be broken are inputs for the algorithm, provided by aprecedence table. To evaluate our approach (Sect. 4), we usethe matrices and precedence tables provided in [5].

Based on the measures and constraints presented above,the problem is the search for an order that minimizes twoobjectives: the method and attribute complexities.

3.3 The PACO algorithm

The ACO algorithm was introduced in [12]. ACO is inspiredby the behavior of real ant colonies, in particular, by their for-aging behavior and ability to find the shortest paths from theirnest to food locations. When searching for food, ants initiallyexplore the area surrounding their nest in a random manner.While moving, ants leave a chemical pheromone trail on theground and other ants can smell pheromone. When choosingtheir way, they tend to choose, in probability, paths markedby strong pheromone concentrations. As soon as an ant findsa food source, it evaluates the quantity and the quality ofthe food and carries some of it back to the nest. During thereturn, the quantity of pheromone that an ant leaves on theground depends on the quantity and quality of the food. Thepheromone trails will guide other ants to the food source [4].

Given the CITO problem, first, the available classes arelisted. Second, the pheromone values (or the pheromonemodel) need to be defined. The pheromone model Γ is themain component of the ACO metaheuristic. The pheromonevalues τi j ∈ Γ are associated to links between the classes.A link (i, j) represents that in the test order, the class j isafter i . Thus, the pheromone values are kept on a pheromonematrix. The pheromone model is used to probabilisticallygenerate test orders for the CITO problem by assembling theavailable classes. The ACO is an iterative algorithm with twomain steps:

– test orders are built by using pheromone model;– the test orders are used to modify the pheromone values

to bias future selections toward high quality test orders.

The pheromone model is also called transition rule andconsiders not only the pheromone values, but also a heuris-tic value η(i, j) related to the quality of the partial test order(ending with class i) built by adding class j . Here, we usedthe information of both matrices A( j, i) and M( j, i) (analo-gously to traveling salesman problem that uses the distancebetween i and j); see Eq. 4.

η(i, j) = 1

A( j, i) × M( j, i)(4)

Moreover, in this work, we have implemented PACO,the Pareto ant colony algorithm, which is based on the antcolony algorithm, and was originally proposed to solve themulti-objective portfolio selection problem [11]. It workswith k pheromone matrices, where k is the number of objec-tives. Each pheromone matrix will lead to the better choiceobserving the objective function. The pheromone matricesare initialized with low values near 0 and these values willbe updated by the best ant (or the test order). The transitionrule used to choose the next class j to be included in a testorder o is given by Eq. 5

p( j) =

⎧⎪⎪⎪⎨

⎪⎪⎪⎩

[∑kh=1 ph · τ h

i j

]α · ηβi j

∑l∈U

[∑kh=1 ph · τ h

il

]α if j ∈ U

0 otherwise

(5)

where the pheromone matrices are represented by τ h and Uis the set of available class.

This rule is a straightforward extension for multiple pher-omone matrices of the rule used in the ant colony system[12,26]. The parameters α and β determine the influence ofthe pheromone and heuristic information on the choice ofthe next added class; ph ∈ [0, 1] are weights, which are uni-formly distributed such that

∑kh=1 = 1. At each iteration,

a weight vector is assigned to each ant, one weight for eachpheromone matrix. Thus, each ant prioritizes the differentobjectives, methods or attributes complexities depending onthese weights.

The transition rule is used according to Eq. 6, where theparameter q is a random number uniformly distributed inthe interval [0, 1], and q0 ∈ [0, 1] is a parameter that definesthe intensification (the transition rule is not used) anddiversification (the transition rule is used) properties of thealgorithm. This type of construction is called pseudo-randomproportional [4].

j =⎧⎨

⎩argmax j∈U

[∑kh=1 ph · τ h

i j

]α · ηβi j if q ≤ q0

p( j) otherwise(6)

where p( j) is given by the probability, represented in Eq. 5.The local pheromone update for all matrices is performed

whenever an ant chooses a sequence of classes (i, j) and uses

123

MOA applied to the CITO problem 467

the following rule: τ hi j = (1 − ρ) · τ h

i j +ρ · τ0, where ρ is therate of evaporation and τ0 is the initial pheromone value.

After all ants of the population have built a test order, theglobal pheromone is updated. For each objective, the bestand second-best solutions are determined and then the fol-lowing rule is used: τ h

i j = (1 − ρ) · τ hi j + ρ · τ h

i j , where

τ hi j receives as value: 15 if the component (i, j) belongs to

the best and second best solutions; 10 if (i, j) belongs onlyto the best path; 5 if it belongs just to the second best path;and 0 otherwise. These values were suggested by [11].

Algorithm 1: Pseudocode of PACO/* Let N be the number of classes */1/* Let max_i terations the number of iterations and Ants2the number of ants */F1, F2 are the objective functions3Initialize pheromone (F1, F2, τ0)4counter = 0; Initialize number of generations5while counter ≤ max_iterations do6

forall ant in Ants do7p1 = rand (0, 1);8p2 = 1 − p1;9BuildPath s = (s, q, q0, p1, p2, F1, F2);10LocalPheromoneUpdate (s, F1, F2);11s1 = LocalSearch (s, F1);12s2 = LocalSearch (s, F2);13/* the ant has two associated solutions, s1 and s2 */14

end15forall objective k do16

Determine the best b and the second-best b′;17

GlobalPheromoneUpdate(b, b′, Fk

);18

end19ParetoSetUpdate (P, s1, s2);20counter = +;21

end22

Algorithm 1 describes the main steps of the PACO algo-rithm. First, an initial procedure is executed where the phero-mone matrices are set to τ0. The next step is an iterative loopin which each ant built a set, s, i.e., a test order solution.

The BuildPath procedure is used to build a test order,line 10. The ant chooses, based on a probabilistic decision(Equation 6), a class that has not yet been included in the testorder s. The ant uses a candidate list composed of classes,which do not have any precedence constraint, i.e., either theclass does not have any precedence by itself or all its prece-dence classes had already been placed on the test order vector(precedence table).

When the ant obtains a test order vector, a local pheromoneupdate is performed, LocalPheromoeeUpdate, line 11. Afterthat, for each objective, a local search is performed on eachant (LocalSearch), lines 12 and 13. Finally, all test orders(s1,s2) of the ants are evaluated and a global pheromoneupdate is performed (GlobalPheromoneUpdate) based on thebest solutions of the iteration, lines 16–19. The archive of the

best solutions found so far, the approximation of the Paretofront, is also updated (ParetoSetUpdate), line 20.At the end of the iterative loop, the solutions presented inthe archive are the solutions obtained by the algorithm.

The local search ( fi ,s) starts from a test order s and theniteratively moves to a neighbor test order s′. The neighborstructure is generated by swapping two classes of the solu-tion. Each time the classes are chosen at random, the algo-rithm uses the precedence table to verify if the swap betweenthe classes can be made. Otherwise, two other classes areselected. If the fi of s′ is better, the movement is accepted,s = s′, and the algorithm moves to another test order. Other-wise, the local search stops and returns s.

3.4 The MTabu algorithm

The Tabu search approach was proposed by Glover [15] toallow local search to avoid local optima. The basic princi-ple of Tabu search is to pursue the search whenever a localoptimum is encountered by allowing non-improving states.Cycling back to previously visited solutions is prevented bythe use of a memory, called Tabu list, which keeps the recenthistory of the search.

In this work, the MTabu algorithm (multi-objective opti-mization using Tabu search) [2], a multi-objective version ofthe Tabu algorithm has been implemented. This algorithmsearches over a neighborhood using the Pareto dominanceconcept.

The idea of applying Tabu search to multiple objectiveoptimization is inspired from its solution structure, in whichTabu search works with more than one solution (neighbor-hood solutions) at a time. This situation creates the oppor-tunity to evaluate multiple objectives simultaneously in onerun. MTabu algorithm has two more lists in addition to theTabu list.

– candidateList receives all the non-dominated neighborsof the current solution; these solutions are used as seedsto diversify the search.

– paretoList contains already explored non-dominatedsolutions.

– T abuList keeps all explored solutions. This list is usedto diversify the search, since it redirects the search towardregions that have been under-explored so far.

The MTabu algorithm (see Algorithm 3) begins with theprocedure SeedSolution to generate current Solution (a ran-dom solution), line 6. Its neighborhood is generated and keptin candidateList , lines 8–14. After that, the paretoListand tabuList are updated with the current Solution (forparetoList , if the current solution is a non-dominated solu-tion, the dominated solutions are eliminated from the list),lines 15 and 16.

123

468 S. R. Vergilio et al.

Finally, in the inner loop, a new current Solution fromcandidateList is randomly chosen to continue; line 17.This inner loop is repeated a number of times dependingon the maximum number of iterations (max_i terations).An external loop controls the number of different generatedsolutions (maxseed) to be explored.

Algorithm 2: Pseudocode of MTabu/* Let max_i terations the number of iterations and1neighborhoodSize the number of neighbor */;Initialize numberseed , candidateList , paretoList , tabuList ;2while numberseed < maxseed do3

Initialize the number of generations;4counter = 0;5current Solution = SeedSolution;6while counter < max_iterations do7

for i = 1 to neighborhood Size do8candidate = neighbor(current Solution);9if (not contains(tabuList , candidate)) then10

Evaluate(candidate);11Update(candidateList , candidate);12

end13end14Update(tabuList , current Solution);15Update(paretoList , current Solution);16current Solution = Select(candidateList);17counter = + ;18

end19numberseed = + ;20

end21

The neighbor structure is generated by the application ofthe following operator: select two classes of the solution andtheir positions are switched. Depending on the size of theneighborhood (neighborhood Size), this operator is appliedmany times. Each time, the classes are chosen at random andthe algorithm uses the precedence table to verify whether theswap between the classes can be made (inheritance and com-position dependencies are not broken). Otherwise, two otherclasses are selected.

3.5 The NSGA-II algorithm

The NSGA-II is an extended genetic algorithm (GA) for find-ing multiple Pareto optimal solutions in a multi-objectiveoptimization problem [10]. This algorithm has the followingfeatures: (a) it uses an elitist principle; (b) it uses an explicitdiversity preserving mechanism; and, (c) it emphasizes thenon-dominated solutions.

Algorithm 3 shows the pseudocode. At the beginning,NSGA-II generates a random population P0, with |P0| = P ,line 3. By applying the usual genetic operators (tournamentselection, crossover (pc), and mutation (pm)), the first childpopulation Q0 (|Q0| = P) is generated, lines 5–8. Both P0

and Q0 are joined to form R0 (|R0| = 2P). In all the follow-

ing n = 1, 2, 3, . . . , max_i terations, NSGA-II deals with apopulation Rn ; line 10. Rn is sorted by non-dominance, lines11–17. It means that, initially, all non-dominated solutionsare put in the front F0. Then, points of F0 are discarded fromthe global set of solutions (only for the sort procedure), andthe new set of non-dominated values form the front F1, andso on.

To form the population, Pn+1, the procedure Calculates-CrowdingDistances for the point of Fj (solutions of thefront j) is used. The NSGA-II estimates the density of solu-tions surrounding a particular solution in the population bycomputing the average distance of two points on either sideof this point along each of the problem objectives. This pro-cedure returns for each solution the crowding distance.

During selection, the NSGA-II uses a crowded-comparisonoperator, which takes into consideration both the non-domination rank of an individual in the population and itscrowding distance (i.e., non-dominated solutions are pre-ferred over dominated solutions, but between two solutionswith the same non-domination rank, the one that resides inthe less crowded region is preferred). Then, only the best Psolutions of Pn+1 remain in the population, line 18. Finally,the genetic operators are applied in Pn+1 to form the newchild population Qn+1.

Algorithm 3: Pseudo-code of NSGA-II/* Let max_i terations the number of iterations */;1P0 = Q0 = ∅;2Initialize populations P0;3counter = 0 ; Initialize the number of generations;4Tournament Selection;5Crossover;6Mutation;7Generate population Q0;8while counter < max_iterations do9

Rn = Pn ∪ Qn ;10Sort Rn by non-dominance;11Pn+1 = ∅;12while |Pn+1| ≤ P do13

CalculateCrowdingDistancesFj ;14Pn+1 = Pn+1 ∪ Fj ;15

end16Sort Pn+1 using the ≥n operator;17Pn+1 = Pn+1[0 : (P − 1)]; Choose P first;18Crossover;19Mutation;20Generate population Qn+1;21counter = + ;22

end23

Observe that the algorithm combines the parents and off-spring, and then classifies the individuals into different layersor fronts in base of the non-domination relation. Since indi-viduals in the first front have the maximum fitness value,they always get more copies than the rest of the population.

123

MOA applied to the CITO problem 469

This characterizes an elitism strategy. Moreover, it uses thecrowding distance in its selection operator to keep a diversefront by making sure each member stays a crowding distanceapart. This is an explicit diversity preserving mechanism.

The crossover operator used is the one point crossoveroperator. As explained before (Sect. 3.2), all classes mustoccur once, and only once in the test order, i.e., individualscannot have repeated values. Due to this restriction, a littlechange was made in the original one point crossover oper-ator. The algorithm that makes the crossover between twoindividuals i1 and i2 works as follows:

1. A random position p of the vector is chosen.2. Exchange all i initial positions of i1 and i2.3. In case of having a repeated class either in i1 or i2, the

class at the lower position is substituted by a missingclass.

This algorithm always looks to the precedence table in orderto preserve the validity of the test order vector, provided thatthey do not violate dependency relations and break inher-itance and compositions. Beyond crossover, the mutationoperator was also modified. The mutation works by swap-ping two classes in the test order vector. Again, the algo-rithm looks to the precedence table to maintain the solutionvalidity.

4 Experimental evaluation

This section describes the experiment conducted to validatethe use of the multi-objective approach to the CITO prob-lem in real systems, and the comparison with the SOGAapproach. We first describe the used systems and methodol-ogy adopted. At the end of this section, the results are pre-sented and analysed.

4.1 Used systems

To allow comparison with the SOGA approach of Briandet al., we used the same matrices of dependencies andprecedence tables presented in [5]. The data are from fivereal systems: ATM, ANT, SPM, BCEL and DNS. Accord-ing to [5], these systems are deemed to be of sufficient sizeand of varying complexity. This allows a better evaluationconsidering different characteristics of the systems. We canobserve in Table 2 that the systems ATM, ANT, SPM, andBCEL have class diagrams of reasonable (and comparable)sizes (between 19 and 45), but with very different numbersof cycles (from 16 for DNS to 416,091 for BCEL). On theother hand, the DNS system has the greatest number of clas-ses and almost the same number of relationships of BCELsystem, but the smallest number of cycles. The table also

Table 2 Detailed information about the systems

System ATM ANT BCEL DNS SPM

Classes 21 25 45 61 19

Uses 39 54 18 211 24

Associations 9 16 226 23 34

Compositions 15 2 4 12 10

Inheritance 4 11 46 30 4

Cycles 30 654 416,091 16 1,178

LOC 1,390 4,093 3,033 6,710 1,198

Table 3 Parameters of the used algorithms

System ATM ANT BCEL DNS SPM

PACO

Ants 20 20 40 60 20

max_i terations 40 40 80 80 40

q0 0.4 0.4 0.4 0.4 0.4

ρ 0.5 0.5 0.5 0.5 0.5

α 1 1 1 1 1

β 1 1 1 1 1

t0 10−4 10−4 10−4 10−4 10−4

MTabu

maxseed 5 5 5 5 5

max_i terations 500 500 500 500 500

tabuList Si ze 10 20 20 10 25

neighborhood Size 10 10 10 10 10

NSGA-II

Population size 200 200 300 200 200

max_i terations 100 100 150 100 100

pc (%) 60 60 60 60 60

pm (%) 10 10 10 10 10

presents the number and kinds of dependency relations: use,association (including simple association and aggregation ofclasses), composition, and inheritance.

4.2 Multi-objective algorithms

Differently from our previous work [8] that explored the useof PACO to solve the CITO problem, in this paper the multi-objective approach is evaluated with two other algorithms:MTabu and NSGA-II (all of them described in Sect. 3). More-over, the adopted methodology is different. The three algo-rithms, similarly to the SOGA algorithm as described in [5],were run ten times. As a consequence, the results hereinpresented differ from the ones found in [8], since the algo-rithms are non-deterministic.

The algorithms parameters were defined empirically basedon initial experiments. In these experiments, we run the

123

470 S. R. Vergilio et al.

Table 4 Solutions found byPACO

ATM system ANT system

1 (39,13) 1 (185, 20) (136,29) (157,26) (183,22) (214,190) (131,33) (162,25)

2 (39,13) 2 (157, 26) (136,29) (208,19) (131,33) (185,20) (183,22) (164,23)

3 (39,13) 3 (157, 26) (136,29) (178,19) (170,23)

4 (39,13) 4 (136,29) (157,26) (131,33) (164,23) (159,25) (183,22) (184,19)

5 (39,13) 5 (157,26) (136,29) (131,33) (183,21) (184,19) (164,23)

6 (39,13) 6 (157,26) (183,22) (136,29) (162,25) (184,19)

7 (39,13) 7 (136,29) (157,26) (184,19) (162,25) (183,22)

8 (39,13) 8 (157,26) (136,29) (170,25) (185,20) (183,22)

9 (39,13) 9 (157,26) (136,29) (214,19) (183,22) (170,25) (185,20)

10 (39,13) 10 (157,26) (136,29) (185,20) (162,25) (183,22) (131,33) (214,19) (164,23)

BCEL system

1 (61,69) (45,76) (131,66) (129,68) (130,67) (53,75) (54,70)

2 (133,66) (51,73) (54,68) (45,76) (124,67)

3 (81,69) (49,75) (45,76) (52,73) (54,72) (50,74) (126,68) (128,66)

4 (116,69) (63,70) (45,76) (53,75) (130,66) (129,68) (59,71) (56,73)

5 (131,66) (53,73) (45,76) (56,68) (55,71)

6 (45,77) (56,70) (50,75) (130,66) (129,67) (49,76) (57,68)

7 (109,68) (128,67) (55,69) (53,71) (45,76) (130,66)

8 (45,77) (57,70) (54,71) (84,69) (103,68) (46,75) (128,66)

9 (45,77) (110,68) (55,70) (130,66) (48,76)

10 (133,66) (45,77) (102,70) (122,67) (46,75) (51,73) (108,69) (57,71)

DNS system SPM system

1 (19,11) 1 (146,27) (148,26)

2 (19,11) 2 (146,27) (148,26)

3 (19,11) 3 (148,26) (146,27)

4 (19,11) 4 (146,27) (148,26)

5 (19,11) 5 (148,26) (146,27)

6 (19,11) 6 (146,27) (148,26)

7 (19,11) 7 (146,27) (148,26)

8 (19,11) 8 (148,26) (146,27)

9 (19,11) 9 (146,27) (148,26)

10 (19,11) 10 (148,26) (146,27)

different algorithms and the obtained solutions were exam-ined in terms of the non-domination relation. The set ofparameters that obtained results that dominate solutions ofother sets is considered the best for the algorithm. The param-eters are in Table 3. For NSGA-II, we selected a mutation rateof 0.10, consistent with the previous related work [5]. We alsotried 0.15, but 0.10 had a slightly better performance. Regard-ing the crossover rate, we tried both 0.5 and 0.6, noticingthat 0.6 had better results. The work [5] used a population of100 chromosomes, and 500 as the maximum number of iter-ations. We tried with different population sizes of 100, 200,and 300 and observed the convergence. The parameters ofPACO were set using the work [11] as reference. Differentexperiments were made to set the number of ants (5, 10, 20,

30, 40, 60). The maximum number of iterations was alwaysset observing the convergence. For MTabu, the parametermaxseed was set using [2] as reference. Different experi-ments were made to set the size of Tabu list and neighborhood(10, 20, 25). When the same results were found, the smallersizes were selected. The maximum number of iterations wasset observing the convergence.

4.3 Results

The main results of the algorithms PACO, MTabu and NSGA-II are presented respectively in Tables 4, 5 and 6. In thesetables, the pairs represent attribute and method complexitiesof the solutions. Each row corresponds to an execution of the

123

MOA applied to the CITO problem 471

Table 5 Solutions found byMTabu

ATM system ANT system

1 (39,13) 1 (136,29) (157,22) (157,22) (178,19) (131,33) (136,29) (136,29) (178,19) (157,22)

2 (39,13) 2 (136,29) (157,22) (157,22) (178,19) (131, 33)

3 (39,13) 3 (136, 29) (157,22) (157,22) (131,33) (178,19)

4 (39,13) 4 (136,29) (136,29) (178,19) (178,19) (157,22) (131,33)

5 (39,13) 5 (136,29) (178,19) (178,19) (157,22) (131,33)

6 (39,13) 6 (136,29) (187,21) (157,22) (131,33)

7 (39,13) 7 (136,29) (178,19) (178,19) (157,22) (131,33)

8 (39,13) 8 (136,29) (136,29) (178,19) (157,22) (131,33)

9 (39,13) 9 (136,29) (157,22) (157,22) (178,19) (131,33)

10 (39,13) 10 (136,29) (178,19) (157,22) (131,33)

BCEL system DNS system

1 (49,74) (71,73) (45,76) (46,75) 1 (22,11) (21,13)

2 (60,83) (55,85) (54,88) (59,84) 2 (19,11)

3 (46,74) (50,72) (51,71) (49,73) (83,70) 3 (19,11)

4 (48,73) (48,73) 4 (19,11)

5 (134,70) (132,71) (45,76) (72,72) (46,73) 5 (22,11) (21,13)

6 (45,76) (66,74) (66,74) (46,75) (123,73) 6 (19,11)

7 (132,72) (45,77) (80,73) (46,74) 7 (19,11)

8 (46,73) (129,70) (51,71) (130,68) (50,72) 8 (22,11) (21,13)

9 (50,77) (48,81) (49,78) 9 (19,11)

10 (125,71) (48,75) (126,70) (47,78) (53,74) 10 (19,11)

SPM system

1 (146,27) (148,26) (146,27) (148,26)

2 (152,26) (152,26)

3 (151,27) (151,27)

4 (146,27) (148,26) (146,27)

5 (148,26) (146,27)

6 (146,27) (148,26)

7 (149,28) (152,31)

8 (148,26) (146,27)

9 (146,27) (148,26) (148,26)

10 (167,31) (165,31) (168,29)

algorithm (out of 10). The boldface solutions belong to theapproximation of the Pareto front (PFApprox), which is com-posed of the non-dominated solutions considering all runs.Observe that the solutions obtained in one execution are non-dominated for this execution; however, considering anotherexecution those solutions obtained before can be dominated.

4.4 Discussion

By analyzing Tables 4, 5 and 6, it is possible to note diverse(boldface) solutions that have in common a good trade-offbetween the two complexity metrics. For each execution,these solutions are non-dominated; hence, no one can be

considered to be better than any other with respect to bothcomplexity metrics. Table 7 presents the complexities of thesolutions in the approximation of the Pareto front (PFap-prox) obtained after analyzing all the returned solutions ofall algorithms executions. Next, this front is used to comparethe algorithms.

For ATM system, all the MOA found solutions with attri-bute complexity of 39 and method complexity of 13. Thesevalues are the best approximation of the Pareto front. For theSOGA algorithm, the same solutions are found when the usedfunction is the average of the attribute and method complex-ities (see Table 1, boldface solutions). On the other hand,when the function uses only one complexity measure, thesolutions are only good with respect to the metric employed.

123

472 S. R. Vergilio et al.

Table 6 Solutions found byNSGA-II

ATM system ANT system

1 (39,13) 1 (178,19) (131,33) (157,22) (136,29)

2 (39,13) 2 (178,19) (131,33) (157,22) (136,29)

3 (39,13) 3 (178,19) (131,33) (157,22) (136,29)

4 (39,13) 4 (178,19) (131,33) (157,22) (136,29)

5 (39,13) 5 (178,19) (131,33) (157,22) (136,29)

6 (39,13) 6 (178,19) (131,33) (157,22) (136,29)

7 (39,13) 7 (178,19) (131,33) (157,22) (136,29)

8 (39,13) 8 (178,19) (131,33) (157,22) (136,29)

9 (39,13) 9 (178,19) (131,33) (157,22) (136,29)

10 (39,13) 10 (178,19) (131,33) (157,22) (136,29)

BCEL system

1 (45,76) (46,74) (49,72) (51,71) (53,68) (52,69)

2 (123,67) (45,77) (53,68) (49,72) (48,75) (52,69) (51,70) (50,71)

3 (45,76) (50,74) (51,72) (128,66) (126,67) (53,68)

4 (45,76) (130,66) (53,68) (48,74) (50,71) (49,73) (46,75) (51,70) (52,69)

5 (45,76) (46,72) (50,71)

6 (130,66) (46,79) (128,67) (53,68) (47,77) (50,71) (48,75) (49,73) (52,69) (51,70)

7 (45,76) (46,74) (49,73) (53,68) (51,70) (50,71) (52,69)

8 (130,66) (45,80) (129,67) (53,68) (50,71) (49,74) (48,75) (46,79) (47,77) (51,70) (52,69)

9 (45,77) (128,66) (53,68) (46,73) (49,72) (52,69) (51,70) (50,71)

10 (70,71) (45,76) (50,72) (46,73)

DNS system SPM system

1 (19,11) 1 (146,27) (148,26)

2 (19,11) 2 (146,27) (148,26)

3 (19,11) 3 (148,26) (146,27)

4 (19,11) 4 (146,27) (148,26)

5 (19,11) 5 (148,26) (146,27)

6 (19,11) 6 (146,27) (148,26)

7 (19,11) 7 (146,27) (148,26)

8 (19,11) 8 (148,26) (146,27)

9 (19,11) 9 (146,27) (148,26)

10 (19,11) 10 (148,26) (146,27)

In the ANT system, the PFapprox is composed of foursolutions. MTabu and NSGA-II always found these solu-tions; however, PACO algorithm found these solutions onlyin some runs. The SOGA algorithm also found some of thesesolutions when the used function was the average of the attri-bute and method complexities (see Table 1). However, thesolution with complexities (157,22) was not found by SOGA.Again, when the SOGA algorithm uses a function based ononly one complexity measure, the solutions are only goodwith respect to the metric employed, but SOGA has moreproblems to find the best solutions and some runs found sub-optimal solutions.

The BCEL system has the greatest number of solutionsin the approximation of the Pareto front, eight solutions. All

algorithms present difficulties to find the solutions. However,on average, NSGA-II presents the best performance obtain-ing seven solutions of the eight solutions found so far. Notethat there are no SOGA solutions (Table 1) in the approxi-mation of the Pareto front table.

The DNS system has the greatest number of classes, andthe smallest number of cycles. Similarly to the ATM system,the MOA found only one solution with (19,11) as attributeand method complexities, respectively. These values are thebest approximation of the Pareto front. Again, the SOGAalgorithm with the aggregation-based fitness found the samesolution. However, when the fitness uses only one complex-ity measure, the solutions are only good with respect to themeasure used.

123

MOA applied to the CITO problem 473

Table 7 Complexities of the solutions in the PFApprox

System Complexities Cardinality

ATM (39,13) 1

ANT (178,19), (131,33), (157,22),(136,29) 4

BCEL (45,76), (46,72), (50,71), (51,70) 8

(52,69), (53,68), (122,67, (128,66)

DNS (19,11) 1

SPM (148,26), (146,27) 2

Table 8 Mean number of solutions on the PFapprox

Systems ATM ANT BCEL DNS SPM

GA-depend. 0 0 0 0.3 0.2

GA-attrib. 0.5 1 0 1 1

GA-method. 0.1 0.1 0 0.4 1

GA-average. 1 1 0 1 1

PACO 1 1.6 0.9 1 2

MTabu 1 4 0.3 0.7 1.2

NSGA-II 1 4 3.7 1 2

For SPM, the approximation of the Pareto front containstwo non-dominated solutions with (146,27) and (148,26),respectively, for attribute and method complexities. ThePACO, NSGA-II and SOGA algorithms found these solu-tions. But, the SOGA has found one solution when the attri-bute complexity function is used and the other one when themethod complexity function is used. Moreover, these solu-tions were not found when the aggregation function was used.For this system, MTabu does not always find all the solutionsof the PFapprox.

Table 8 summarizes the performance of each multi-objec-tive algorithm considering the PFapprox. Each entry corre-sponds to the mean of number of non-dominated solutionsfound in ten runs. This number is calculated by summingthe number of non-dominated solutions found in each run.The value obtained is divided by ten. Observe that the SOGAalgorithm with the different used fitness functions finds onlya solution by execution; therefore, the maximum value thatcan be obtained for this algorithm is 1.

It is important to remark that SOGA used four differ-ent complexity functions and the solutions were not alwaysfound by the same function. Consequently, some effort mustbe spent to determine the best function for the SOGAapproach.

The aggregation function that uses the average betweenthe complexities measures seems to be more adequate. More-over, it seems that the SOGA approach does not work well forcomplex systems, i.e., systems with a great number of depen-dency cycles, such as BCEL. For such system, the SOGA

Table 9 Number of different solutions on the PFapprox

Algorithm ATM ANT BCEL DNS SPM

PACO 1 3 3 1 2

MTabu 1 4 1 1 2

NSGA-II 1 4 7 1 2

approach does not find any solution in the PFapprox. Thisdoes not happen with the multi-objective approach, indepen-dently of the MOA being considered. Comparing the differ-ent MOA it is possible to observe that NSGA-II reaches thebest results for all systems. This good performance can beclearly observed at BCEL system.

The complexity of each algorithm was not calculated,but in this kind of algorithm, the number of evaluations isaccepted as computational cost [13]. For the NSGA-II algo-rithm, the evaluation of the fitness function is calculatedafter the creation of each new individual. So, the numberof evaluations will be equal to the number of new indi-viduals created in each generation multiplied by the num-ber of iterations. For example, BCEL system, 300 × 150 =45,000 evaluations. For PACO and MTabu, there is no directrelation between the parameters as it depends on the suc-ceed of the local search. But, considering for MTabu andBCEL system, the parameters and a candidateList with10, 5 × 500 × 10 × 10 = 250, 000 evaluations are neces-sary and for PACO and BCEL system, considering for eachant one step of local search (at least) exploring 10 neigh-bors, 40 × 80 × 2 × 10 = 64, 000 evaluations. It is possibleto observe that NSGA-II has less computational cost. How-ever, such comparison factor should be evaluated in futureexperiments.

Another factor that can be considered in the comparisonof the MOA is the number of different solutions in the PFAp-prox found, which is presented in Table 9. Considering thisfact, NSGA-II presents the best results again, followed byPACO and MTabu.

When the multi-objective approach is used, a great vari-ety of good solutions can be used. In the front, there can bedifferent solutions with the same complexity, as observed inthe tables of appendix. If the SOGA approach is used, onlyone solution is provided and a priori effort is necessary toderive the ideal fitness function for a given program.

Those advantages can be illustrated considering the BCELsystem. Here, we compare the solutions of the NSGA-II thatare in PFApprox (see Table 10) and the best SOGA solutiono (reported in [5]). We can observe that o was obtained byusing the number of attributes as fitness function (Table 1).

This solution, in spite of being dominated, is very close(considering the values for both objectives) to solutions A andF obtained by NSGA-II. It is reasonable to argue that the useof MOA does not produce benefit in practice, since solutions

123

474 S. R. Vergilio et al.

Table 10 NSGA-II and SOGA Solutions for BCEL

Solution Complexity Order(A,M)

A (45,76) 35 5 19 15 17 41 32 16 6 30 10 40 8

0 1 26 4 27 18 39 31 33 11 24 36 43

3 28 13 20 38 21 7 34 25 42 29 44 2

37 14 9 12 23 22

B (50,71) 35 5 41 16 40 7 15 8 14 12 0 19 6 30

9 17 32 11 18 1 10 4 27 39 31 24 33

21 36 43 3 13 20 38 26 28 34 25 42

29 44 2 37 23 22

C (51,70) 35 5 19 41 16 40 7 9 8 13 12 14 6 30

15 17 32 11 18 1 4 10 27 43 28 0 39

37 31 33 26 24 38 42 3 21 36 20 34

25 29 44 2 23 22

D (52,69) 35 5 41 16 10 40 7 14 9 8 11 13 12

0 19 15 17 32 18 1 37 4 27 20 39 30

33 21 28 36 31 38 26 42 3 43 24 15

34 29 44 2 6 23 22

E (53,68) 35 5 41 16 40 7 10 15 8 14 11 13 12

0 19 6 30 9 17 32 18 1 4 27 39 31

43 21 42 24 33 38 26 3 20 36 37

28 25 34 29 44 2 22 23

F (46,72) 41 40 0 35 5 15 19 7 30 9 14 12

17 16 32 1 8 6 18 27 13 10 4

21 42 26 36 39 31 11 37 43 28 20

24 33 38 3 25 34 29 44 2 22 23

o (46,76) 36 6 20 31 16 18 17 33 28 2 9 11 39

14 7 1 12 32 38 42 41 19 40 21 22

25 44 5 27 34 43 26 4 37 29 8 13

35 10 30 45 15 23 3 24 46 76 71

(A, F and o) are very close. However, the multi-objectiveapproach provides to the tester different good trade-offs toselect. The tester can take advantage of this, and selects anorder according to his (or her) preferences (needs). He (orshe) can conduct the integration test by prioritizing eitherattribute or method complexities. Moreover, the tester canuse other preference information about the classes, such asconstraint related to contractual aspects to select an orderfrom a range of good solutions. For example, suppose thatdue to a distributed development, the classes 25, 34 29, and44 need to be tested in sequence and at the end of the testactivity. Therefore, solution F should be chosen. Other goodoptions are solutions D and E. However, these alternativesare not provided by SOGA.

As mentioned in Sect. 3, an analysis of the solutions onthe PFAprox can also provide useful information about theproblem. See in Table 10 that in five orders, the first classesto be tested are 35 and 5, and in all NSGA-II orders the last

classes are 23 and 22. Those solutions provide knowledgeabout critical classes, i.e., classes 35 and 5 need to be firstavailable.

5 Conclusions

In this paper, we introduced a new approach to solve theCITO problem based on multi-objective optimization. Theapproach was evaluated by using three different MOA:PACO, MTabu and NSGA-II. The algorithms were adaptedto produce test orders that represent a good trade-off betweenthe number of attributes and methods in the stubbing process.

The algorithms were evaluated in a benchmark of fivereal programs and their results were compared to the SOGAapproach, considering the approximation of the Pareto front.In this evaluation, we observe that the multi-objectiveapproach is very advantageous because different factors canbe considered and a set of good solutions that achieve abalanced compromise between the considered measures isobtained without human intervention.

We can observe that for some systems the attribute andmethod complexity measures are similar, i.e., when the attri-bute complexity grows the method complexity also grows. Insuch cases, the use of the SOGA approach with an aggregatedfunction can be a good alternative. On the other hand, sys-tems like BCEL, which present a great number of dependencycycles, exhibit different behavior for attribute and methodcomplexities. In these cases the MOA are most suitable. Theapproach presented better results than the SOGA based one,independently of the MOA used.

Comparing the used MOA, NSGA-II presented the bestresults for all systems considering the mean number ofsolutions in the approximation to the Pareto front, and thenumber of different non-dominated solutions found.

It is possible to argue that the greater the number of solu-tions found in the approximation of the Pareto front and theirdistributions, the greater are the ways to satisfy the integra-tion test plans considering the real world needs. This factpoints out that the MOA are more suitable to solve softwareengineering complex problems, like CITO.

Future works include performing further experiments withother relevant complexities measures such as number of dif-ferent types of parameters and returned values of the methodsthat need to be emulated in the stub; number of different attri-butes type; etc. The performance of the MOA with more thantwo objectives should be also evaluated in further studies,with other benchmarks.

Acknowledgments The authors received the following financial sup-port: Aurora Pozo (CNPq productivity grant number 303761/2009-1and Universal Project grant number 472510/2010-0); Silvia ReginaVergilio (CNPq productivity grant number 306608/2009-0 and Univer-sal Project grant number 481397/2009-4); Tiago Nobre (CNPq grantnumber 579097/2008-0).

123

MOA applied to the CITO problem 475

References

1. Abdurazik, A., Offutt, J.: Coupling-based class integration and testorder. In: International Workshop on Automation of Software Test.ACM, Shanghai, China, May 2006

2. Baykasoglu, A., Owen, S., Gindy, N.: A taboo search basedapproach to find the pareto optimal set in multiple objective opti-misation. Eng. Optim. 31, 731–748 (2007)

3. Binder, R.V.: Testing object-oriented systems: models, patterns,and tools. Addison-Wesley, Reading (2000)

4. Blum, C.: Ant colony optimization: introduction and recenttrends. Phys. Life Rev. 2(4), 353–373 (2005)

5. Briand, L.C., Feng, J., Labiche, Y.: Experimenting with geneticalgorithms and coupling measures to devise optimal integrationtest orders. Carleton University, Technical report SCE-02-03, Oct2002

6. Briand, L.C., Feng, J., Labiche, Y.: Using genetic algorithms andcoupling measures to devise optimal integration test orders. In: 14thInternational Conference on Software Engineering and KnowledgeEngineering (SEKE), pp. 43–50. Ischia, Italy, July 2002

7. Briand, L.C., Labiche, Y., Wang, Y.: An investigation of graph-based class integration test order strategies. IEEE Trans. Softw.Eng. 29(7), 594–607 (2003)

8. Cabral, R., Pozo, A., Vergilio, S.: A Pareto ant colony algorithmapplied to the class integration and test order problem. In: Pro-ceedings of the 22nd IFIP WG 6.1 International Conference onTesting Software and Systems—ICTSS’10, pp. 16–29. Springer,Berlin (2010)

9. Coello, C.A.C., Lamont, G.B., Veldhuizen, D.A.V.: Evolutionaryalgorithms for solving multi-objective problems (genetic and evo-lutionary computation). Springer, New York (2006)

10. Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.: A fast elitist non-dominated sorting genetic algorithm for multi-objective optimiza-tion: NSGA-II. In: Lecture Notes in Computer Science, pp. 849–858 (2000)

11. Doerner, K., Gutjahr, W.J., Hartl, R.F., Strauss, C., Stummer, C.:Pareto ant colony optimization: a metaheuristic approach to mul-tiobjective portfolio selection. Ann. Oper. Res. 1–4(131), 79–99(2004)

12. Dorigom, M., Socha, K.: An Introduction to Ant Colony Optimiza-tion. Technical Report, TR/IRIDIA/2006-010, Bruxelles, Belgium,Apr 2006

13. García, S., Molina, D., Lozano, M., Herrera, F.: A study on the useof non-parametric tests for analyzing the evolutionary algorithms’behaviour: a case study on the CEC’2005 special session on realparameter optimization. J. Heurist. 15, 617–644 (2009)

14. Gendreau, M., Potvin, J.Y.: Tabu search. In: Gendreau, M., Potvin,J.Y., Hillier, F.S. (eds.) Handbook of Metaheuristics, InternationalSeries in Operations Research and Management Science, vol. 146,pp. 41–59. Springer, USA (2010)

15. Glover, F.: Future paths for integer programming and links to arti-ficial intelligence. Comput. Oper. Res. 13, 533–549 (1986)

16. Harrold, M.J., McGregor, J.D., Fitzpatrick, K.J.: Incrementaltesting of object-oriented class structures. In: 14th InternationalConference on Software Engineering, pp. 68–80. IEEE ComputerSociety, Melbourne, Australia, May 1992

17. Ishibuchi, H., Sakane, Y., Tsukamoto, N., Nojima., Y.: Evolution-ary many-objective optimization by NSGA-II and MOEA/D withlarge populations. In: IEEE International Conference on Systems,Man, and Cybernetics, pp. 1758–1763. IEEE Computer Society,Oct 2009

18. Jaroenpiboonkit, J., Suwannasart, T.: Finding a test order usingobject-oriented slicing technique. In: 14th Asia-Pacific SoftwareEngineering Conference, Washington, DC, USA, pp. 49–56 (2007)

19. Jaroenpiboonkit, J., Suwannasart, T.: Class ordering tool—a toolfor class ordering in integration testing. In: International Confer-ence on Advanced Computer Theory and Engineering, pp. 724–728, Dec 2008

20. Knowles, J., Thiele, L., Zitzler, E.: A Tutorial on the Perfor-mance Assessment of Stochastic Multiobjective Optimizers, vol.214, Computer Engineering and Networks Laboratory (TIK), ETHZurich, Switzerland, Feb 2006

21. Kraft, N.A., Lloyd, E.L., Malloy, B.A., Clarke, P.J.: The implemen-tation of an extensible system for comparison and visualization ofclass ordering methodologies. J. Syst. Softw. 79, 1092–1109 (2006)

22. Kung, D., Gao, J., Hsia, P., Toyoshima, Y., Chen, C.: A test strat-egy for object-oriented programs. In: 19th International ComputerSoftware and Applications Conference. IEEE Computer Society,USA, Aug 1995

23. Mao, C., Lu, Y.: AICTO: an improved algorithm for planning inter-class test order. In: Proceedings of the The Fifth International Con-ference on Computer and Information Technology, pp. 927–931.CIT ’05, IEEE Computer Society, Washington, DC, USA (2005)

24. Melton, H., Tempero, E.: An empirical study of cycles among clas-ses in Java. Empir. Softw. Eng. 12, 389–415 (2007)

25. Pareto, V.: Manuel D’Economie Politique. Ams Press, Paris (1927)26. Pasia, J.M., Hart, R., Doerner, K.F.: Solving a bi-objective flow-

shop scheduling problem by Pareto-ant colony optimization. In:Lecture Notes in Computer Science, pp. 294–305, Aug 2006

27. Pressman, R.: Software Engineering: A Practitioner’s Approach.McGraw-Hill, New York (2006)

28. Tai, K.C., Daniels, F.J.: Test order for inter-class integration testingof object-oriented software. In: 21st International Computer Soft-ware and Applications Conference, pp. 602–607. IEEE ComputerSociety, USA, Aug 1997

29. Traon, Y.L., Jéron, T., Jézéquel, J.M., Morel, P.: Efficient object-oriented integration and regression testing. IEEE Trans. Reliab.49(1), 12–25 (2000)

123