Cross–Project Defect Prediction with Respect to Code Ownership … · the very beginning to which cluster a project belongs. Asubsetofthesamedatasethadal-readybeenanalysedin[16,17].In[16]5industrial

e-Informatica Software Engineering Journal, Volume 9, Issue 1, 2015, pages: 21–35, DOI 10.5277/e-Inf150102

Cross–Project Defect Prediction with Respect to CodeOwnership Model: an Empirical Study

Marian Jureczko∗, Lech Madeyski∗∗∗Institute of Computer Engineering, Control and Robotics, Wroclaw Univeristy of Technology

∗∗Faculty of Computer Science and Management, Wroclaw University of [email protected], [email protected]

AbstractThe paper presents an analysis of 83 versions of industrial, open-source and academic projects. Wehave empirically evaluated whether those project types constitute separate classes of projects withregard to defect prediction. Statistical tests proved that there exist significant differences betweenthe models trained on the aforementioned project classes. This work makes the next step towardscross-project reusability of defect prediction models and facilitates their adoption, which has beenvery limited so far.

Keywords: software metrics, software defect prediction, cross-project prediction

1. Introduction

Assuring software quality is known to requiretime-consuming and expensive development pro-cesses. The costs generated by those processesmay be minimized when the defects are predictedearly on, which is possible by means of defectprediction models [1]. Those models, based onsoftware metrics, have been developed by a num-ber of researchers (see Section 2). The softwaremetrics which describe artifacts of the softwaredevelopment process (e.g. software classes, files)are generally used as the models’ input. Themodel output usually estimates the probabilityof failure, the occurrence of a defect or the ex-pected number of defects. The predictions aremade for a given artifact. The idea of buildingmodels on the basis of experienced facts, calledinductive inference, is discussed in the context ofsoftware engineering by Samuelis [2].

Defect prediction models are extraordinarilyuseful in software testing process. The availableresources are usually limited and, therefore, itmay be difficult to conduct the comprehensive

tests on, and the reviews of, all artifacts. Defectprediction models are extraordinarily useful inthe software testing process. The available re-sources are usually limited and, therefore, it maybe difficult to conduct the comprehensive testson, and the reviews of, all artifacts [1]. The pre-dictions may be used to assign varying prioritiesto different artifacts (e.g. classes) under test [3,4].According to the 80:20 empirical rule, a smallamount of code (often quantified as 20% of thecode) is responsible for the majority of softwaredefects (often quantified as 80% of the knowndefects in the system) [5, 6]. Therefore, it maybe possible to test only a small amount of arti-facts and find a large amount of defects. In short,a well-designed defect prediction model may savea lot of testing efforts without decreasing softwarequality.

The defect prediction models may give sub-stantial benefits, but in order to build a model,a software measurement program must belaunched. According to Kaner and Bond [7], onlyfew companies establish such programs, evenfewer succeed with them and many of the com-

http://www.e-informatyka.pl/wiki/e-Informaticahttp://www.e-informatyka.pl/attach/e-Informatica_-_Volume_9/eInformatica2015Art2.pdf

22 Marian Jureczko, Lech Madeyski

panies use them only to conform to the criterialaid down in the Capability Maturity Model [8].The costs may be one of the reasons behind thelimited adoption of the defect prediction mod-els, e.g. an average overhead for metrics collec-tion was estimated to be 4–8% of the overallvalue [8, 9]. Using cross-project defect predictioncould reduce the expenditure, since one modelmay be used in several software projects, whichmeans that it is not necessary to launch a com-pletely new software measurement program foreach project. The cross–project defect predic-tion is also helpful with solving the problemsconnected with the lack of historical data indis-pensable to train a model [10]. Unfortunately,the body of knowledge of cross–project defectprediction does not support us with the resultsthat are advanced enough to be used (details inthe next section). The intention of this work isto reveal the facts regarding high level predic-tion boundaries, particularly the possibility ofusing prediction models across different projectcode ownership models. The conducted exper-iments aiming at verifying whether cluster ofprojects can be derived from the source codeownership model, where the cluster is a group ofsoftware projects that share a common predic-tion model. Such finding eases the applicationof defect prediction by removing the necessity oftraining the model. It is enough to identify thecluster a given project belongs to and use thecommon model.

This study investigates whether there is a rel-evant difference between industrial, open-sourceand academic software projects with regard todefect prediction. In order to explore that po-tential disparity, the data from 24 versions of 5industrial projects, 42 versions of 13 open-sourceprojects and 17 versions of 17 academic projectswere collected. Several defect prediction modelswere built and the efficiency of the predictionswas compared. Statistical methods were used inorder to decide whether the obtained differenceswere significant or not. It is worth mentioningthat a somehow related question of suitability ofsoftware quality model for projects within differ-ent application domains was raised by Villalbaet al. [11], whereas the results of other works

which investigate the cross-project defect predic-tion (i.e. [12,13]) suggest that the code ownershipmodel might be a relevant factor with regard tothe prediction performance.

The rest of this paper is organized as follows:subsequent sections describe related work, thedesign of empirical evaluation used in this study(including the details of data collection and analy-sis methodology), the descriptive statistics of thecollected data, the results of empirical evaluation,threats to the validity of the empirical study, andconclusions.

2. Related Work

Cross-project reusability of defect predictionmodels would be extremely useful. Such general-ized prediction models would serve as a startingpoint in software development environments thatcannot provide historical data and, as a result,they would facilitate the adoption of defect pre-diction models.

A preliminary work in this area was conductedby Subramanyam and Krishnan [14]. The authorsinvestigated a software project where C++, aswell as Java, were employed and found substantialdifferences between classes written in differentprogramming languages with regard to defectprediction, e.g. the interaction effect (the defectcount grows with the CBO value for C++ classbut decreases for the Java classes; the relationwas calculated with respect to the DIT metric).Hence, the results indicate issues with regard tocross-language predictions. The authors investi-gated only one project, nevertheless, similar dif-ficulties might arise in cross-project predictions.

Nagappan et al. [15] analyzed whether defectpredictors obtained from one project history areapplicable to other projects. It turned out thatthere was no single set of metrics that wouldfit into all five investigated projects. The defectprediction models, however, could be accuratewhen obtained from similar projects (the similar-ity was not precisely defined, though). The studywas extended by Zimmerman et al. [12], whoperformed 622 cross-project predictions for 12real world applications. A project was considered

Cross–Project Defect Prediction with Respect to Code Ownership Model: an Empirical Study 23

as a strong predictor for another project whenall precision, recall, and accuracy were greaterthan 0.75. Only 21 cross-project validations sat-isfied this criterion, which sets the success rateat 3.4%. Subsequently, the guidelines to assessthe cross-project prediction chance of successwere given. The guidelines were summarized ina decision tree. The authors constructed separatetrees for assessing prediction precision, recall, andaccuracy, but only the tree for precision was givenin the paper.

A study of cross-company defect predictionwas conducted by Turhan et al. [10]. The authorsconcluded that there is no single set of staticcode features (metrics) that may serve as defectpredictor for all the software projects. The ef-fectiveness of the defect prediction models wasmeasured using probability of detection (pd) andprobability of false alarm (pf). Cross-companydefect prediction dramatically increased pd aswell as pf. The authors were also able to decreasethe pf by applying the nearest neighbor filtering.The similarity measure was the Euclidean dis-tance between the static code features. However,there was still a drawback for cross-company de-fect prediction models: the project features whichmight influence the effectiveness of cross-companypredictions were not identified.

In an earlier paper [13], we presented an em-pirical study showing that data mining tech-niques may be used to identify project clus-ters with regard to cross-project defect predic-tion. The k-means and Kohonen’s neural net-works were applied to correlation vectors in or-der to identify the clusters. The correlation vec-tors were calculated for each version of eachproject respectively and represented Pearson’scorrelation coefficients between software metricsand numbers of defects. Subsequently, a defectprediction model was created for each identi-fied cluster. In order to validate the existenceof a cluster, the efficiency of the cluster modelwas compared with the efficiency of a generalmodel. The general model was trained usingdata from all the projects. Six different clus-ters were identified and the existence of two ofthem was statistically proven. The clusters char-acteristics were consistent with Zimmerman’s

findings [12] about the factors that are criti-cal in cross-project prediction. In our paper, wemake a step towards simplifying the setup ofdefect prediction in the software developmentprocess. The laborious activities regarding cal-culation of correlation vectors and mining theclusters are not needed as it is obvious fromthe very beginning to which cluster a projectbelongs. A subset of the same data set had al-ready been analysed in [16,17]. In [16] 5 industrialand 11 open–source projects were investigated.The study was focused on the role of the sizefactor in defect prediction. The other paper wasfocused on the cross-project defect prediction.In [17], published in Polish, being a preliminarystudy to this one, we focused on the differencesand similarities between industrial, open–sourceand academic projects, whereas in this paper, weadditionally performed a comprehensive statisti-cal analysis. As a result, new projects may takeadvantage of the prediction models developedfor the aforementioned classes of the existingprojects.

It is also worth mentioning that a different ap-proach, based on the idea of inclusion additionalsoftware projects during the training process, canprovide a cross-project perspective on softwarequality modelling and prediction [18].

A comprehensive study of cross-project defectprediction was conducted by He et al. [19]. Theauthors investigated 10 open source projects tocheck whether training data from other projectscan provide better prediction results than train-ing data from the same project – in the bestcases it was possible. Furthermore, in 18 out of34 cases the authors were able to obtain a Recallgreater than 70% and a Precision greater than50% for the cross-project defect prediction.

3. Empirical Evaluation Design

3.1. Data Collection

The data from 83 versions of 35 projects was col-lected and analyzed. It covers 24 versions of 5 in-dustrial projects, 42 versions of 13 open–sourceprojects and 17 versions of 17 academic projects.


The number of versions is greater than the num-ber of projects because there were projects inwhich data from several versions were collected.For example, in the case of the Apache Antproject (http://ant.apache.org), versions 1.3, 1.4,1.5, 1.6 and 1.7 were analyzed. For each of theanalysed versions there was an external release,which was visible to the customer or user.

Each of the investigated industrial projectsis a custom-built enterprise solution. All of theindustrial projects have already been success-fully developed by different teams of 10 to 40developers and installed in the customer envi-ronments. All of them belong to the insurancedomain but implement different feature sets ontop of Java-based frameworks. Each of the in-dustrial projects were developed by the samevendor.

The following open-source projects were in-vestigated: Apache Ant, Apache Camel, ApacheForrest, Apache Log4j, Apache Lucene, ApachePOI, Apache Synapse, Apache Tomcat, ApacheVelocity, Apache Xalan, Apache Xerces, Ckjmand Pbeans.

The academic projects were developed by thefourth and the fifth-year graduate MSc computerscience students. The students were divided intothe groups of 3, 4 or 5 persons. Each group de-veloped exactly one project. The developmentprocess was highly iterative (feature driven de-velopment). Each project lasted one year. Duringthe development for each feature UML documen-tation was prepared. Furthermore, high level oftest code coverage was obtained by using thelatest testing tools, e.g. JUnit for unit tests, Fit-Nesse for functional tests. The last month ofdevelopment was used for additional quality as-surance and bug fixing; the quality assurance wasconducted by an external group of subjects (i.e.not the subjects involved in the development).The projects were from different domains, werebuilt using different frameworks, had differentarchitectures and covered different sets of func-tionalities, nonetheless all of them were writtenin Java.

The objects of the measurement were soft-ware development products (Java classes). Thefollowing software metrics were used in the study:

– Chidamber & Kemerer metrics suite [20]:Weighted Method per Class (WMC), Depth ofInheritance Tree (DIT), Number Of Children(NOC), Coupling Between Object classes(CBO), Response For a Class (RFC) and Lackof Cohesion in Methods (LCOM);

– a metric suggested by Henderson-Sellers [21]:Lack of Cohesion in Methods (LCOM3);

– Martin’s metrics [22]: Afferent Couplings (Ca)and Efferent Couplings (Ce).

– QMOOD metrics suite [23]: Number of PublicMethods (NPM), Data Access Metric (DAM),Measure Of Aggregation (MOA), Measure ofFunctional Abstraction (MFA) and CohesionAmong Methods (CAM);

– quality oriented extension of Chidamber& Kemerer metrics suite [24]: InheritanceCoupling (IC), Coupling Between Methods(CBM) and Average Method Complexity(AMC),

– two metrics which are based on the McCabe’scyclomatic complexity measure [25]: Maxi-mal Cyclomatic Complexity (Max_CC) andAverage Cyclomatic Complexity (Avg_CC),

– Lines Of Code (LOC),– Defects — the dependent variable; in the case

of industrial and open source projects thesources of the defect data were testers andend users, in the case of academic projectsthe source of the defect data were students(not involved in a development of a partic-ular projects) who validated the developedsoftware against the specification during thelast month (devoted to testing) of the 1-yearproject.Definitions of the metrics listed above can

be found in [16]. In order to collect the met-rics, we used a tool called Ckjm (http://gromit.iiar.pwr.wroc.pl/p_inf/ckjm). The version ofCkjm employed here was reported earlier byJureczko and Spinellis [16]). The defect countwas collected with a tool called BugInfo (http://kenai.com/projects/buginfo). The collectedmetrics are available online in a Metric Repository(all metrics: http://purl.org/MarianJureczko/MetricsRepo, metrics used in this research:http://purl.org/MarianJureczko/MetricsRepo/IET_CrossProjectPrediction). Data sets related

http://ant.apache.orghttp://gromit.iiar.pwr.wroc.pl/p_inf/ckjmhttp://gromit.iiar.pwr.wroc.pl/p_inf/ckjmhttp://kenai.com/projects/buginfohttp://kenai.com/projects/buginfohttp://purl.org/MarianJureczko/MetricsRepohttp://purl.org/MarianJureczko/MetricsRepohttp://purl.org/MarianJureczko/MetricsRepo/IET_CrossProjectPredictionhttp://purl.org/MarianJureczko/MetricsRepo/IET_CrossProjectPrediction


to the analyzed software defect prediction modelsare available from the R package [26] to streamlinereproducible research [27,28].

The employed metrics might be consideredas the classic ones. Each of them has been in usefor at least several years. Hence, the metrics arewell known, already recognized by the industryand have a good tool support. There is a numberof other metrics, some of them very promising inthe field of defect prediction, e.g. the cognitive,the dynamic and the historical metrics [15,29,30].A promising set of metrics are process metricsanalysed by Madeyski and Jureczko [31]. Someof them, like Number of Distinct Committers(NDC) or Number of Modified Lines (NML), cansignificantly improve defect prediction modelsbased on classic software product metrics [31].Nevertheless, we decided not to use them, sincethose metrics are not so popular yet (especially inthe industry) and the collecting process could bechallenging. One may expect that the limited toolsupport would result in decreasing the number ofinvestigated projects, which is a crucial factor incross-project defect prediction. We fully under-stand and accept the value of using metrics thatdescribe a variety of features. Furthermore, weare involved in the development of a tool whichincludes support for the historical metrics [32].We are going to use the tool to collect metricsfor future experiments.

3.2. Data Analysis Methodology

The empirical evaluation described further wasperformed to verify whether there is a differencebetween industrial, open–source and academicprojects with regard to defect prediction. Sta-tistical hypotheses were formulated. The defectprediction models were built and applied to theinvestigated projects. The efficiency of predictionwas used to evaluate the models and to verifythe hypotheses.

To render that in a formal way, it is neces-sary to assume that E(M, v) is the evaluationfunction. The function assesses the efficiency ofprediction of the model M on the version v ofthe investigated software project. Let c1, c2, ..., cnbe the classes from the version v in descending

order of predicted defects according to the modelM , and let d1, d2, ..., dn be the number of de-fects in each class. Di is the

∑(d1, ..., di) , i.e.,

the total defects in the first i classes. Let k bethe smallest index so that Dk > 0.8 ∗Dn, thenE(M,v) = k/n ∗ 100%. Such evaluation functionhas been used as it clearly corresponds with thesoftware projects reality we faced. It is closelyrelated to the quality goal: detect at least 80% ofdefects. The evaluation function shows how manyclasses must be tested (in practice it correspondswell to how much effort must be committed) toreach the goal when testing according to predic-tion model output. The properties of the groupof evaluation functions that the one selected byus belongs to have been analysed by Weyuker etal. [33].

The empirical evaluation is defined ina generic way which embraces three investigatedclasses of software projects. Let A,B and C bethose classes (i.e. industrial, open–source and aca-demic, respectively). Let us interpret the classesof software projects as the sets of versions ofsoftware projects. Let A be the object of thecurrent empirical evaluation. B and C will beinvestigated in subsequent experiments using theanalogous procedure. Let a be a member of theset A, a ∈ A. Let Mx be the defect predictionmodel which was trained using data from ver-sions that belong to set X. Specifically, there areMA, MB, MC and MB∪C(B ∩ C = ¬A). Subse-quently, the following values were calculated foreach a ∈ A:– E(MA, a) – let us call the set of obtained

values EA,– E(MB, a) – let us call the set of obtained

values EB,– E(MC , a) – let us call the set of obtained

values EC ,– E(MB∪C , a) – let us call the set of obtained

values EB∪C .The following statistical hypothesis may be

formulated with the use of EA, EB , EC and EB∪Csets:– H0,A,B – EA and EB come from the same

distribution.– H0,A,C – EA and EC come from the same

distribution.


Table 1. Descriptive statistics of metrics in different projects classes (X – mean; s – standarddeviation; r – Pearson correlation coefficient; ∗ – correlation significant at 0,05 level)

Industrial Open–source AcademicX s r X s r X s r

WMC 5.2 8.6 0.13∗ 10.5 13.7 0.29∗ 9.7 10.8 0.38∗DIT 3 1.7 −0.07∗ 2.1 1.3 −0.01 2.1 1.7 0.29∗NOC 0.6 8.8 0 0.5 3.4 0.03∗ 0.2 1.5 −0.04CBO 15.1 20.8 0.26∗ 10.3 16.9 0.20∗ 8 8.1 0.25∗RFC 24.7 27.5 0.28∗ 27.1 33.3 0.34∗ 26 30.6 0.53∗

LCOM 37.6 660.2 0.03∗ 95.2 532.9 0.19∗ 62 276.5 0.37∗Ca 2.8 17.7 0.16∗ 5.1 15.3 0.11∗ 3.8 6.7 0.19∗Ce 12.3 10.3 0.24∗ 5.4 7.4 0.26∗ 4.7 5.9 0.38∗

NPM 3.4 7.9 0.08∗ 8.4 11.5 0.22∗ 7.4 8.7 0.08∗LCOM3 1.4 0.6 −0.04∗ 1.1 0.7 −0.07∗ 1.1 0.6 −0.11∗

LOC 170.4 366.7 0.26∗ 281.3 614.7 0.29∗ 248.1 463.4 0.53∗DAM 0.2 0.3 0.02∗ 0.5 0.5 0.06∗ 0.6 0.5 0.07∗MOA 0.1 1 0.03∗ 0.8 1.8 0.27∗ 0.8 1.6 0.27∗MFA 0.6 0.4 −0.06∗ 0.4 0.4 −0.02∗ 0.3 0.4 0.16∗CAM 0.6 0.2 −0.13∗ 0.5 0.3 −0.19∗ 0.5 0.2 −0.17∗

IC 1.1 1.1 −0.04∗ 0.5 0.8 0.06∗ 0.3 0.5 −0.03CBM 1.7 2.4 −0.03∗ 1.5 3.1 0.10∗ 0.5 1.5 0.02AMC 30.4 39.8 0.14∗ 28.1 80.7 0.07∗ 21.2 24.7 0.24∗

Max_cc 3.3 5.7 0.17∗ 3.8 7.5 0.17∗ 3.2 5.5 0.31∗Avg_cc 1.3 1.5 0.13∗ 1.3 1.1 0.12∗ 1.1 0.8 0.17Defects 0.232 0.887 0.738 1.906 0.382 1.013

– H0,A,B∪C – EA and EB∪C come from thesame distribution.The alternative hypothesis:

– H1,A,B – EA and EB come from different dis-tributions.

– H1,A,C – EA and EC come from different dis-tributions.

– H1,A,B∪C – EA and EB∪C come from differentdistributions.When the alternative hypothesis is accepted

and mean(EA) < mean(EX), there is a signifi-cant difference in prediction accuracy: the modeltrained on the data from set A gives a signifi-cantly better prediction than the model trainedon the data from set X. The predictions are madefor all the project versions which belong to theset A. Hence, the data from set X should notbe used to build defect prediction models for theproject versions which belong to set A.

The hypotheses were evaluated by the para-metric t-test for dependent samples. The generalassumptions of parametric tests were investigatedbeforehand. The homogeneity of variance wastested using Levene’s test and the normality of

distribution was tested using the Shapiro-Wilktest [34]. All hypotheses were tested on the de-fault significance level: α = 0.05.

4. Experiments and Results

4.1. Descriptive Statistics

The three aforementioned classes of softwareprojects, namely: industrial, open–source andacademic, were described in Table 1. The descrip-tion provides information about the mean valueand standard deviation of each of the analyzedsoftware metrics. Each metric is calculated perJava class, and the statistics are based on all Javaclasses in all projets that belong to a given classof projects. Moreover, the correlations with thenumber of defects were calculated and presented.Most of the size related metrics (WMC, LCOM,LOC, Max_cc and Avg_cc) had higher valuesin open-source projects, while coupling metrics(CBO and Ce) had the greatest values in theindustrial projects. In the case of eachof the in-


Table 2. The number of classes per project

Industrial Open–source AcademicX s X s X s

No of classes 2806.3 831.7 302.6 219.4 56.4 58.4

Table 3. The number of defectsper Java class

Type X s

Open–source .7384 1.9Industrial .2323 .9Academic .3816 1.0

vestigated project classes, the RFC metric has ahigher correlation coefficient with the number ofdefects. However, there are major differences inits value (it is 0.28 in the case of the industrialprojects, and 0.53 in the case of the academicprojects) and in the sets of other metrics thatare highly correlated with the number of defects.

Collected data suggest that in all kinds of theprojects (industrial, open–source and academic)the mean LOC per class follows the rule of thumbpresented by Kan in his book [35] (Table 12.2 inChapter 12) that LOC per C++ class should beless than 480. A similar value is expected for Java.Bigger classes would suggest poor object-orienteddesign. Low LOC per class does not mean thatthe code under examination is small. To givean impression with regard to the size of the in-vestigated projects Table 2 presents numbers ofclasses per project.

The number of defects per Java class is pre-sented in Table 3. It appears that the number ofdefects per Java class in industrial and academicprojects is close with regards to standard devi-ation and mean, while open source projects arecharacterized by higher standard deviation as wellas mean. A plausible explanation is that lowerstandard deviations in academic and industrialprojects come from a more homogeneous develop-ment environment than in open source projects.

4.2. Empirical Evaluation Results

The empirical evaluation has been conductedthree times. Each time a different class of soft-ware projects was used as the object of study.

4.2.1. Industrial Projects

The descriptive statistics are summarized in Ta-ble 4. The models that were trained on the datafrom the industrial projects (’Industrial models’

in Table 4) gave the most accurate prediction.The predictions were made only for the indus-trial projects. Furthermore, the mean value of theevaluation function E(M, v) was equal to 50.82in the case of the ’Industrial models’. In the caseof the other models, the mean values equaled53.96, 55.38 and 73.59. A smaller value of theevaluation function implies better predictions.

The predictions obtained from the modelstrained on the data from the industrial projectswere compared with the predictions from theother models. The predictions were made only forthe industrial projects. The difference was statis-tically significant only in the case of comparisonwith the predictions from models trained on thedata from the academic projects (see Table 5).In this case the calculated effect size d = 1.56is extremely high (according to magnitude la-bels proposed by Cohen d effect size equal to.2 is considered small, equal to .5 is consideredmedium, while equal to .8 is considered high)while the power of a test (i.e., the probabilityof rejecting H0 when in fact it is false) is equalto 1. This important finding shows that thereexist significant differences between industrialand academic software projects with respect todefect prediction.

4.2.2. Open–source Projects

The descriptive statistics of the results of ap-plying defect prediction models to open–sourceprojects are presented in Table 6. The mod-els, which were trained on the data from theopen–source projects (’Open–source models’ inTable 6), gave the most accurate prediction.

Table 7 shows the t–test statistics, powerand effect size calculations for the open-sourceprojects.

In all of the cases p-value is lower than .05.However, instead of coming to the conclusions


Table 4. Models evaluations for the industrial projects

Model Non–industrial Open–source Academic Industrial

X 53.96 55.38 73.59 50.82s 13.29 12.01 9.68 9.86

Table 5. Dependent samples t-test for the industrialprojects.

H0,ind,open∪acad H0,ind,open H0,ind,acad

t, df = 23 −.979 −1.482 −7.637p .338 .152 .000

effect size d .200 .302 1.559power .244 .418 1

Table 6. Models evaluations for the open–source projects.

Model Industrial Non-open–source Academic Open–source

X 57.67 57.26 65.17 54.00s 19.22 18.02 14.31 16.66

now we suggests performing the Bonferroni cor-rection (explained in Section 4.2.4) beforehand.The calculated effect sizes are between mediumand small int the first two cases, while medium tolarge in the last case which is an important find-ing suggesting serious differences between opensource and academic projects with respect todefect prediction. The power of a test is cal-culated as well very high (close to 1) proba-bility of rejecting H0 is suggested when in factit is false.

4.2.3. Academic Projects

The descriptive statistics of the results of apply-ing defect prediction models to academic projectsare presented in Table 8. The obtained results aresurprising. The ’Academic models’ gave almostthe worst predictions. Slightly worse were onlythe ’Industrial models’, whilst the ’Open–sourcemodels’ and ’Non–academic models’ gave defi-nitely better predictions.

The analysis presented in Table 9 is based onthe t–test for dependent samples. The predictionsobtained from the models which were trained onthe data from the academic projects, were com-pared with predictions from the other models.The predictions were made only for the academic

projects. The differences were not statisticallysignificant, the effect sizes were below small (inthe first and third case) and between small andmedium in the second case.

4.2.4. Bonferroni Correction

Since several different hypotheses were testedon the same data set, the following kinds oferrors were likely to occur: errors in inference,including confidence intervals which fail to in-clude their corresponding population parame-ters, or hypothesis tests that incorrectly rejectthe null hypothesis. Among several statisticaltechniques that have been developed to pre-vent such instances is the Bonferroni correc-tion. The correction is based on the idea thatif n dependent or independent hypotheses arebeing tested on a data set, then one way ofmaintaining the familywise error rate is to testeach individual hypothesis at a statistical sig-nificance level of 1/n times. In our case n = 9as there are nine different hypotheses. Hence,the significance level should be decreased to0.05/9 = 0.0055. Consequently, the H0,ind,acadand H0,open,acad hypotheses will be rejected butH0,open,ind and H0,open,ind∪acad hypotheses willnot be rejected.


Table 7. Dependent samples t–test for the open–sourceprojects.

H0,open,ind H0,open,ind∪acad H0,open,acad

t, df = 41 −2.363 −2.193 −4.325p .023 .034 .000

effect size d .365 .338 .667power .752 .695 .995

Table 8. Models evaluations for the academic projects.

Model Industrial Open–source Non–academic Academic

X 56.34 50.60 53.19 55.02s 20.71 15.56 18.54 20.21

Table 9. Dependent samples t–test for the academic projects.

H0,acad,ind H0,acad,open H0,acad,ind∪open

t, df = 16 0.312 −1.484 −0.696p .759 .157 .496

effect size d .076 .360 .169power .009 .412 .164

4.3. Which Metrics are Relevant?

There is some evidence that the three investigatedcode ownership models differ with respect to de-fect prediction. Therefore, it could be helpful forfurther research to identify the casual relationsthat drive those differences. Unfortunately, thescope of software metrics (independent variablesof the defect prediction models) does not covermany aspects that may be a direct cause of de-fects. Let us assume that there is an inexperienceddeveloper who gets a requirement to implementand he does something wrong, he introduces a de-fect into the system. The true cause of the defectis a composition of several factors including thedeveloper’s experience, requirement complexityand the level of maintainability of the parts ofthe system that were changed by the developer.Only a fraction of the aforementioned factors canbe covered by the metrics available from softwarerepositories and hence many of them must beignored by the defect prediction models. Takinginto consideration the above arguments we de-cided not to define the casual relations upfront,but investigate what emerges from the models weobtained. Since it does not follow the commonly

used procedure (start with a theory and thenlook for confirmation in empirical data), it isimportant to keep in mind that results of suchanalysis do not indicate casual relations, but onlya coexistence of some phenomena.

The analysis is based on the relevancy ofparticular metrics in models obtained for differ-ent code ownership types. The defect predictionmodel has the following form:

ExpectedNumberOfDefects = a1∗M1+a2∗M2 . . .

where ai represents coefficients obtained from re-gression, Mi software metrics (independent vari-ables in the prediction). For each metric we cal-culated its importance factor (the factors arecalculated for each code ownership model respec-tively) using the following form:

IFMi =ai ∗Mi∑

j

|aj ∗Mj |

whereMi is the average value of metricMi in thetype of code ownership for which the factor is cal-culated (the averages are reported in Tab. 1). Theabove definition results in a factor that shows for


Model Industrial Open–source AcademicWMC −0.06 0 0DIT −0.08 0 0.08NOC −0.01 0 0CBO 0.17 0 −0.36RFC 0.19 0.35 0.04LCOM 0 0 0Ca 0 0.05 0.17Ce 0 0.17 0.20NPM 0 0 0.01LCOM3 0.13 0 −0.05LOC 0.05 0.14 0DAM −0.01 −0.17 −0.04MOA 0 0.12 0MFA 0.05 0 −0.02CAM −0.16 0 0IC 0 0 −0.02CBM −0.03 0 0.01AMC −0.04 0 0Max_cc 0.01 0 0Avg_cc 0 0 0

Table 10. Importance of metrics in different code ownership models.

given metric what part of the prediction modeloutput is driven by the metric and additionallypreserves the sign of the metric’s contribution.The obtained values of importance factors arereported in Tab. 10.

The importance factors show that there aresimilarities as well as differences between theprediction models trained for different types ofcode ownership. In all of them an important roleis played by the RFC metric and all of themtake into consideration coupling related metrics.However, in the case of academic projects theseare Ca and Ce, in the case of open–source Ceis much more important than Ca and for in-dustrial projects only CBO matters. There arealso metrics with positive contribution in onlyone type of projects (positive contribution meansthat the metric value grows with the numberof expected defects), i.e. LCOM3 for industrialprojects (as well as the mentioned earlier CBO),MOA and LOC for the open–source projects,DIT for the academic ones. More significant dif-ferences have been observed with regard to neg-ative contribution. The greatest negative con-tribution for academic projects have CBO andLCOM3, for open–source DAM and for indus-trial CAM.

5. Threats to Validity

5.1. Construct Validity

Threats to construct validity refer to the extentto which the measures accurately reflect the theo-retical concepts they are intended to measure [34].The mono-method bias reflects the risk of a sin-gle means of recording measures. As a result, animportant construct validity threat is that wecannot guarantee that all the links between bugsand versioning system files, and, subsequently,classes, are retrieved (e.g. when there is no bugreference identifier in the commit comment), asbugs are identified according to the comments inthe source code version control system. In fact,this is a widely known problem and the methodthat we adopted is not only broadly used butalso represents the state of the art with respectto linking bugs to versioning system files andclasses [36,37].

A closely related threat concerns anonymousinner classes. We cannot distinguish whethera bug is related to anonymous inner classes ortheir containing class, due to the file-based natureof source code version control systems. Hence, itis a common practice not to take into consider-


ation the inner classes [38–40]. Fortunately, theinner classes usually constitute a small portionof all classes (in our study it was 8.84%).

Furthermore, the guidelines of commentingbugfixes may vary among different projects.Therefore, it is possible that the interpretationof the term bug is not unique among the investi-gated projects. Antoniol et al. [41] showed thata fraction of issues marked as bugs are problemsunrelated to corrective maintenance. We did ourbest to remove such occurrences manually but infuture research we plan to apply the suggestionby Antoniol et al. to filter the non-bug issues out.

It is also worth mentioning that it was notpossible to track operations like changing theclass name or moving the class between pack-ages. Therefore, after such a change, the class isinterpreted as a new one.

5.2. Statistical Conclusion Validity

Threats to statistical conclusion validity relate tothe issues that affect the validity of inferences. Inour study we used robust statistical tools: SPSSand Statistica.

5.3. Internal Validity

The threats to internal validity concern the truecauses (e.g., external factors) that may affectthe outcomes observed in the study. The exter-nal factor we are aware of is the human factorpointed out by D’Ambros et al. [38]. D’Ambroset al. decided to limit the human factor as far aspossible and chose not to consider bug severityas a weight factor when evaluating the numberof defects. We decided to follow this approach asOstrand et al. reported how those severity ratingsare highly subjective and inaccurate [42].

Unfortunatley, each of the investigated clus-ters (i.e. code ownership models) has limited vari-ability. All academic projects were developed atthe same university. However, they differ a lotwith respect to requirements and architecture.Only two open–source projects (PBeans andckjm) do not come from Apache and all of themcan be classified as a tool or library what is inopposition to industrial projects which are en-

terprise solutions that employ database systems.Furthermore, all the industrial projects were de-veloped by the same vendor which poses a majorthreat to external validity. In consequence, it ispossible to define an alternative hypothesis thatexplains the differences between clusters, e.g. theopen–source cluster can be redefined into tools& libraries, while the industrial cluster can beredefined into enterprise database oriented solu-tion and then a hypothesis that regards differencebetween such clusters may be formulated. Un-fortunately, those alternative hypotheses cannotbe invalidated without additional projects andeven with additional projects we cannot avoidfurther alternative hypotheses with more fancydefinitions of cluster boundaries. The root causeof this issue is the sample selection procedurewhich does not guarantee random selection. Onlya small part of the population of software projectsis available for researchers and collecting datafor an experiment is a huge challenge takinginto account how difficult is to get access to thesource code or software metrics of real, industrialprojects. We were addressing this issue by takinginto consideration the greatest possible numberof projects we were able to cover with a commonset of software metrics. It does not solve theissue, but reduces the risk of accepting wronghypothesis due to some data constellations thatare a consequence of sample selection.

5.4. External Validity

The threats to external validity refer to the gen-eralization of research findings. Fortunately, inour study we considered a wide range of differentkinds of projects. They represent different own-ership models (industrial, open source and aca-demic), belong to different application domainsand have been developed according to differentsoftware development processes. However, ourselection of projects is by no means representa-tive and that poses a major threat to externalvalidity. For example, we only considered softwareprojects developed in the Java programming lan-guage. Fortunately, thanks to this limitation allthe code metrics are defined identically for eachsystem, so we have alleviated the parsing bias.


6. Conclusions

Our study has compared three classes of soft-ware projects (industrial, open–source and aca-demic) with regard to defect prediction. Theanalysis comprised the data collected from 24versions of 5 industrial projects, 42 versions of13 open–source projects, and 17 versions of 17academic projects. In order to identify differ-ences among the classes of software projects listedabove, defect prediction models were created andapplied. Each of the software project classes wasinvestigated through verifying three statistical hy-potheses. The following two noteworthy findingswere identified: two of the investigated hypotheseswere rejected: H0,ind,acad and H0,open,acad. In thecase of H0,open,ind∪acad and H0,open,ind p-valueswere below .05, but the hypotheses can not berejected due to the Bonferroni correction. Suchresults are not conclusive due to threats to ex-ternal validity discussed in Secton 5.4, as wellas the possibility that even small changes in theinput data may change the decision regardinghypothesis rejection in both directions and thuswe encourage further investigation.

As a result, we obtained some evidencethat the open–source, industrial and academicprojects may be treated as separate classes ofprojects with regard to defects prediction. Inconsequence, we we do not recommend usingmodels trained on the projects from differentcode ownership model, e.g. making predictionsfor an industrial project with a model trainedon academic or open–source projects. Of coursethe investigated classes (i.e. academic, industrialand open-source) may not be optimal and smallerclasses could be identified in order to increase theprediction performance. Identification of smallerprojects classes constitutes a promising directionfor further research.

The prediction models trained for each of theinvestigated classes of projects were further anal-ysed in order to reveal key differences betweenthem. Let us focus on the differences betweenopen–sources and industrial ones as we have verylimited evidence to support the thesis that aca-demic projects constitutes a solid class of projectswith respect to defect prediction. However, the

analysis revealed an almost self–explaining factwith regard to this class of projects. Namely, thedeeper the inheritance tree the more likely it isthat a student will introduce a defect in such aclass. Typically, API of a class with a numberof ancestors is spread among the parent classesand thus a developer that looks at only someof them may not understand the overall conceptand introduce changes that are in conflict withthe source code of one of the other classes. Inconsequence, we may expect that inexperienceddeveloper, e.g. a student, will miss some impor-tant details.

The differences between open–source and in-dustrial projects can be explained by the so calledcrowd–driven software development model com-monly used in open–source projects. High valueof the model output in those projects is mainlydriven by the LOC and MOA metrics. The firstof them simply represents the size of a class andit is not surprising that it could be challengingto understand a big class for someone who com-mits to a project only occasionally. Furthermore,the MOA metric can be considered in terms ofthe number of classes that must be known andunderstood to effectively work with a given one,which creates additional challenges for developersthat do not work in a project in a daily manner.There also is the negative contribution of theDAM metrics which also fits well to the pictureas high values of this metric correspond withlow number of public attributes and thus nar-rows the scope of source code that a developershould be familiar with. Both, industrial andopen–source models use the RFC metric, how-ever, in the case of open–source its more relevantwhich also supports the aforementioned hypoth-esis regarding crowd-driven development. Theindustrial projects are usually developed by peo-ple who know the project under development verywell. That does not mean that everyone knowseverything about each class. When it is neces-sary to be familiar with a number of differentclasses to make a single change in the project itis still likely to introduce a defect. However, inthe case of industrial projects the effect is notso strong. Furthermore, the industrial predictionmodel uses metrics that regard flaws in the class


internal structure, i.e. LCOM3 and CAM), whichmake challenges in the development regardless ofdeveloper knowledge about other classes.

The models that were trained on the academicprojects usually gave the worst predictions. Evenin the case of making predictions for academicprojects, the models trained on the academicprojects did not perform well. It was not the pri-mary goal of the study, but the obtained resultsmade it possible to arrive at that newsworthyconclusion. The academic projects are not goodas a training set for the defect prediction models.Probably, they are too immature and thus havetoo chaotic structure. The obtained results pointto the need of reconsidering the relevancy of thestudies on defect prediction that rely solely on thedata from the academic projects. Academic datasets were used even in the frequently cited [43–45]and recently conducted [46,47] studies.

Detailed data are available via a web–basedmetrics repository established by the authors(http://purl.org/MarianJureczko/MetricsRepo).The collected data may be used by other re-searchers for replicating presented here experi-ments as well as conducting own empirical studies.The obtained defect prediction models relatedto the conducted empirical study presented inthis paper are available online (http://purl.org/MarianJureczko/IET_CrossProjectPrediction).However, we recommend using the models forcross–project defect prediction with great cau-tion, since the obtained prediction performanceis moderate and presumable in most cases canbe surpassed by a project specific model.

References

[1] L. Briand, W. Melo, and J. Wust, “Assessing theapplicability of fault-proneness models acrossobject-oriented software projects,” IEEE Trans-actions on Software Engineering, Vol. 28, No. 7,2002, pp. 706–720.

[2] L. Samuelis, “On principles of softwareengineering-role of the inductive inference,”e-Informatica Software Engineering Journal,Vol. 6, No. 1, 2012, pp. 71–77.

[3] L. Fernandez, P. J. Lara, and J. J. Cuadrado,“Efficient software quality assurance approaches

oriented to UML models in real life,” Idea GroupPulishing, 2007, pp. 385–426.

[4] M. L. Hutcheson and L. Marnie, Software testingfundamentals. John Wiley & Sons, 2003.

[5] B. W. Boehm, “Understanding and controllingsoftware costs,” Journal of Parametrics, Vol. 8,No. 1, 1988, pp. 32–68.

[6] G. Denaro and M. Pezzè, “An empirical evalua-tion of fault-proneness models,” in Proceedingsof the 24rd International Conference on SoftwareEngineering, 2002. ICSE 2002. IEEE, 2002, pp.241–251.

[7] C. Kaner and W. P. Bond, “Software engineeringmetrics: What do they measure and how do weknow?” in 10th International Software MetricsSymposium. IEEE, 2004, p. 6.

[8] N. E. Fenton and M. Neil, “Software metrics:successes, failures and new directions,” Journalof Systems and Software, Vol. 47, No. 2, 1999,pp. 149–157.

[9] T. Hall and N. Fenton, “Implementing effec-tive software metrics programs,” IEEE Software,Vol. 14, No. 2, 1997, pp. 55–65.

[10] B. Turhan, T. Menzies, A. B. Bener, andJ. Di Stefano, “On the relative value ofcross-company and within-company data for de-fect prediction,” Empirical Software Engineering,Vol. 14, No. 5, 2009, pp. 540–578.

[11] M. T. Villalba, L. Fernández-Sanz, andJ. Martínez, “Empirical support for the gener-ation of domain-oriented quality models,” IETsoftware, Vol. 4, No. 1, 2010, pp. 1–14.

[12] T. Zimmermann, N. Nagappan, H. Gall, E. Giger,and B. Murphy, “Cross-project defect prediction:a large scale experiment on data vs. domain vs.process,” in Proceedings of the the 7th joint meet-ing of the European software engineering confer-ence and the ACM SIGSOFT symposium on Thefoundations of software engineering. ACM, 2009,pp. 91–100.

[13] M. Jureczko and L. Madeyski, “Towards iden-tifying software project clusters with regard todefect prediction,” in Proceedings of the 6th In-ternational Conference on Predictive Models inSoftware Engineering. ACM, 2010, p. 9.

[14] R. Subramanyam and M. S. Krishnan, IEEETransactions on Software Engineering, Vol. 29,No. 4, 2003, pp. 297–310.

[15] N. Nagappan, T. Ball, and A. Zeller, “Miningmetrics to predict component failures,” in Pro-ceedings of the 28th international conference onSoftware engineering. ACM, 2006, pp. 452–461.

[16] M. Jureczko and D. Spinellis, “Usingobject-oriented design metrics to predict

http:// purl.org/MarianJureczko/MetricsRepohttp://purl.org/MarianJureczko/IET_CrossProjectPredictionhttp://purl.org/MarianJureczko/IET_CrossProjectPrediction


software defects,” in Models and Methods ofSystem Dependability. Oficyna WydawniczaPolitechniki Wrocławskiej, 2010, pp. 69–81.

[17] M. Jureczko and L. Madeyski, “Predykcja de-fektów na podstawie metryk oprogramowania– identyfikacja klas projektów,” in Proceedingsof the Krajowa Konferencja Inżynierii Opro-gramowania (KKIO 2010). PWNT, 2010, pp.185–192.

[18] Y. Liu, T. M. Khoshgoftaar, and N. Seliya, “Evo-lutionary optimization of software quality mod-eling with multiple repositories,” Software Engi-neering, IEEE Transactions on, Vol. 36, No. 6,2010, pp. 852–864.

[19] Z. He, F. Shu, Y. Yang, M. Li, and Q. Wang, “Aninvestigation on the feasibility of cross-projectdefect prediction,” Automated Software Engi-neering, Vol. 19, No. 2, 2012, pp. 167–199.

[20] S. R. Chidamber and C. F. Kemerer, “A metricssuite for object oriented design,” IEEE Trans-actions on Software Engineering, Vol. 20, No. 6,1994, pp. 476–493.

[21] B. H. Sellers, Object-Oriented Metrics. Measuresof Complexity. Prentice Hall, 1996.

[22] R. Martin, “OO design quality metrics,” An anal-ysis of dependencies, 1994.

[23] J. Bansiya and C. G. Davis, “A hierarchicalmodel for object-oriented design quality assess-ment,” IEEE Transactions on Software Engi-neering, Vol. 28, No. 1, 2002, pp. 4–17.

[24] M.-H. Tang, M.-H. Kao, and M.-H. Chen, “Anempirical study on object-oriented metrics,” inSixth International Software Metrics Symposium,1999. Proceedings. IEEE, 1999, pp. 242–249.

[25] T. J. McCabe, “A complexity measure,” IEEETransactions on Software Engineering, No. 4,1976, pp. 308–320.

[26] L. Madeyski, reproducer: Reproduce Statis-tical Analyses and Meta-Analyses, 2015, Rpackage. [Online]. http://CRAN.R-project.org//package=reproducer

[27] L. Madeyski and B. A. Kitchenham, “Re-producible Research – What, Why andHow,” Wroclaw University of Technology, PREW08/2015/P-020, 2015.

[28] L. Madeyski, B. A. Kitchenham, and S. L.Pfleeger, “Why Reproducible Research is Ben-eficial for Security Research,” (under review),2015.

[29] J. K. Chhabra and V. Gupta, “A survey of dy-namic software metrics,” Journal of computerscience and technology, Vol. 25, No. 5, 2010, pp.1016–1029.

[30] S. Misra, M. Koyuncu, M. Crasso, C. Mateos,

and A. Zunino, “A suite of cognitive complex-ity metrics,” in Computational Science and ItsApplications–ICCSA 2012. Springer, 2012, pp.234–247.

[31] L. Madeyski and M. Jureczko, “WhichProcess Metrics Can Significantly Im-prove Defect Prediction Models? An Em-pirical Study,” Software Quality Journal,Vol. 23, No. 3, 2015, pp. 393–422. [Online].http://dx.doi.org/10.1007/s11219-014-9241-7

[32] M. Jureczko and J. Magott, “QualitySpy:a framework for monitoring software develop-ment processes,” Journal of Theoretical and Ap-plied Computer Science, Vol. 6, No. 1, 2012, pp.35–45.

[33] E. J. Weyuker, T. J. Ostrand, and R. M. Bell,“Comparing the effectiveness of several model-ing methods for fault prediction,” EmpiricalSoftware Engineering, Vol. 15, No. 3, 2010, pp.277–295.

[34] L. Madeyski, Test-driven development: An em-pirical evaluation of agile practice. Springer,2010.

[35] S. H. Kan,Metrics and models in software qualityengineering. Addison-Wesley Longman Publish-ing Co., Inc., 2002.

[36] M. Fischer, M. Pinzger, and H. Gall, “Populatinga release history database from version controland bug tracking systems,” in International Con-ference on Software Maintenance, 2003. ICSM2003. Proceedings. IEEE, 2003, pp. 23–32.

[37] T. Zimmermann, R. Premraj, and A. Zeller,“Predicting defects for eclipse,” in InternationalWorkshop on Predictor Models in Software Engi-neering, 2007. PROMISE’07: ICSE Workshops2007. IEEE, 2007, pp. 9–9.

[38] M. D’Ambros, A. Bacchelli, and M. Lanza, “Onthe impact of design flaws on software defects,”in Quality Software (QSIC), 2010 10th Interna-tional Conference on. IEEE, 2010, pp. 23–31.

[39] M. D’Ambros, M. Lanza, and R. Robbes, “Anextensive comparison of bug prediction ap-proaches,” in 7th IEEE Working Conferenceon Mining Software Repositories (MSR), 2010.IEEE, 2010, pp. 31–41.

[40] A. Bacchelli, M. D’Ambros, and M. Lanza, “Arepopular classes more defect prone?” in Fun-damental Approaches to Software Engineering.Springer, 2010, pp. 59–73.

[41] G. Antoniol, K. Ayari, M. Di Penta, F. Khomh,and Y.-G. Guéhéneuc, “Is it a bug or an enhance-ment?: a text-based approach to classify changerequests,” in Proceedings of the 2008 conferenceof the center for advanced studies on collaborative

http://CRAN.R-project.org//package=reproducerhttp://CRAN.R-project.org//package=reproducerhttp://dx.doi.org/10.1007/s11219-014-9241-7


research: meeting of minds. ACM, 2008, p. 23.[42] T. J. Ostrand, E. J. Weyuker, and R. M. Bell,

“Where the bugs are,” in ACM SIGSOFT Soft-ware Engineering Notes, Vol. 29, No. 4. ACM,2004, pp. 86–96.

[43] V. R. Basili, L. C. Briand, and W. L. Melo,“A validation of object-oriented design metricsas quality indicators,” IEEE Transactions onSoftware Engineering, Vol. 22, No. 10, 1996, pp.751–761.

[44] F. Brito e Abreu and W. Melo, “Evaluating theimpact of object-oriented design on softwarequality,” in Proceedings of the 3rd International

Software Metrics Symposium, 1996. IEEE, 1996,pp. 90–99.

[45] W. L. Melo, L. Briand, and V. R. Basili, “Mea-suring the impact of reuse on quality and pro-ductivity in object-oriented systems,” 1998.

[46] K. Aggarwal, Y. Singh, A. Kaur, and R. Malho-tra, “Empirical study of object-oriented metrics,”Journal of Object Technology, Vol. 5, No. 8, 2006,pp. 149–173.

[47] P. Martenka and B. Walter, “Hierarchicalmodel for evaluating software design quality,”e-Informatica Software Engineering Journal,Vol. 4, No. 1, 2010, pp. 21–30.

IntroductionRelated WorkEmpirical Evaluation DesignData CollectionData Analysis Methodology

Experiments and ResultsDescriptive StatisticsEmpirical Evaluation ResultsIndustrial ProjectsOpen–source ProjectsAcademic ProjectsBonferroni Correction

Which Metrics are Relevant?

Threats to ValidityConstruct ValidityStatistical Conclusion ValidityInternal ValidityExternal Validity

ConclusionsReferences

Documents

Cross–Project Defect Prediction with Respect to Code Ownership … · the very beginning to which cluster a project belongs. Asubsetofthesamedatasethadal-readybeenanalysedin[16,17].In[16]5industrial