SoftwareDevelopmentEffortEstimationUsingRegression …downloads.hindawi.com/journals/cin/2019/8367214.pdf · 2019-07-30 · (iii)Test the eﬀect of outliers on the performance of

Research ArticleSoftware Development Effort Estimation Using RegressionFuzzy Models

Ali Bou Nassif 12 Mohammad Azzeh 3 Ali Idri 4 and Alain Abran 5

1Department of Electrical and Computer Engineering University of Sharjah PO Box 27272 Sharjah UAE2Department of Electrical and Computer Engineering University of Western Ontario London Ontario Canada3Department of Software Engineering Applied Science Private University PO Box 166 Amman Jordan4Software Project Management Research Team ENSIAS Mohammed V University Rabat Morocco5Department of Software Engineering Ecole de Technologie Superieure Montreal Quebec Canada

Correspondence should be addressed to Ali Bou Nassif anassifsharjahacae

Received 27 October 2018 Revised 31 December 2018 Accepted 24 January 2019 Published 20 February 2019

Academic Editor Maciej Lawrynczuk

Copyright copy 2019 Ali Bou Nassif et al1is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Software effort estimation plays a critical role in project management Erroneous results may lead to overestimating orunderestimating effort which can have catastrophic consequences on project resources Machine-learning techniques are in-creasingly popular in the field Fuzzy logic models in particular are widely used to deal with imprecise and inaccurate data 1emain goal of this research was to design and compare three different fuzzy logic models for predicting software estimation effortMamdani Sugeno with constant output and Sugeno with linear output To assist in the design of the fuzzy logic models weconducted regression analysis an approach we call ldquoregression fuzzy logicrdquo State-of-the-art and unbiased performance evaluationcriteria such as standardized accuracy effect size and mean balanced relative error were used to evaluate the models as well asstatistical tests Models were trained and tested using industrial projects from the International Software Benchmarking StandardsGroup (ISBSG) dataset Results showed that data heteroscedasticity affected model performance Fuzzy logic models were foundto be very sensitive to outliers We concluded that when regression analysis was used to design the model the Sugeno fuzzyinference system with linear output outperformed the other models

1 Introduction and Motivation

Generally estimating project resources continues to be acritical step in project management including softwareproject development [1] Ability to predict the cost or effortof a software project has a direct impact on managementdecision to accept or reject any given project For exampleoverestimating software costs may lead to resource wastageand suboptimal delivery time while underestimation maylead to project understaffing over budgeting expenses anddelayed delivery time [2 3] 1is can lead to loss of contractsand thus potentially substantial financial losses Although inpractice there is a difference between the expressionsldquosoftware cost estimationrdquo and ldquosoftware effort estimationrdquomany authors use either to express the effort required tobuild a software project measured in person-hours In thispaper the two expressions are used interchangeably

Accurate estimation of software resources is very chal-lenging and many techniques have been investigated inorder to improve the accuracy of software estimationmodels[4 5] 1e techniques used in software effort estimation(SEE) are organized into three main groups expert judg-ment algorithmic models and machine learning [6] Expertjudgment depends on the estimatorrsquos experience while al-gorithmic models use mathematical equations to predictsoftware cost On the other hand machine-learning modelsare based on nonlinear characteristics [4] Algorithmicmodels andmachine-learningmodels depend on project andcost factors Among machine-learning models the fuzzylogic model first proposed by Zadeh [7] has been in-vestigated in the area of software cost estimation by manyresearchers who have proposed models that outperform theclassical SEE techniques [5 6 8] Even so significant lim-itations of such models have been identified

HindawiComputational Intelligence and NeuroscienceVolume 2019 Article ID 8367214 17 pageshttpsdoiorg10115520198367214

(i) When examined individually the performance ofdifferent fuzzy logic models seem to fluctuate whentested on different datasets which can in turn causeconfusion around determining the best model [9]

(ii) Most fuzzy logic models were evaluated using meanmagnitude of relative error (MMRE) mean mag-nitude of error relative to the estimate (MMER)relative error (RE) and prediction level (Pred) Allthese performance evaluation criteria are consid-ered biased [10ndash12]

(iii) Several previous studies did not use statistical teststo confirm if the proposed models were statisticallydifferent from other models Failure to employproper statistical tests would invalidate the results[13]

(iv) Effective design of Sugeno fuzzy logic models withlinear outputs which are scarce in the field ofsoftware effort estimation is a challenging taskespecially for such models with multiple inputswhere identifying the number of input fuzzy sets isin itself challenging

To address the above limitations we developed andevaluated three different fuzzy logic models using properstatistical tests and identical datasets extracted from theInternational Software Benchmarking Standards Group(ISBSG) [14] according to the evaluation criteria proposedby Shepperd and MacDonell [10] 1e three models werecompared using a multiple linear regression (MLR) andfeed-forward artificial neural network models developedwith the same training and testing datasets used for the fuzzylogic models 1is MLR type model was taken to be the basemodel for SEE

Among the challenges in designing fuzzy logic models isto determine the number of model inputs and the parametersfor the fuzzy Sugeno linear model To tackle these challengeswe proposed regression fuzzy logic where regression analysiswas used to determine the optimal number of model inputsas well as the parameters for the fuzzy Sugeno linear modelNote that our regression fuzzy logic (RFL) model should notbe confused with fuzzy regression 1e latter is actually aregression model that uses fuzzy logic as an input layer [15]whereas RFL is a fuzzy model that uses regression as an inputlayer Regarding the fuzzy Sugeno linear model (n + 1) pa-rameters are required if the number of inputs is n MLRmodels are used to find the (n + 1) parameters

In this study we investigated the following researchquestions

RQ1 What is the impact of using regression analysis totune the parameters of fuzzy models

To answer this question we used stepwise regression todetermine the number of model inputs and multiple linearregression to adjust the parameters of the Sugeno fuzzy linearmodel 1en the three fuzzy logic models as well as themultiple linear regression model were evaluated using fourdatasets based on several evaluation performance criteria suchas themean absolute error mean balanced relative error meaninverted balanced relative error standardized accuracy and

the effect size Statistical tests such as theWilcoxon test and theScott-Knott test were used to validate model performance1emean error of all models was evaluated to determine if themodels were overestimating or underestimating

RQ2 How might data heteroscedasticity affect theperformance of such models

Heteroscedasticity exists as a problem when the vari-ability of project effort increases with projects of the samesize To answer this question we filtered the ISBSG datasetand divided it into four datasets based on project pro-ductivity (effortsize) Homoscedastic datasets are those thathave very few variations in project productivity We studiedwhether the performance of each model fluctuates when aheteroscedasticity problem exists

RQ3 How do outliers affect the performance of themodels

To answer this question we conducted experiments withdatasets containing outliers and then repeated the experi-ments with the outliers removed We studied the sensitivityof all four models to outliers

In real life a machine-learning software estimationmodel has to be trained on historical datasets 1e mainobjectives of RQ2 and RQ3 are to show that data het-eroscedasticity and outliers have a big impact on theperformance of the fuzzy-regression estimation models1is would be very helpful in organizations where theyhave several historical projects 1is implies that datacleansing such as removing outliers and minimizing thedata heteroscedasticity effect would be very useful beforetraining the machine-learning prediction model Soidentifying these characteristics is of paramount impor-tance and this is precisely what best-managed organi-zations are interested in for estimation purposes Whenthe software requirements are in such a state of un-certainty best-managed organizations will work first atreducing these uncertainties of product characteristicsFor instance in the medical field data cleansing is highlyimportant Causes and effects are identified within ahighly specialized context within very specific parame-ters and generalization is avoided outside of these se-lected limitations and constraints

1e contributions of this paper can be summarized asfollows

(i) To the best of our knowledge this is the first SEEstudy that compares the three different fuzzy logicmodels Mamdani fuzzy logic Sugeno fuzzy logicwith constant output and Sugeno fuzzy logic withlinear output Both the training and testingdatasets were the same for all models In additionthe three fuzzy logic models were compared to anMLR model 1e datasets are from the ISBSGindustry dataset 1e algorithm provided in Sec-tion 4 shows how the dataset was filtered andprocessed

(ii) Investigation of the use of regression analysis indetermining the number of model inputs as well asthe parameters of the Sugeno model with linearoutput We call this approach ldquoregression fuzzyrdquo

2 Computational Intelligence and Neuroscience

(iii) Test the effect of outliers on the performance offuzzy logic models

(iv) Investigation of the influence of the hetero-scedasticity problem on the performance of fuzzylogic models

1e paper is organized as follows Section 2 summarizesrelated work in the field Section 3 presents additionalbackground information on techniques used in the exper-iments 1e preparation and characteristics of the datasetsare defined in Section 4 Section 5 demonstrates how themodels were trained and tested Section 6 discusses theresults Section 7 presents some threats to validity and lastlySection 8 concludes the paper

2 Related Work

Software effort estimation (SEE) plays a critical role inproject management Erroneous results may lead to over-estimating or underestimating effort which can have cat-astrophic consequences on project resources [16] Manyresearchers have studied SEE by combining fuzzy logic (FL)with other techniques to develop models that predict effortaccurately Table 1 lists research in FL related to our work

Table 1 also shows many studies that used datasets fromthe 1970s to the 1990s such as COCOMO NASA andCOCOMO II to train and test FL models and comparesperformance with linear regression (LR) and COCOMOequations Moreover most measured software size asthousands of line of codes (KLOC) several used thousandsof delivered source instruction (KDSI) and two used use casepoints (UCP)

Most studies showed promising results for fuzzy logic(FL) models Much of the research focus was on Mamdanifuzzy logic models rather than Sugeno fuzzy logic Only onepaper studied the difference between MLR Mamdani fuzzylogic and Sugeno fuzzy logic with constant parameters [29]Our study is the first to compare Mamdani to Sugeno withconstant output and Sugeno with linear output 1e columnldquostandalonerdquo in Table 1 indicates whether an FL model wasused as a standalone model to predict software effort oralternatively used in conjunction with other models Insome papers FL models were compared to neural network(NN) fuzzy neural network (FNN) linear regression (LR)and SEER-SEM models 1e evaluation criteria used inrelated work can be summarized as follows

(i) AAE average absolute error(ii) ARE average relative error(iii) AE absolute error(iv) Pred (x) prediction level(v) MMER mean magnitude of error relative to the

estimate(vi) MMRE mean magnitude of relative error(vii) VAF variance-accounted-for is the criterion

measuring the degree of closeness between esti-mated and actual values

(viii) RMSE root mean squared error

(ix) MdMER medianmagnitude of error relative to theestimate

(x) MdMRE median magnitude of relative error(xi) ANOVA analysis of variance(xii) RE relative error(xiii) MSE mean squared error

Several limitations are evident in the reported workFirst the majority of the above studies used single datasetsfor model evaluations 1is is a major drawback since theperformance of machine-learning models might excel onone dataset and deteriorate on other datasets [39] Secondmost of the models in Table 1 were tested using only MMREMMER and Pred (x) Moreover researchers concentratedon Mamdani-type fuzzy logic and ignored Sugeno fuzzylogic especially Sugeno with linear output Furthermorevery few studies used statistical tests to validate their resultsMyrveit and Stensrud [13] state that it is invalid to confirmthat one model is better than another without using properstatistical tests

Our paper addressed the above limitations We de-veloped and compared three different fuzzy logic modelsusing four different datasets We also used the statistical testsand evaluation criteria proposed by Shepperd and Mac-Donell [10]

3 Background

31 Fuzzy Logic Model In attempting to deal with un-certainty of software cost estimation many techniques havebeen studied yet most fail to deal with incomplete data andimpreciseness [40] Fuzzy logic has been more successful[17 41] 1is is due to the fuzzy nature of fuzzy logic wheremodel inputs have multiple memberships Fuzzy logic tendsto smoothen the transition from one membership to another[7]

Fuzzy logic (FL) models generally are grouped intoMamdani models [42] and Sugeno models [43] Inputs in FLare partitioned to membership functions with shape typessuch as triangular trapezoidal bell etc which representshow input points are mapped to output [44] 1e output ofan FL model depends on the model type ie Mamdani orSugeno Mamdani FL has its output(s) partitioned tomemberships with shapes [45 46] On the other hand inSugeno models (aka Takagi-Sugeno-Kang model) the out-put is represented as a linear equation or constant 1eSugeno fuzzy format [43] is given below

If f(x1 A1 xk is Ak) is the input group then theoutput group is y g(x1 xk) 1us the rules are asfollows

If x1 is A1 and xK is Ak then y p0 + p1x1 + middot middot middot +

pkxk where k is the number of inputs in the model and pn arethe coefficients of the linear equation When the outputequation is zero-order y will be equal to a constant value Inboth model types fuzzy logic has four main parts [47]

(i) Fuzzification which maps the crisp input data tofuzzy sets in order to obtain the degree of equivalentmembership

Computational Intelligence and Neuroscience 3

(ii) Rules where expert knowledge can be expressed asrules that define the relationship between the in-put(s) and output

(iii) Aggregation which involves firing the rulesmentioned above 1is occurs by inserting data forthe fuzzy model after which the resulting shapesfrom each output are added to generate one fuzzyoutput

(iv) Defuzzification which involves conversion of thefuzzy output back to numeric output

32 Multiple Linear Regression Model Regression is onemethod for representing the relationship between twokinds of variables [48] 1e dependent variable repre-senting the output is the one that needs to be predicted1e others are called independent variables Multipleregression involves many independent variables A lin-ear relationship between the predicted (dependent) var-iable and the independent variables can be expressed asfollows

Y β0 + β1X1 + β2X2 + middot middot middot + βpXp + ε (1)

Table 1 Related work on fuzzy logic (FL) models for software effort estimation

Ref noDataset Standalone

(yes + typeno)Comparisonconducted

Softwaresize unit

Evaluationcriteria

PublicationyearSource Size

1 [17] COCOMOrsquo81 63 projects YesSugeno FL COCOCMOmodels KDSI AAE ARE 2004

2 [18] Artificial +COCOMO81 53 projects YesMamdani FL COCOMO NampC LOC AE Pred (025) 2005

3 [19] Private 41 modules YesMamdani FL LR LOC MMER Pred(020) 2005

4 [20] NASA 18 projects YesSugeno FL LR KLOC MMRE VAFRMSE 2006

5 [6] Collected by experimentteam from 37 developers 125 projects YesMamdani FL LR NampC LOC MMER MMRE

Pred (025) 2006

6 [21] Collected by experimentteam from 37 developers 125 projects YesMamdani

FL (differentmemberships

functions types) LRNampC LOC MdMER Pred

(025) 2007

7 [22] From source no 3 amp 6 10 projects YesMamdani No comparison LOC MMRE 20098 [23] Private 200 projects YesMamdani FL LR NampC LOC MMER 20109 [24] Artificial +COCOMO81 mdash YesMamdani COCOMOFL KDSI MMRE 2010

10 [25] Private 24 projects NoSugeno Use case point (UCP) UCP MMRE Pred(035) 2011

11 [26] Private 24 projects NoMamdani Use case point (UCP) UCP MMRE Pred(035) 2011

12 [27]

COCOMO I NASA98datasets 4 project fromsoftware company in

Malaysia

160 projects NoSugeno FL-COCOMO IICOCOMO II KSLOC MMRE Pred

(025) 2011

13 [28] Collected by experimentteam from 74 developers 231 projects YesMamdani FL LR

NampCLOCreusedcode

MMER+ANOVA 2011

14 [29] Collected by experimentteam from 37 developers 125 projects YesMamdani +

Sugeno_constantFL-Mamdani FL-

Sugeno LR NampC LOC MMER Pred(025) 2013

15 [30] COCCOMO NASA 7 projects Yes FL FLNN LOC MMRE Pred(025) 2013

16 [31] COCOMO 69 projects No FNN COCOMO KESLOC MMER 200317 [32] Artificial mdash NoMamdani COCOMO81 KLOC RE 2000

18 [33] COCOMOrsquo81 69 projects No FNN COCOMO KSLOC Pred (025)MMER 2007

19 [34] COCOMO 21 projects No FNN ANNCOCOMO KLOC MMRE Pred

(025) MdMRE 2007

20 [35] ISBSG release 9 3024 projects No FNN SLOC MMRE MMERPred (025) 2009

21 [36] NASA 31 projects No FNNother tools DKLOC RMSE MMRE 2012

22 [37] NASA+ industrial 99 projects No FNN-SEERSEMSEERSEM KLOC

MMRE Pred(03) Pred (05)

MSE2015

23 [38] Private mdash No NN UCP MMRE PredMMER 2012


where Y is the dependent variable X1 X2 Xp are theindependent variables for p number of variables andβ1 β2 βp are constant coefficients that are producedfrom the data using different techniques such as least squareerror or maximum likelihood that aim to reduce the errorbetween the approximated and real data Regardless oftechnique error will exist which is represented by ε in theabove equation

33 Evaluation Criteria Examining the prediction accuracyof models depends upon the evaluation criteria used Cri-teria such as the mean magnitude of relative error (MMRE)the mean magnitude of error relative to the estimate(MMER) and the prediction level (Pred (x)) are well knownbut may be influenced by the presence of outliers and be-come biased [10 49] therefore other tests were employed inorder to improve the efficiency of the experiments

(i) Mean absolute error (MAE) calculates the average ofdifferences in the absolute value between the actualeffort (e) and each predicted effort (1113954e) 1e totalnumber of projects is represented as N

MAEi 1N

1113944

N

i1ei minus 1113954ei

11138681113868111386811138681113868111386811138681113868 (2)

(ii) Standardized accuracy (SA) measures the mean-ingfulness of model results which ensures our modelis not a random guess More details can be found in[10]

SA 1minusMAEMAEp

(3)

where MAEp is the mean value of a large numberruns of random guessing

(iii) Effect size (Δ) tests the likelihood the model predictsthe correct values rather than being a chanceoccurrence

Δ MAEminusMAEp

SP0 (4)

where SP0 is the sample standard deviation of therandom guessing strategy

(iv) Mean balance relative error (MBRE) is given by

MBRE 1N

1113944

N

i1

AEi

min ei 1113954ei( 1113857 (5)

where AEi is the absolute error and is calculated asAEi |(ei minus 1113954ei)|

(v) Mean inverted balance relative error (MIBRE) isgiven by

MIBRE 1N

1113944

N

i1

AEi

max ei 1113954ei( 1113857 (6)

(vi) Mean error (ME) is calculated as

ME 1N

1113944

N

i1ei minus 1113954ei( 1113857 (7)

4 Datasets

For this research the ISBSG release 11 [14] dataset wasemployed to examine the performance of the proposedmodels According to Jorgensen and Shepperd [1] utilizingreal-life reliable projects in SEE increases the reliability of thestudy 1e dataset contains more than 5000 industrialprojects written in different programming languages anddeveloped using various software development life cyclesProjects are categorized as either a new or enhanced de-velopment Also the software size of all projects wasmeasured in function points using international standardssuch as IFPUG COSMIC etc 1erefore to make the re-search consistent only projects with IFPUG-adjustedfunction points were considered 1e dataset containsmore than 100 attributes for each project and includes suchitems as project number project completion date softwaresize etc Also ISBSG ranks project data quality into fourlevels ldquoArdquo to ldquoDrdquo where ldquoArdquo indicates projects with thehighest quality followed by ldquoBrdquo and so on

After examining the dataset we noticed that while someprojects had similar software size effort varied extensively1e ratio between software effort (output) and software size(the main input) is called the productivity ratio We noticeda substantial difference in the productivity ratio amongprojects with similar software size For instance for the sameadjusted function point (AFP) productivity (effortsize)varied from 02 to 300 1e large difference in pro-ductivity ratio makes the dataset heterogeneous Applyingthe same model for all projects was therefore not practicalTo solve this issue projects were grouped according toproductivity ratio making the datasets more homogeneous1e main dataset was divided into subdatasets whereprojects in each subdataset had only small variations inproductivity [50] For this research the dataset was dividedinto three datasets as follows

(i) Dataset 1 small productivity ratio (P) where02lePlt 10

(ii) Dataset 2 medium productivity projects where10lePlt 20 and

(iii) Dataset 3 high productivity (Pge 20)

Also to evaluate the effect of mixing projects withdifferent productivities together a fourth dataset was addedwhich combined all three datasets Dataset 3 was not ashomogeneous as the first two since productivity in thisdataset varied between 20 and 330 1is dataset was used tostudy the influence of data heteroscedasticity on the per-formance of fuzzy logic models

Given the ISBSG dataset characteristics discussed above aset of guidelines for selection of projects was needed to filterthe dataset 1e attributes chosen for analysis were as follows


(i) AFP adjusted function points which indicatessoftware size

(ii) Development type it indicates whether the projectis a new development enhancement orredevelopment

(iii) Team size it represents the number of members ineach development team

(iv) Resource level it identifies which group was in-volved in developing this project such as develop-ment team effort development support computeroperation support and end users or clients

(v) Software effort the effort in person-hours

In software effort estimation it is important to choosenonfunctional requirements as independent variables inaddition to functional requirements [51] All of the abovefeatures are continuous variables except Resource levelwhich is categorical 1e original raw dataset contained 5052projects Using the following guidelines to filter the datasetsprojects were selected based on the following

(1) Data quality only projects with data quality A and Bas recommended by ISBSG were selected whichreduced dataset size to 4474 projects

(2) Software size in function points(3) Four inputs AFP team size development type and

resource level and one output variable softwareeffort

(4) New development projects only projects that wereconsidered enhancement development re-development or other types were ignored bringingthe total projects to 1805

(5) Missing information filtering the dataset by deletingall the rows with missing data leaving only 468 fullydescribed projects

(6) Dividing the datasets according to their productivityas explained previously to generate three distinctdatasets and a combined one

(7) Dividing each dataset into testing and trainingdatasets by splitting them randomly into 7030where 70 of each dataset was used for training and30 for testing

1e resulting datasets after applying steps 6 and 7

(a) Dataset 1 with productivity 02lePlt 10 consisted of245 projects with 172 projects for training and 73projects for testing

(b) Dataset 2 with productivity 10lePlt 20 consisted of116 projects with 81 projects for training and 35projects for testing

(c) Dataset 3 with productivity higher than or equal to20 (Pge 20) consisted of 107 projects with 75 projectsfor training and 32 projects for testing

(d) Dataset 4 combining projects from all three datasetsconsisted of 468 projects with 328 projects fortraining and 140 projects for testing

Table 2 presents some statistical characteristics of theeffort attribute in the four datasets Before using the dataseta check is needed as to whether or not the attributes datatype can be used directly in the models As discussed inSection 3 FL models divide the input into partitions toensure smoothness of transition among input partitionsthese inputs should be continuous If one of the inputs iscategorical (nominal) a conversion to a binary input isrequired [52] 1us the resource attribute a categoricalvariable was converted to dummy variables A furtheroperation was performed on the datasets to remove outliersfrom the testing dataset1e aim here was to study the effectson the results of statistical and error measurement tests Inother words we analyzed the datasets with outliers thenwithout outliers A discussion of the results is presented inSection 6 Figure 1 shows the boxplot of the four datasetswhere stars represent outliers Datasets 1 3 and 4 hadoutliers while Dataset 2 had none Removing the outliersfrom Datasets 1 3 and 4 reduced their sizes to 65 29 and130 respectively and Dataset 2 remained unchanged

5 Model Design

In this section the methods used to design the four modelsMLR Sugeno linear FL Sugeno constant FL and MamdaniFL are presented 1e training dataset for each of the fourdatasets was used to train each model and then tested usingthe testing datasets Performances were analyzed and resultsare presented in Section 6

As mentioned in Section 4 since all projects have thesame development type the latter was removed as an inputsuch that three inputs remained for each model 1ey aresoftware size (AFP) team size and resource level 1eresource-level attribute was replaced by dummy variablessince it was a categorical variable A stepwise regression wasapplied to exclude input variables that were not statisticallysignificant 1e same inputs were then utilized for all modelsin each dataset

A multiple linear regression model was generated fromevery training dataset 1e fuzzy logic models were thendesigned using the same input dataset

To design the Mamdani FL model the characteristics ofeach input were examined first specifically the min maxand average 1is gives us a guideline as to the overall shapeof memberships 1en considering that information allinputs and output were divided into multiple overlappingmemberships Simple rules were written to enable outputgeneration Usually simple rules take each input and map itto the output in order to determine the effect of every inputon the output 1is step can be shortened if some knowledgeof the data is available In our case since this knowledgeexisted setting the rules was expedited1en to evaluate andimprove the performance of the model training datasetswere randomly divided into multiple sections and a groupwas tested each time Rules and memberships were updateddepending on the resulting error from those small tests

Sugeno constant FL has similar characteristics to Mam-dani FL so the same steps were followed except for the output


design 1e output was divided into multiple constantmembership functions Initial values for each membershipfunction were set by dividing the output range into multiplesubsections and then calculating the average of each sub-section1en the performance of the model was improved byutilizing the training datasets as explained previously

Lastly the Sugeno linear FL model was designed Asexplained in Section 3 this model is a combination of fuzzylogic and linear regression concepts each of which is reflectedin the design 1e steps for designing the input membershipswere similar to the steps followed in theMamdani and Sugenoconstant models whereas the output required a differentmethodology 1e output was divided into multiple mem-berships where each membership was represented by a linearregression equation Hence the output of the dataset wasdivided into corresponding multiple overlapping sectionsand a regression analysis was applied to each in order togenerate the MLR equation Subsequently model perfor-mance was improved using the training dataset as mentionedpreviously Note that overimproving the models usingtraining datasets leads to overfitting where training results areexcellent but testing results are not promising 1ereforecaution should be taken during the training steps Aftertraining all the models were tested on the testing datasets thatwere not involved in the training steps

A summary of the system is shown in Figure 2Table 3 depicts the membership functions (mfs) of the

Mamdani Sugeno constant and Sugeno linear models in thepresence of outliers Tables 4ndash6 display the parameters of thefuzzy logic models for Dataset 1 Dataset 2 and Dataset 3respectively Table 7 displays the parameters of the ANN andMLR models

Regarding the software tools used in this researchMATLAB was used in designing fuzzy logic and neuralnetwork models For statistical tests and analysis MATLABMinitab and Excel have been used Testing results are an-alyzed and discussed in Section 6

6 Model Evaluation amp Discussion

1e following subsections discuss the performance of themodels with and without outliers

61 Testing Models with Outliers 1e three fuzzy logicmodels Sugeno linear Sugeno constant and Mamdaniwere tested on four testing datasets from ISBSG and thencompared to the multilinear regression model 1e resultingactual and estimated values were examined using the errorcriteria MAE MBRE MIBRE SA and Δ Table 8 presentsthe results of the comparisons

Table 2 Description of effort attribute in all datasets

Dataset N Mean St dev Min Max Median Skewness KurtosisEffort_dataset 1 245 8836 1486 12 14656 397 523 3717Effort_dataset 2 116 643 8873 31 4411 280 228 5Effort_dataset 3 107 367 391 11 2143 254 247 69Effort_dataset 4 468 706 1194 11 14656 310 58 505Note N number of projects St dev standard deviation

60000

50000

40000

30000

20000

10000

0

Effo

rt

Q125 565Median 50 1750Q3 75 3954

Boxplot of effort for dataset 1


Stars (lowast) denote outliers

Stars (lowast) denote outliers Stars (lowast) denote outliers

Outliers

25000

20000

15000

10000

5000

0

Effo

rt

Q1 25 1536Median 50 3524

Q3 75 13843


140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q3 75 18067Median 50 8191Q1 25 4182

Outliers

140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q1 25 1155Median 50 3440Q3 75 9285

Boxplot of effort

Outliers

Figure 1 Boxplot for effort for each dataset


Since MAE measures the absolute error between theestimated and actual value the model that has the lowestMAE generated more accurate results As shown in Table 8Sugeno linear FL generated results (bold) had the lowestMAE among the four datasets Additional tests using MBRE

and MIBRE criteria were also used to examine the accuracyof the data results 1e results as shown in Table 8 indicatethat Sugeno linear FL outperformed the other models AlsoSA measures the meaningfulness of the results generated bythe models and Δmeasures the likelihood that the data were

Data preprocessing

Dataset splitting trainingtesting

Feature selection using stepwise

regression

MLR models

Fuzzy logic

models

ANN models

Performance analysis with and without outliers

Dataset

Figure 2 Block diagram of model design steps

Table 3 Fuzzy models memberships

VariableModel

Mamdani Sugeno constant Sugeno linear Datasets of mf Type of mf of mf Type of mf of mf Type of mf Data1 Data2 Data3 Data4

AFP (input) 3 Trimf 3 Trimf 3 Trimf Included Included Included IncludedTeam size (input) 3 Trimf 3 Trimf 3 Trimf Included Included Included IncludedResource level (input) 1 Trapmf 1 Trapmf 1 Trapmf Included Excluded Included IncludedEffort (output) 3 Trimf 3 Const 3 Linear Included Included Included Included

Table 4 Parameters of Fuzzy models for Dataset 1

Mamdani Sugeno constant Sugeno linear

AFPSmall [minus350 0 350] Small [minus350 0 350] Small [minus350 0 350]

Average [140 820 1500] Average [140 820 1500] Average [140 820 1500]Large [1200 15e+ 04 2e+ 04] Large [1200 15e+ 04 2e+ 04] Large [1200 15e+ 04 2e+ 04]

Team sizeSmall [minus8 0 8] Small [minus8 0 8] Small [minus8 0 8]

Average [7 20 33] Average [7 20 33] Average [7 20 33]Large [30 50 70] Large [30 50 70] Large [30 50 70]

Resource Level 1 One [07 07 1 1] One [07 07 1 1] One [07 07 1 1]Resource Level 2 NA NA NA

EffortSmall [minus2600 0 2600] Small [973] Small [3 116 385 minus289]

Average [1500 6000 12e+ 04] Average [2882] Average [4 278 633 minus1332]Large [9500 56e+ 04 784e+ 04] Large [1242e+ 04] Large [43 361 827 minus2013]

Table 5 Parameters of Fuzzy models for Dataset2






Resource Level 1 NA NA NAResource Level 2 NA NA NA

EffortSmall [minus3000 0 3000] Small [1100] Small [1356 153 minus104]

Average [1000 1e+ 04 22e+ 04] Average [7000] Average [1212 1352 477]Large [1e+04 65e+ 04 91e+ 04] Large [2e+ 04] Large [124 115 111]


generated by chance Table 8 shows that the Sugeno linear FLpredicted more meaningful results than other techniquesacross the four datasets It is also clear from the SA and deltatests that the fuzzy Mamdani model does not predict wellwhen outliers are present as shown in Table 8

We also examined the tendency of a model to over-estimate or underestimate which was determined by themean error (ME) ME was calculated by taking the mean ofthe residuals (difference between actual effort and estimatedeffort) from each dataset with outliers As shown in Table 8all models tended to overestimate in Dataset 3 three modelsoverestimated in Dataset 1 and three models under-estimated in Dataset 2 Surprisingly Dataset 2 was the onlydataset not containing outliers Nonetheless the Sugenolinear model outperformed the other models We thencontinued to study this problem by repeating the sameprocess after removing the outliers

To confirm the validity of results we applied statisticaltests to examine the statistical characteristics of the esti-mated values resulting from the models as shown inTable 9 We chose the nonparametric Wilcoxon test tocheck whether each pair of the proposed models is sta-tistically different based on the absolute residuals 1erationale for choosing the nonparametric test was becausethe absolute residuals were not normally distributed asconfirmed by the Anderson-Darling test 1e hypothesistested was

H0 1ere is no significant difference between model(i)and model(j)H1 1ere is a significant difference between model(i)and model(j)




Average [200 900 1100] Average [200 900 1100] Average [200 900 1100]Large [8929 15e+ 04 2e+ 04] Large [8929 15e+ 04 2e+04] Large [8929 15e+ 04 2e+ 04]



Resource Level 1 One [07 07 1 1] One [07 07 1 1] One [07 07 1 1]Resource Level 2 One [07 07 1 1] One [07 07 1 1] One [07 07 1 1]

Effort

Small [minus3000 0 3000] Small [4500] Small [347 243 minus4331 0 2345]Average [1000 1e+ 04 22e+ 04] Average [15e+ 04] Average [222 884 minus1096e+ 04 0 1308e+ 04]

Large [1e+04 65e+ 04 91e+ 04] Large [348e+ 04] Large [2223 808 minus2042e+ 04 minus2748e+ 04245e+ 04]

Table 7 Parameters of ANN and MLR models for every dataset

ANN (feed-forward backprop) MLR

Dataset 1 No of hidden layers 1 Y_estminus26745 + 7529xTeam_Size +194xAFP+ 141327xldquoResource_Level 1rdquoNo of hidden neurons 8

Dataset 2 No of hidden layers 1 Y_estminus1385828 +AFPlowast 126030+Team_Sizelowast 1093311No of hidden neurons 3

Dataset 3

No of hidden layers 1 Y_est 86303198 +AFPlowast 269786 +Team_Sizelowast 851768 + ldquoResource_Level 1rdquolowastminus80826417 + ldquoResource_Level

2rdquolowastminus136874085No of hidden neurons 6

Dataset 4No of hidden layers 1 Y_est 7845531 +AFPlowast 5895416 +

Team_Sizelowast 2353906 +ldquoResource_Level 4rdquolowast 3121556No of hidden neurons 9

Table 8 Error measures and meaningfulness tests

MAE MBRE MIBRE SA Δ MEDataset 1

MLR_out 27458 77 2206 61 03 11299Fuzzy Lin_out 18426 317 395 738 04 12251Fuzzy Const_out 27795 2449 451 605 03 1599Fuzzy Mam_out 4118 3032 55 415 02 minus2454

Dataset 2MLR_out 14186 261 192 809 09 minus9102Fuzzy Lin_out 13429 21 163 819 09 minus8016Fuzzy Const_out 36747 858 402 505 05 22684Fuzzy Mam_out 32688 928 371 56 06 minus2219

Dataset 3MLR_out 75286 48 341 626 04 36963Fuzzy Lin_out 72414 2966 323 64 04 27963Fuzzy Const_out 88499 821 322 561 04 77218Fuzzy Mam_out 93322 766 376 537 04 28686

Dataset 4MLR_out 55363 3192 497 496 03 2855Fuzzy Lin_out 49253 1761 609 551 03 minus589Fuzzy Const_out 66469 4135 572 394 02 11414Fuzzy Mam_out 72657 3349 552 338 02 minus1759


If the resulting P value is greater than 005 the nullhypothesis cannot be rejected which indicates that the twomodels are not statistically different On the other hand ifthe P value is less than 005 then the null hypothesis isrejected Table 9 reports the results of theWilcoxon test withtest results below 005 given in bold 1e results of Dataset 1show that Sugeno linear FL was significantly different fromall the other models while for Datasets 2 and 4 the Sugenolinear FL amp MLR performed similarly and both were sta-tistically different from Mamdani and Sugeno constant FLFor Dataset 3 none of the models performed differently Forthis dataset based on theWilcoxon test the models were notstatistically different 1is is because a heteroscedasticityproblem exists in this dataset 1e productivity ratio for thisdataset (Dataset 3) was between 20 and 330 as discussed inSection 4 1is huge difference in productivity led to theheteroscedasticity problem and affected the performance ofthe models

One of the tests used to examine the stability of themodels was the Scott-Knott test which clusters the modelsinto groups based on data results using multiple compari-sons in one-way ANOVA [53] Models were groupedwithout overlapping ie without classifying one model intomore than one group Results were obtained simply fromthe graphs

1e Scott-Knott test uses the normally distributed ab-solute error values of the compared models 1erefore if thevalues are not normally distributed a transformation shouldtake place using the Box-Cox algorithm [54] which was thecase in our study

1e models to be compared are lined along the x-axissorted according to rank with transformed mean errorshowing across the y-axis 1e farther a model from the y-axis is the higher the rank is 1e vertical lines indicate thestatistical results for each model Models grouped together

have the same color1emean of transformed absolute erroris shown as a circle in the dashed line 1e results of Scott-Knott tests are shown in Figure 3 1e Sugeno linear modelwas grouped alone in Dataset 1 and was also the highestrank in Datasets 1 2 and 4 In Dataset 3 where there was aheteroscedasticity issue the models showed similar behav-ior Nevertheless the Sugeno linear model was among thehighest ranked MLR was ranked second twice and thirdtwice generally showing stable average performance whilethe other FL models did not show stable behavior 1isdemonstrates that the Sugeno linear model was stable andprovides higher accuracy

62 Testing Models without Outliers In this section themodels were examined again to study the effect of outliers onmodel performance 1e outliers were removed from thefour datasets and the same statistical tests and error mea-surement tools were applied to the generated results 1efiltered datasets were then used for testing the models Weused the interquantile range (IQR) method to determine theoutliers 1e IQR is defined as IQRQ3minusQ1 where Q3 andQ1 are the upper and lower quantile respectively Any objectthat is greater than Q3 + 15 IQR or less than Q1minus 15 IQRwas considered an outlier since the region between Q1minus 15IQR and Q3 + 15 IQR contains 993 of the objects [55]

An interval plot for mean absolute error was generatedfor all the models using the four testing datasets with andwithout outliers as depicted in Figure 4 Since the intervalplot was for MAE results the closer the midpoint of eachvariable to zero the better it performed Also the shorter theinterval range the better and more accurate the results1erefore it can be concluded from the plots that the generalbehavior of all the models was improved after removing theoutliers 1e results were more accurate and the range

Table 9 Wilcoxon test results

MLR_out Fuzzy Lin_out Fuzzy Const_out Fuzzy Mam_outStatistical Test (dataset 1)

MLR_out X 0002824 0567709 0007086Fuzzy Lin_out 0002824 X 0007004 194E2 06Fuzzy Const_out 0567709 0007004 X 0001765Fuzzy Mam_out 0007086 194E2 06 0001765 X

Statistical test (Dataset 2)MLR_out X 0510679 0012352 0093017Fuzzy Lin_out 0510679 X 0005372 0024118Fuzzy Const_out 0012352 0005372 X 0646882Fuzzy Mam_out 0093017 0024118 0646882 X

Statistical test (Dataset 3)MLR_out Fuzzy Lin_out Fuzzy Const_out Fuzzy Mam_out

MLR_out X 0877285 0456147 0643195Fuzzy Lin_out 0877285 X 0456147 0464303Fuzzy Const_out 0456147 0456147 X 0177199Fuzzy Mam_out 0643195 0464303 0177199 X



Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20

FuzzyMam MLR FuzzyConst

Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

FuzzyConst FuzzyMam MLR

Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33

FuzzyMam MLR FuzzyLin

Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)

Figure 3 Scott-Knott test results in datasets with outliers (a) Dataset 1 (b) Dataset 2 (c) Dataset 3 (d) Dataset 4

6000

5000

4000

3000

2000

1000

0

MA

E

Individual standard deviations were used to calculate the intervals

MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)

Figure 4 Interval plots for estimated results with and without outliers (a) Dataset 1 (b) Dataset 2 (c) Dataset 3 (d) Dataset 4


interval decreased while the midpoint was closer to zero1e Sugeno linear FL model was markedly more accuratethan the other models with or without outliers It is fair tonote that the MLR model had equivalent behavior to theSugeno linear FL in Dataset 2

To examine the improvement resulting from removal ofthe outliers the same error measures were applied todatasets without outliers Table 10 presents the results forMAE MBRE MIBRE SA and Δ

Finally the mean error (ME) from each dataset wascalculated to check the effect of removing outliers onoverestimating and underestimating project effort Wenoticed that the majority of models tend to underestimateafter removing the outliers 1is confirms the findings of thetest on the datasets with outliers where models tended tooverestimate

1e performance of all models without outliers wasimproved as the data in Table 10 indicatesWe conclude thatFL models are sensitive to outliers

In addition we examined the effect of outlier removalusing the Scott-Knott test Figure 5 shows the results of theScott-Knott test Generally our conclusions about modelstability did not change However we noted that the meanof transformed absolute error decreased 1is shows thatremoving the outliers increases the accuracy of the modelsWe conclude that the Sugeno linear FL model was thesuperior model both in the presence and absence ofoutliers

To visualize the effect of the outliers in the result of allmodels a Scatterplot was extracted for the Sugeno linearmodel in each dataset (with outliers and without outliers)where the x-axis is the actual effort and the y-axis is theestimated effort as shown in Figure 6 It is evidentthat removing the outliers decreased the drifting effecton the linear line generated Note that Dataset 2 has nooutliers

To validate the conclusion drawn about Sugeno linearoutperformance in estimating software costs its results werecompared to Forward Feed Artificial Neural Networkmodel1e ANN model created were trained and tested in the 8datasets that used in this research 4 with outliers and 4without outliers A comparison between the MAE of bothmodels is shown in Table 11 1e Fuzzy linear outperformedthe ANN model in all the datasets

63 Answers toResearchQuestions RQ1 What is the impactof using regression analysis on tuning the parameters offuzzy models

Based on the results in Section 6 we conclude thatSugeno linear FL model combined the fuzziness charac-teristics of fuzzy logic models with the nature of regressionmodels 1e different membership functions and rules usedallowed the model to cope with software parameter com-plexity 1e Sugeno linear FL model showed stable behaviorand high accuracy compared to the MLR and other modelsas shown in Scott-Knott plots We conclude that regressionanalysis can assist in designing fuzzy logic models especiallythe parameters of Sugeno fuzzy with linear output


A heteroscedasticity issue appears when the productivity(effortsize) fluctuates among projects in the same datasetTo see this impact we divided the datasets into four setscontaining different groups of productivity as described inSection 4 Heteroscedasticity appeared in the third datasetMultiple tests were applied on all the datasets to identify thedifference in performance We concluded that hetero-scedasticity had a detrimental effect on the performance offuzzy logic models but when we applied statistical tests wefound that in those datasets where heteroscedasticity existednone of the models were statistically different However weconcluded that the Sugeno linear FL model outperformedother models in the presence and absence of the hetero-scedasticity issue


After generating four datasets we extracted the outliersfrom each testing dataset We then applied the same errormeasurements and statistical tests on each as described inSection 62 We extracted interval plots for mean absoluteerror of predicted results with and without outliers as shownin Figure 4 A general improvement was noticed after re-moving outliers since we observed a major decrease in MAEand the interval range shortened (decreased) Furthermoreresults showed that datasets became more homogenous afterremoving the outliers We also found that the models tend tounderestimate in the presence of outliers and overestimatewhen outliers are removed yet the performance of allmodels improved when outliers were removed Despite thefact that outliers affect the performance of the models theSugeno linear model still proved to be the best performingmodel

We have proven in this research that the Sugeno linearfuzzy logic model outperforms other models in thepresence of outliers and absence of outliers and when thedataset is homogenous or heterogeneous We mentionedldquothe same model for all projects was therefore not prac-ticalrdquo this is because each model was trained using adifferent dataset To predict the effort of a new project in acertain organization the Sugeno linear fuzzy logic modelcan be retrained on some historical projects in the sameorganization and thus can be used to predict futureprojects

7 Threats to Validity

1is section presents threats to the validity of this researchspecifically internal and external validity Regarding internalvalidity the datasets used in this research work were dividedrandomly into training and testing groups 70 and 30respectively Although the leave-one-out (LOO) cross val-idation method is less biased than the random splittingmethod [56] the technique was not implemented because ofthe difficulty of designing fuzzy logic models with the LOOmethod In order to apply the LOO in our work more than1000 models would have had to be manually generated in


order to conduct all experiments with and without outlierswhich is extremely difficult to implement In our case fuzzylogic models were designed manually from the trainingdatasets

External validity questions whether or not the findingscan be generalized In this work four datasets were

generated from the ISBSG dataset with projects ranked Aand B Moreover unbiased performance evaluation criteriaand statistical tests were used to affirm the validity of theresults So we can conclude that the results of this paper canbe generalized to a large degree However using moredatasets would yield more robust results

FuzzyLinFuzzyConstMLRFuzzyMam

Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)

FuzzyLinMLRFuzzyMamFuzzyConstModels

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)

FuzzyConstFuzzyLinMLRFuzzyMam

Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)

FuzzyLinMLRFuzzyConstFuzzyMamModels

Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)

Figure 5 Scott-Knott test results in datasets without outliers (a) Dataset 1 (b) Dataset 2 (c) Dataset 3 (d) Dataset 4

Table 10 Error measures and meaningfulness tests for datasets without outliers


MLR_out 15184 724 2417 361 03 minus2965Fuzzy Lin_out 720 265 393 697 06 266Fuzzy Const_out 11113 2556 448 532 04 minus2145Fuzzy Mam_out 2834 3301 566 minus192 02 minus27745


Dataset 3MLR_out 47421 minus22 336 532 05 5134Fuzzy Lin_out 43763 21149 319 568 06 minus5286Fuzzy Const_out 41875 667 287 587 06 28913Fuzzy Mam_out 56085 707 358 447 05 minus15239

Dataset 4MLR_out 3982 3337 50 322 03 minus1673Fuzzy Lin_out 36137 1818 625 385 04 minus1287Fuzzy Const_out 43777 4215 561 254 03 minus1551Fuzzy Mam_out 58976 3482 559 minus04 0 minus3807Note MAE mean absolute error SA for standardized Δ (delta) effect size MBRE mean balance relative MIBRE mean inverted balance relative error


600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000

FuzzyLin_out lowast effort FuzzyLin_out (no outlier) lowast effort (no outlier)

Scatterplot of effort vs Fuzzy Lin_out and effort (N_O) vs FuzzyLin_out (N_O)

(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out

Scatterplot of effort vs FuzzyLin_out

(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000

FuzzyLin_out lowast Effort FuzzyLin_out (nooutlier) lowast Effort (nooutlier)

Scatterplot of effort vs FuzzyLin_out effort (N_O) vs FuzzyLin_out (N_O)

(c)

Figure 6 Continued


8 Conclusions

1is paper compared four models Sugeno linear FL Sugenoconstant FL Mamdani FL and MLR Models were trainedand tested using four datasets extracted from ISBSG 1enthe performance of the models was analyzed by applyingvarious unbiased performance evaluation criteria and sta-tistical tests that included MAE MBRE MIBRE SA andScott-Knott1en outliers were removed and the same testswere repeated in order to draw a conclusion about superiormodels 1e inputs for all models were software size (AFP)team size and resource level while the output was softwareeffort 1ree main questions were posed at the beginning ofthe research

RQ1What is the impact of using regression analysis ontuning the parameters of fuzzy modelsRQ2 How might data heteroscedasticity affect theperformance of such modelsRQ3 How do outliers affect the performance of themodels

Based on the discussions of the results in Section 6 weconclude the following

(1) Combining the multiple linear regression conceptwith the fuzzy concept especially in the Sugeno fuzzy

model with linear output led to a better design offuzzy models especially by learning the optimizednumber of model inputs as well as the parametersfor the fuzzy linear model

(2) Where a heteroscedasticity problem exists theSugeno fuzzy model with linear output was the bestperforming among all models However we notethat although the Sugeno linear is the superiormodel it is not statistically different from theothers

(3) When outliers were removed the performance of allthe models improved 1e Sugeno fuzzy model withlinear output did however remain the superiormodel

In conclusion results showed that the Sugeno fuzzymodel with linear output outperforms Mamdani and Sugenowith constant output Furthermore Sugeno with linearoutput was found to be statistically different from the othermodels onmost of the datasets usingWilcoxon statistical testsin the absence of the heteroscedasticity problem 1e validityof the results was also confirmed using the Scott-Knott testMoreover results showed that despite heteroscedasticity andthe influence of outliers on the performance of all the fuzzylogic models the Sugeno fuzzy model with linear outputremained the model with the best performance

150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000


Scatterplot of effort vs FuzzyLin_out and effort (N_O) vs FuzzyLin_out (N_O)

(d)

Figure 6 Scatter plots for efforts predicted by FL-Sugeno linear and actual effort withwithout the presence of outliers (a) Dataset 1(b) Dataset 2 (c) Dataset 3 (d) Dataset 4

Table 11 Comparison between Sugeno FL and ANN model based on MAE

With outliers Without outliersDataset 1 Dataset 2 Dataset 3 Dataset 4 Dataset 1 Dataset 2 Dataset 3 Dataset 4

Fuzzy Lin_out 184261 13423 724136 492523 72005 134292 43763 361367ANN_out 204165 32082 849906 569496 9618 320823 43993 449282


Data Availability

1e dataset used in this study (ISBSG) is publicly availablebut not for free It is copy-righted and it is illegal to share itwith anyone However a detailed algorithm is written inSection 4 (Datasets) to explain how the datasets are used andfiltered

Conflicts of Interest

1e authors declare that they have no conflicts of interest

Acknowledgments

1e authors thank part-time research assistant Omnia AbuWaraga Eng for conducting experiments for this paper AliBou Nassif extends thanks to the University of Sharjah forsupporting this research through the Seed Research Projectnumber 1602040221-P 1e research was also supported bythe Open UAE Research and Development Group at theUniversity of Sharjah Mohammad Azzeh is grateful to theApplied Science Private University Amman Jordan for thefinancial support granted to conduct this research

References

[1] M Jorgensen and M Shepperd ldquoA systematic review ofsoftware development cost estimation studiesrdquo IEEE Trans-actions on Software Engineering vol 33 no 1 pp 33ndash532007

[2] F J Heemstra ldquoSoftware cost estimationrdquo Information andSoftware Technology vol 34 no 10 pp 627ndash639 1992

[3] M Azzeh A B Nassif and S Banitaan ldquoComparativeanalysis of soft computing techniques for predicting softwareeffort based use case pointsrdquo IET Software vol 12 no 1pp 19ndash29 2018

[4] R Silhavy P Silhavy and Z Prokopova ldquoAnalysis and se-lection of a regression model for the use case points methodusing a stepwise approachrdquo Journal of Systems and Softwarevol 125 pp 1ndash14 2017

[5] R Silhavy P Silhavy and Z Prokopova ldquoEvaluating subsetselection methods for use case points estimationrdquo In-formation and Software Technology vol 97 pp 1ndash9 2018

[6] C Lopez-Martin C Yantildeez-Marquez and A Gutierrez-Tornes ldquoA fuzzy logic model for software development effortestimation at personal levelrdquo in Lecture Notes in ComputerScience pp 122ndash133 Springer Berlin Germany 2006

[7] L A Zadeh ldquoFuzzy setsrdquo Information and Control vol 8no 3 pp 338ndash353 1965

[8] M Hosni A Idri A Abran and A B Nassif ldquoOn the value ofparameter tuning in heterogeneous ensembles effort esti-mationrdquo Soft Computing vol 22 no 18 pp 5977ndash6010 2017

[9] N Mittas and L Angelis ldquoRanking and clustering softwarecost estimation models through a multiple comparisons al-gorithmrdquo IEEE Transactions on Software Engineering vol 39no 4 pp 537ndash551 2013

[10] M Shepperd and S MacDonell ldquoEvaluating prediction sys-tems in software project estimationrdquo Information and Soft-ware Technology vol 54 no 8 pp 820ndash827 2012

[11] T Foss E Stensrud B Kitchenham and I Myrtveit ldquoAsimulation study of the model evaluation criterion MMRErdquo

IEEE Transactions on Software Engineering vol 29 no 11pp 985ndash995 2003

[12] A Idri I Abnane and A Abran ldquoEvaluating Pred(p) andstandardized accuracy criteria in software development effortestimationrdquo Journal of Software Evolution and Processvol 30 no 4 p e1925 2017

[13] I Myrtveit and E Stensrud ldquoValidity and reliability ofevaluation procedures in comparative studies of effort pre-diction modelsrdquo Empirical Software Engineering vol 17no 1-2 pp 23ndash33 2011

[14] ISBSG International Software Benchmarking StandardsGroup 2017 httpisbsgorg

[15] H Liu J Wang Y He and R A R Ashfaq ldquoExtreme learningmachine with fuzzy input and fuzzy output for fuzzy re-gressionrdquo Neural Computing and Applications vol 28 no 11pp 3465ndash3476 2017

[16] A R Gray and S G MacDonell ldquoA comparison of techniquesfor developing predictive models of software metricsrdquo In-formation and Software Technology vol 39 no 6 pp 425ndash437 1997

[17] Z Xu and T M Khoshgoftaar ldquoIdentification of fuzzy modelsof software cost estimationrdquo Fuzzy Sets and Systems vol 145no 1 pp 141ndash163 2004

[18] M A Ahmed M O Saliu and J AlGhamdi ldquoAdaptive fuzzylogic-based framework for software development effort pre-dictionrdquo Information and Software Technology vol 47 no 1pp 31ndash48 2005

[19] C L Martin J L Pasquier C M Yanez and A G TornesldquoSoftware development effort estimation using fuzzy logic acase studyrdquo in Proceedings of Sixth Mexican InternationalConference on Computer Science (ENC 2005) pp 113ndash120Puebla Mexico September 2005

[20] A Sheta ldquoSoftware effort estimation and stock market pre-diction using takagi-sugeno fuzzy modelsrdquo in Proceedings of2006 IEEE International Conference on Fuzzy Systemspp 171ndash178 Melbourne Australia December 2006

[21] C Lopez-Martın C Yantildeez-Marquez and A Gutierrez-Tornes ldquoPredictive accuracy comparison of fuzzy models forsoftware development effort of small programsrdquo Journal ofSystems and Software vol 81 no 6 pp 949ndash960 2008

[22] I Attarzadeh and S H Ow ldquoSoftware development effortestimation based on a new fuzzy logic modelrdquo InternationalJournal of Computer Geory and Engineering vol 1 no 4pp 473ndash476 2009

[23] C Lopez-Martın and A Abran ldquoNeural networks for pre-dicting the duration of new software projectsrdquo Journal ofSystems and Software vol 101 pp 127ndash135 2015

[24] H K Verma and V Sharma ldquoHandling imprecision in inputsusing fuzzy logic to predict effort in software developmentrdquo inProceedings of 2010 IEEE 2nd International Advance Com-puting Conference (IACC) pp 436ndash442 Patiala India Feb-ruary 2010

[25] A B Nassif L F Capretz and D Ho ldquoEstimating softwareeffort based on use case point model using Sugeno FuzzyInference Systemrdquo in Proceedings of 2011 IEEE 23rd In-ternational Conference on Tools with Artificial Intelligence(ICTAI) pp 393ndash398 2011

[26] A B Nassif L F Capretz and D Ho ldquoA regression modelwith Mamdani fuzzy inference system for early software effortestimation based on use case diagramsrdquo in Proceedings ofGird International Conference on Intelligent Computing andIntelligent Systems pp 615ndash620 Prague Czech RepublicAugust 2011


[27] I Attarzadeh and S H Ow ldquoImproving estimation accuracyof the COCOMO II using an adaptive fuzzy logic modelrdquo inProceedings of 2011 IEEE International Conference on FuzzySystems (FUZZ-IEEE 2011) pp 2458ndash2464 Taipei TaiwanJune 2011

[28] C Lopez-Martin ldquoA fuzzy logic model for predicting thedevelopment effort of short scale programs based upon twoindependent variablesrdquo Applied Soft Computing vol 11 no 1pp 724ndash732 2011

[29] N Garcia-Diaz C Lopez-Martin and A Chavoya ldquoAcomparative study of two fuzzy logic models for softwaredevelopment effort estimationrdquo Procedia Technology vol 7pp 305ndash314 2013

[30] S Kumar and V Chopra ldquoNeural network and fuzzy logicbased framework for software development effort estimationrdquoInternational Journal of Advanced Research in ComputerScience and Software Engineering vol 3 no 5 2013

[31] X Huang L F Capretz J Ren and D Ho ldquoA neuro-fuzzymodel for software cost estimationrdquo in Proceedings of 2003Gird International Conference on Quality Softwarepp 126ndash133 Dallas TX USA 2003

[32] A Idri and A Abran ldquoCOCOMO cost model using fuzzylogicrdquo in 7th International Conference on Fuzzy Geory andTechnology pp 1ndash4 Atlantic City NJ USA February-March2000

[33] X Huang D Ho J Ren and L F Capretz ldquoImproving theCOCOMO model using a neuro-fuzzy approachrdquo AppliedSoft Computing vol 7 no 1 pp 29ndash40 2007

[34] S-J Huang and N-H Chiu ldquoApplying fuzzy neural networkto estimate software development effortrdquo Applied Intelligencevol 30 no 2 pp 73ndash83 2007

[35] J Wong D Ho and L F Capretz ldquoAn investigation of usingneuro-fuzzy with software size estimationrdquo in Proceedings of2009 ICSE Workshop on Software Quality (WOSQrsquo09)pp 51ndash58 Washington DC USA May 2009

[36] U R Saxena and S P Singh ldquoSoftware effort estimation usingneuro-fuzzy approachrdquo in 2012 CSI Sixth InternationalConference on Software Engineering (CONSEG) pp 1ndash6Indore India September 2012

[37] W L Du L F Capretz A B Nassif and D Ho ldquoA hybridintelligent model for software cost estimationrdquo Journal ofComputer Science vol 9 no 11 pp 1506ndash1513 2013

[38] A B Nassif Software Size and Effort Estimation from Use CaseDiagrams Using Regression and Soft Computing ModelsUniversity of Western Ontario London Canada 2012

[39] A B Nassif M Azzeh L F Capretz and D Ho ldquoNeuralnetwork models for software development effort estimation acomparative studyrdquo Neural Computing and Applicationsvol 27 no 8 pp 2369ndash2381 2016

[40] E Manalif L F Capretz A B Nassif and D Ho ldquoFuzzy-ExCOM software project risk assessmentrdquo in Proceedings of2012 11th International Conference on Machine Learning andapplications (ICMLA 2012) vol 2 pp 320ndash325 2012

[41] E Ehsani N Kazemi E U Olugu E H Grosse andK Schwindl ldquoApplying fuzzy multi-objective linear pro-gramming to a project management decision with nonlinearfuzzy membership functionsrdquo Neural Computing and Ap-plications vol 28 no 8 pp 2193ndash2206 2017

[42] E H Mamdani ldquoApplication of fuzzy logic to approximatereasoning using linguistic synthesisrdquo IEEE Transactions onComputers vol C-26 no 12 pp 1182ndash1191 1977

[43] M Sugeno and T Yasukawa ldquoA fuzzy-logic-based approachto qualitative modelingrdquo IEEE Transactions on Fuzzy Systemsvol 1 no 1 pp 7ndash31 1993

[44] A Mittal K Parkash and HMittal ldquoSoftware cost estimationusing fuzzy logicrdquo ACM SIGSOFT Software EngineeringNotes vol 35 no 1 pp 1ndash7 2010

[45] S Sotirov V Atanassova E Sotirova et al ldquoApplication of theintuitionistic fuzzy InterCriteria analysis method with triplesto a neural network preprocessing procedurerdquo ComputationalIntelligence and Neuroscience vol 2017 Article ID 21578529 pages 2017

[46] C-C Chen and Y-T Liu ldquoEnhanced ant colony optimizationwith dynamic mutation and ad hoc initialization for im-proving the design of TSK-type fuzzy systemrdquo ComputationalIntelligence and Neuroscience vol 2018 Article ID 948547815 pages 2018

[47] M Negnevitsky Artificial Intelligence A Guide to IntelligentSystems Addison WesleyPearson Boston MA USA 2011

[48] S Chatterjee and A S Hadi Regression Analysis by ExampleJohn Wiley amp Sons Hoboken NJ USA 2015

[49] M Azzeh A B Nassif S Banitaan and F Almasalha ldquoParetoefficient multi-objective optimization for local tuning ofanalogy-based estimationrdquo Neural Computing and Applica-tions vol 27 no 8 pp 2241ndash2265 2016

[50] L L Minku and X Yao ldquoHow to make best use of cross-company data in software effort estimationrdquo in Proceedingsof 36th International Conference on Software Engineering(ICSE 2014) pp 446ndash456 Hyderabad India MayndashJune 2014

[51] S Kopczynska J Nawrocki and M Ochodek ldquoAn empiricalstudy on catalog of non-functional requirement templatesusefulness andmaintenance issuesrdquo Information and SoftwareTechnology vol 103 pp 75ndash91 2018

[52] V Cheng C-H Li J T Kwok and C-K Li ldquoDissimilaritylearning for nominal datardquo Pattern Recognition vol 37 no 7pp 1471ndash1477 2004

[53] A J Scott and M Knott ldquoA cluster analysis method forgrouping means in the analysis of variancerdquo Biometricsvol 30 no 3 pp 507ndash512 1974

[54] M Azzeh and A B Nassif ldquoAnalyzing the relationship be-tween project productivity and environment factors in the usecase points methodrdquo Journal of Software Evolution andProcess vol 29 no 9 p e1882 2017

[55] J Han M Kamber and J Pei Data Mining Concepts andTechniques Morgan Kaufmann Burlington MA USA 2012

[56] E Kocaguneli and T Menzies ldquoSoftware effort models shouldbe assessed via leave-one-out validationrdquo Journal of Systemsand Software vol 86 no 7 pp 1879ndash1890 2013


Computer Games Technology

International Journal of

Hindawiwwwhindawicom Volume 2018

Hindawiwwwhindawicom

Journal ofEngineeringVolume 2018

Advances in

FuzzySystems


Volume 2018


ReconfigurableComputing



Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

thinspArtificial Intelligence

Hindawiwwwhindawicom Volumethinsp2018


Civil EngineeringAdvances in


Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications


Hindawi

wwwhindawicom Volume 2018

Advances in

Multimedia


Biomedical Imaging



Engineering Mathematics


RoboticsJournal of



Computational Intelligence and Neuroscience


Mathematical Problems in Engineering

Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018

Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018


Human-ComputerInteraction

Advances in


Scientic Programming

Submit your manuscripts atwwwhindawicom

(i) When examined individually the performance ofdifferent fuzzy logic models seem to fluctuate whentested on different datasets which can in turn causeconfusion around determining the best model [9]

(ii) Most fuzzy logic models were evaluated using meanmagnitude of relative error (MMRE) mean mag-nitude of error relative to the estimate (MMER)relative error (RE) and prediction level (Pred) Allthese performance evaluation criteria are consid-ered biased [10ndash12]

(iii) Several previous studies did not use statistical teststo confirm if the proposed models were statisticallydifferent from other models Failure to employproper statistical tests would invalidate the results[13]

(iv) Effective design of Sugeno fuzzy logic models withlinear outputs which are scarce in the field ofsoftware effort estimation is a challenging taskespecially for such models with multiple inputswhere identifying the number of input fuzzy sets isin itself challenging

To address the above limitations we developed andevaluated three different fuzzy logic models using properstatistical tests and identical datasets extracted from theInternational Software Benchmarking Standards Group(ISBSG) [14] according to the evaluation criteria proposedby Shepperd and MacDonell [10] 1e three models werecompared using a multiple linear regression (MLR) andfeed-forward artificial neural network models developedwith the same training and testing datasets used for the fuzzylogic models 1is MLR type model was taken to be the basemodel for SEE

Among the challenges in designing fuzzy logic models isto determine the number of model inputs and the parametersfor the fuzzy Sugeno linear model To tackle these challengeswe proposed regression fuzzy logic where regression analysiswas used to determine the optimal number of model inputsas well as the parameters for the fuzzy Sugeno linear modelNote that our regression fuzzy logic (RFL) model should notbe confused with fuzzy regression 1e latter is actually aregression model that uses fuzzy logic as an input layer [15]whereas RFL is a fuzzy model that uses regression as an inputlayer Regarding the fuzzy Sugeno linear model (n + 1) pa-rameters are required if the number of inputs is n MLRmodels are used to find the (n + 1) parameters

In this study we investigated the following researchquestions

RQ1 What is the impact of using regression analysis totune the parameters of fuzzy models

To answer this question we used stepwise regression todetermine the number of model inputs and multiple linearregression to adjust the parameters of the Sugeno fuzzy linearmodel 1en the three fuzzy logic models as well as themultiple linear regression model were evaluated using fourdatasets based on several evaluation performance criteria suchas themean absolute error mean balanced relative error meaninverted balanced relative error standardized accuracy and

the effect size Statistical tests such as theWilcoxon test and theScott-Knott test were used to validate model performance1emean error of all models was evaluated to determine if themodels were overestimating or underestimating


Heteroscedasticity exists as a problem when the vari-ability of project effort increases with projects of the samesize To answer this question we filtered the ISBSG datasetand divided it into four datasets based on project pro-ductivity (effortsize) Homoscedastic datasets are those thathave very few variations in project productivity We studiedwhether the performance of each model fluctuates when aheteroscedasticity problem exists


To answer this question we conducted experiments withdatasets containing outliers and then repeated the experi-ments with the outliers removed We studied the sensitivityof all four models to outliers

In real life a machine-learning software estimationmodel has to be trained on historical datasets 1e mainobjectives of RQ2 and RQ3 are to show that data het-eroscedasticity and outliers have a big impact on theperformance of the fuzzy-regression estimation models1is would be very helpful in organizations where theyhave several historical projects 1is implies that datacleansing such as removing outliers and minimizing thedata heteroscedasticity effect would be very useful beforetraining the machine-learning prediction model Soidentifying these characteristics is of paramount impor-tance and this is precisely what best-managed organi-zations are interested in for estimation purposes Whenthe software requirements are in such a state of un-certainty best-managed organizations will work first atreducing these uncertainties of product characteristicsFor instance in the medical field data cleansing is highlyimportant Causes and effects are identified within ahighly specialized context within very specific parame-ters and generalization is avoided outside of these se-lected limitations and constraints

1e contributions of this paper can be summarized asfollows

(i) To the best of our knowledge this is the first SEEstudy that compares the three different fuzzy logicmodels Mamdani fuzzy logic Sugeno fuzzy logicwith constant output and Sugeno fuzzy logic withlinear output Both the training and testingdatasets were the same for all models In additionthe three fuzzy logic models were compared to anMLR model 1e datasets are from the ISBSGindustry dataset 1e algorithm provided in Sec-tion 4 shows how the dataset was filtered andprocessed

(ii) Investigation of the use of regression analysis indetermining the number of model inputs as well asthe parameters of the Sugeno model with linearoutput We call this approach ldquoregression fuzzyrdquo





2 Related Work












3 Background
















Softwaresize unit

Evaluationcriteria







Pred (025) 2006




(025) 2007




12 [27]


Malaysia


(025) 2011


NampCLOCreusedcode

MMER+ANOVA 2011








(025) MdMRE 2007





MSE2015






MAEi 1N

1113944

N


11138681113868111386811138681113868111386811138681113868 (2)


SA 1minusMAEMAEp

(3)



Δ MAEminusMAEp

SP0 (4)



MBRE 1N

1113944

N

i1

AEi

min ei 1113954ei( 1113857 (5)



MIBRE 1N

1113944

N

i1

AEi

max ei 1113954ei( 1113857 (6)


ME 1N

1113944

N

i1ei minus 1113954ei( 1113857 (7)

4 Datasets




























5 Model Design

















60000

50000

40000

30000

20000

10000

0

Effo

rt

Q125 565Median 50 1750Q3 75 3954





Outliers

25000

20000

15000

10000

5000

0

Effo

rt

Q1 25 1536Median 50 3524

Q3 75 13843


140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q3 75 18067Median 50 8191Q1 25 4182

Outliers

140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q1 25 1155Median 50 3440Q3 75 9285

Boxplot of effort

Outliers





Data preprocessing



regression

MLR models

Fuzzy logic

models

ANN models


Dataset



VariableModel

































Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in







2 Related Work












3 Background
















Softwaresize unit

Evaluationcriteria







Pred (025) 2006




(025) 2007




12 [27]


Malaysia


(025) 2011


NampCLOCreusedcode

MMER+ANOVA 2011








(025) MdMRE 2007





MSE2015






MAEi 1N

1113944

N


11138681113868111386811138681113868111386811138681113868 (2)


SA 1minusMAEMAEp

(3)



Δ MAEminusMAEp

SP0 (4)



MBRE 1N

1113944

N

i1

AEi

min ei 1113954ei( 1113857 (5)



MIBRE 1N

1113944

N

i1

AEi

max ei 1113954ei( 1113857 (6)


ME 1N

1113944

N

i1ei minus 1113954ei( 1113857 (7)

4 Datasets




























5 Model Design

















60000

50000

40000

30000

20000

10000

0

Effo

rt

Q125 565Median 50 1750Q3 75 3954





Outliers

25000

20000

15000

10000

5000

0

Effo

rt

Q1 25 1536Median 50 3524

Q3 75 13843


140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q3 75 18067Median 50 8191Q1 25 4182

Outliers

140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q1 25 1155Median 50 3440Q3 75 9285

Boxplot of effort

Outliers





Data preprocessing



regression

MLR models

Fuzzy logic

models

ANN models


Dataset



VariableModel

































Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in












Softwaresize unit

Evaluationcriteria







Pred (025) 2006




(025) 2007




12 [27]


Malaysia


(025) 2011


NampCLOCreusedcode

MMER+ANOVA 2011








(025) MdMRE 2007





MSE2015






MAEi 1N

1113944

N


11138681113868111386811138681113868111386811138681113868 (2)


SA 1minusMAEMAEp

(3)



Δ MAEminusMAEp

SP0 (4)



MBRE 1N

1113944

N

i1

AEi

min ei 1113954ei( 1113857 (5)



MIBRE 1N

1113944

N

i1

AEi

max ei 1113954ei( 1113857 (6)


ME 1N

1113944

N

i1ei minus 1113954ei( 1113857 (7)

4 Datasets




























5 Model Design

















60000

50000

40000

30000

20000

10000

0

Effo

rt

Q125 565Median 50 1750Q3 75 3954





Outliers

25000

20000

15000

10000

5000

0

Effo

rt

Q1 25 1536Median 50 3524

Q3 75 13843


140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q3 75 18067Median 50 8191Q1 25 4182

Outliers

140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q1 25 1155Median 50 3440Q3 75 9285

Boxplot of effort

Outliers





Data preprocessing



regression

MLR models

Fuzzy logic

models

ANN models


Dataset



VariableModel

































Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in







MAEi 1N

1113944

N


11138681113868111386811138681113868111386811138681113868 (2)


SA 1minusMAEMAEp

(3)



Δ MAEminusMAEp

SP0 (4)



MBRE 1N

1113944

N

i1

AEi

min ei 1113954ei( 1113857 (5)



MIBRE 1N

1113944

N

i1

AEi

max ei 1113954ei( 1113857 (6)


ME 1N

1113944

N

i1ei minus 1113954ei( 1113857 (7)

4 Datasets




























5 Model Design

















60000

50000

40000

30000

20000

10000

0

Effo

rt

Q125 565Median 50 1750Q3 75 3954





Outliers

25000

20000

15000

10000

5000

0

Effo

rt

Q1 25 1536Median 50 3524

Q3 75 13843


140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q3 75 18067Median 50 8191Q1 25 4182

Outliers

140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q1 25 1155Median 50 3440Q3 75 9285

Boxplot of effort

Outliers





Data preprocessing



regression

MLR models

Fuzzy logic

models

ANN models


Dataset



VariableModel

































Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in























5 Model Design

















60000

50000

40000

30000

20000

10000

0

Effo

rt

Q125 565Median 50 1750Q3 75 3954





Outliers

25000

20000

15000

10000

5000

0

Effo

rt

Q1 25 1536Median 50 3524

Q3 75 13843


140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q3 75 18067Median 50 8191Q1 25 4182

Outliers

140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q1 25 1155Median 50 3440Q3 75 9285

Boxplot of effort

Outliers





Data preprocessing



regression

MLR models

Fuzzy logic

models

ANN models


Dataset



VariableModel

































Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in














60000

50000

40000

30000

20000

10000

0

Effo

rt

Q125 565Median 50 1750Q3 75 3954





Outliers

25000

20000

15000

10000

5000

0

Effo

rt

Q1 25 1536Median 50 3524

Q3 75 13843


140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q3 75 18067Median 50 8191Q1 25 4182

Outliers

140000

120000

100000

80000

60000

40000

20000

0

Effo

rt

Q1 25 1155Median 50 3440Q3 75 9285

Boxplot of effort

Outliers





Data preprocessing



regression

MLR models

Fuzzy logic

models

ANN models


Dataset



VariableModel

































Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in






Data preprocessing



regression

MLR models

Fuzzy logic

models

ANN models


Dataset



VariableModel

































Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in















Effort







Dataset 3



























Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in



















Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




Nor

mal

ized

abso

lute

erro

rs108

86

64

42

20


Models

FuzzyLin

(a)

Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18


Models

FuzzyLin

(b)

Nor

mal

ized

abso

lute

erro

rs

115

95

74

54

33


Models

FuzzyConst

(c)

Nor

mal

ized

abso

lute

erro

rs

117

93

70

47

23


Models

FuzzyLin

(d)


6000

5000

4000

3000

2000

1000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yLin

_out

(no

outli

er)

MLR

_out

(no

outli

er)

(a)

5000

4000

3000

2000

1000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yMam

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

(b)16000140001200010000

8000600040002000

0

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(c)

90008000700060005000400030002000

MA

E


MLR

_out

Fuzz

yLin

_out

Fuzz

yCon

st_ou

t

Fuzz

yMam

_out

MLR

_out

(no

outli

er)

Fuzz

yLin

_out

(no

outli

er)

Fuzz

yCon

st_ou

t(n

o ou

tlier

)

Fuzz

yMam

_out

(no

outli

er)

(d)
























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in

























Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in








Nor

mal

ized

abso

lute

erro

rs

102

81

61

41

Models

20

(a)


Nor

mal

ized

abso

lute

erro

rs

96

77

57

37

18

(b)


Nor

mal

ized

abso

lute

erro

rs

102

85

68

51

33

Models

(c)


Nor

mal

ized

abso

lute

erro

rs

113

91

68

46

23

(d)









600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




600004500030000150000

50000

40000

30000

20000

10000

0

20000150001000050000



(a)

2500020000150001000050000

30000

25000

20000

15000

10000

5000

0

Effort

Fuzz

yLin

_out


(b)

150000100000500000

70000

60000

50000

40000

30000

20000

10000

0

400003000020000100000



(c)

Figure 6 Continued


8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




8 Conclusions









150000100000500000

80000

70000

60000

50000

40000

30000

20000

10000

0

20000150001000050000



(d)






Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




Data Availability




Acknowledgments


References

































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in








































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in









Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




Documents

SoftwareDevelopmentEffortEstimationUsingRegression …downloads.hindawi.com/journals/cin/2019/8367214.pdf · 2019-07-30 · (iii)Test the eﬀect of outliers on the performance of