Efficiency analysis of ERP projects—software quality perspective

www.elsevier.com/locate/ijproman

Available online at www.sciencedirect.com

ScienceDirect

International Journal of Project Management xx (2014) xxx–xxx

JPMA-01705; No of Pages 10

Efficiency analysis of ERP projects—softwarequality perspective

Parthasarathy Sudhaman ⁎, Chandrakumar Thangavel

Department of Computer Applications, Thiagarajar College of Engineering, Madurai, India

Received 26 May 2014; received in revised form 15 August 2014; accepted 21 October 2014

Abstract

Enterprise resource planning (ERP) projects are susceptible to changes in the business environment, and the increasing velocity of change inglobal business is challenging the project managers of ERP. Literature on ERP projects reveals that the success of the ERP system greatly dependson the rigour of the software quality processes. The objective of this paper is to analyse the efficiency of ERP projects based on their qualitymeasures (defect counts) using the Data Envelopment Analysis Constant Returns to Scale (DEA CRS) model and identify the most efficient ERPprojects. Such projects may serve as potential role models and the quality processes of these projects may be adopted by the future ERP projectsleading to successful implementation. The implications of the findings for both practice and research are discussed, and probable areas of futureresearch identified.© 2014 Elsevier Ltd. APM and IPMA. All rights reserved.

Keywords: Enterprise resource planning (ERP); Data envelopment analysis (DEA); Software quality; Efficiency; Process; Software projects

1. Introduction

Enterprise resource planning (ERP) systems are integratedinformation systems, also referred to as packaged software, thekey function of which is to integrate all core functions of anenterprise regardless of business type or charter. ERP systemsare viewed as vehicles which the modern organisations use toachieve true business connectivity, a state in which everyoneknows what everyone else is doing in the business all over theworld at the same time (Daneva, 2010). Growing demand forERP projects on the one hand and the failed or out of controlERP projects on the other hand should certainly give the projectmanagers the pause (Daneva, 2010; Shaul and Tauber, 2013).To improve the efficiency of the upcoming ERP projects, it isessential to analyse the efficiency of past ERP projects. Forpractitioners and consultants involved in ERP implementation,

⁎ Corresponding author.E-mail addresses: [email protected] (P. Sudhaman),

[email protected] (C. Thangavel).

http://dx.doi.org/10.1016/j.ijproman.2014.10.0110263-7863/00/© 2014 Elsevier Ltd. APM and IPMA. All rights reserved.

Please cite this article as: P. Sudhaman, C. Thangavel, 2014. Efficiency analysquality perspective, Int. J. Proj. Manag. http://dx.doi.org/10.1016/j.ijproman.20

is of E14.10

extracting the efficient ERP projects is an invaluable source oflearning. By measuring the efficiency of ERP projects, one cancreate incentives that are likely to yield higher performance.

The development and deployment of ERP is quite differentfrom the conventional IT systems (Luo and Strong, 2004).Most ERP systems often involve business process or systemcustomisation during the ERP implementation (Luo and Strong,2004). Hence implementing an ERP system cannot be viewedin the same way as implementing a conventional IT system. ERPpackages are not systems with just software and hardware, butbusiness processes, organisational structure and also culture.

In recent years, the IT industry has been focusing ondelivering software products with utmost quality. Researchershave also indicated that in view of the less efficient softwareproduct being deployed in the organisations, the IT industryneeds to focus on the “quality” related measures of its softwareproducts (Carmel and Agarwal, 2002; Davis et al., 2006). Pastresearch (Aversano and Tortorella, 2013; Parthasarathy andAnbazhagan, 2008; Paschalidou et al., 2013; Stefanou, 2001;Stensurd and Myrtveit, 2003; Teltumbde, 2000) has analysed

RP projects—software.011

mailto:[email protected]

mailto:[email protected]

http://dx.doi.org/10.1016/j.ijproman.2014.10.011





2 P. Sudhaman, C. Thangavel / International Journal of Project Management xx (2014) xxx–xxx

the efficiency of ERP projects either from software productivitymeasures (e.g., project efforts in person-months and softwaresize) or from managerial aspects of ERP. Researchers admit thatthere is inadequacy and insufficient studies on analysing theefficiency of ERP projects from their quality perspective(Kannabiran and Sankaran, 2011; Stensurd and Myrtveit,2003), despite the fact that the software quality is a key factorin evolving strategy for both offshore vendors and customers(Kannabiran and Sankaran, 2011). Therefore, this paper analysesthe efficiency of 10 ERP projects based on their quality measuresusing the DEA CRS model.

In this paper, we define the term ‘efficiency’ as the softwareproductivity of ERP projects measured in terms of its projectefforts and software size, with due consideration for its qualitymeasure (defect counts). We consider ‘defect counts’ as a qualitymeasure, as it is the most widely accepted metric to assess thequality of a software product (Conte et al., 1986; Jalote, 2008).By the term ‘defect’, we mean something in software that causesthe software to behave in a manner that is inconsistent with therequirements or needs of the customer (Conte et al., 1986).

The term ‘efforts’ (also called ‘project efforts’) refers to themanpower required for the development of the softwareproduct in a software project (Pressman, 2010). As the maincost of developing software is the manpower employed, thecost of developing software depends on the project efforts andit is generally measured in terms of person-months. However,different methods exist for measuring the project efforts and,accordingly, the project costs would also vary.

The size of the software is measured by means of functionpoints (FP). The FP measure functionality from the user's pointof view, that is, on the basis of what the end user actually requestsand receives in return from the software system (Pressman,2010).

There are numerous definitions of software quality. In manyprevious studies, the term ‘quality’ is referred to as the ‘the totalityof features and characteristics of a product or service that bears onits ability to satisfy given need’ (Agarwal and Chari, 2007).Software quality is a multidimensional measure and it is essentialto determine what aspects of quality are important to organisations(Kannabiran and Sankaran, 2011). To ensure software quality,two approaches exist in the IT industry, namely: (i) softwarequality assurance of the process by which the software product isdeveloped, and (ii) the evaluation of the quality of the end product(Kannabiran and Sankaran, 2011). In this study, we follow thesecond approach, that is, to measure the defects of the ERPproduct deployed in the organisation to assess its quality.

Even though there are different dimensions of quality, inpractice, quality management often revolves around defects.Hence, in this paper, we define the term ‘quality’ as the numberof defects per unit size in the delivered software (Jalote, 2008).In other words, we view the quality of a software product interms of its defect counts.

For the project managers, the practitioners and the consultants,this research study identifies the efficient ERP projects and enablesthem to adopt the software processes and the quality models ofthese projects for their future ERP projects. This study extends theprevious research work of Stensurd et al. (2003), and Parthasarathy

Please cite this article as: P. Sudhaman, C. Thangavel, 2014. Efficiency analysis of Equality perspective, Int. J. Proj. Manag. http://dx.doi.org/10.1016/j.ijproman.2014.10

et al. (2008) on evaluation of ERP projects from a softwareengineering dimension. This study contributes to research bydemonstrating the application of the DEA CRS model to analysethe efficiency of ERP projects based on quality measures.

The remainder of this paper is organised as follows: Section 2presents the observations made in previous studies, emphasisingthe need to assess the efficiency of ERP projects based on theirquality measures. It also summarises related previous researchworks on evaluating ERP projects from technical as well asmanagerial aspects. Section 3 presents our research objective.Section 4 briefly describes our methodology, followed bySection 5, which discusses the data collection and analysis.Section 6 presents our results, followed by the implications forpractice and research, and discussion on validity threats. Section 7concludes the paper with the directions for future research.

2. Literature review

2.1. Efficiency of ERP — quality perspective

Many IT firms have beenmaking initiatives to strengthen theirquality processes, as it offers them opportunities to reduce costs,improve efficiency and save time (Kannabiran and Sankaran,2011; Rothenberger et al., 2010). Any IT solution would berecognised as efficient, only when it is defect free and delivers thefunctionality desired by the organisation (Jalote, 2000). Numer-ous previous studies claim that software quality is capable ofdetermining the success or failure of a software product (Luftmanand Kempaiah, 2008; Tian, 2004). Evaluation of software systemsfrom a quality perspective is a relevant issue for the developers/programmers, and various quality models have been discussed inthe literature (Aversano and Tortorella, 2013). The traditionalmodels are McCall and Boehm's models (Kan1994); the mostwidely used model being ISO/IEC 9126 model and its recentversion SQuaRE ISO 25000 (Aydin, 2012).

Software quality and IS project performance are inseparableentities and performance enhancement of IT products is possiblethrough its quality process management (Subramanian et al.,2007). A previous study by Jalote (2000) observes that softwarequality is a way to determine how well the product satisfies thecustomer. Another study observes that customer satisfaction isthe key to determine the efficiency of an IT product (Rajendran etal., 2006). Thus, connecting the previous research studies (Jalote,2000; Rajendran et al., 2006) we observe that the analysis of theefficiency of ERP projects based on their quality measures ispertinent to the success of ERP implementation.

In the past decade, research initiatives were put into under-standing the reasons for failure of ERP implementation (Chen etal., 2009; Light, 2005; Parthasarathy, 2012; Shaul and Tauber,2013), the factors that contribute to ERP's performance (Chen etal., 2009; Hwang and Grant, 2011; Parthasarathy, 2011;Rothenberger and Srite, 2009), the need to identify efficientERP projects (Aversano and Tortorella, 2013; Kannabiran andSankaran, 2011; Parthasarathy and Anbazhagan, 2008), learningthe quality processes, and models and project managementaspects of such projects to implement the future ERP projectssuccessfully (Botta-Genoulaz et al., 2005; Kannabiran and



3P. Sudhaman, C. Thangavel / International Journal of Project Management xx (2014) xxx–xxx

Sankaran, 2011; Subramanian et al., 2007). Several recentprevious research studies on evaluating ERP projects (from amanagerial or software productivity perspective) exist in theliterature (Stensurd and Myrtveit, 2003; Sudhaman, 2011), andthis indicates the growing interest among the researchers in thistopic. However, no comprehensive research has been undertakenin the past to guide us on the assessment of ERP projects from aquality perspective.

ERP projects are large-scale software projects and ofteninvolve customisation (Luo and Strong, 2004). The issue ofquality is well bonded to the ERP projects, where the endproduct is delivered only after customisation in most of theimplementations (Aversano and Tortorella, 2013; Aydin,2012). If the ERP implementation involves business processcustomisation, then its impact on the end product would bemuch less (Parthasarathy and Daneva, 2014). However, ifsystem customisation is carried out to fulfil the businessrequirements of the organisation, then most of the softwaredevelopment processes (requirements analysis, software de-sign, coding and testing) will be re-iterated by the developersand this invokes a quality inspection to review the quality of theresulting customised ERP product. In such software projects,the efficiency of the end product relies on the choice of qualitymodels and its adjoining software processes and management(Paschalidou et al., 2013). This clearly emphasises the need toassess the efficiency of the ERP product based on its qualitymeasures (Aversano and Tortorella, 2013; Sarrab et al., 2014).

2.2. Related works on evaluation of ERP projects

Stensurd and Myrtveit (2003) identified the high perfor-mance ERP projects by using the DEA model. This studymeasures the software productivity of ERP projects using thevariables-function points, and project efforts. This work did notconsider the quality measures while evaluating the ERP projects.Parthasarathy and Anbazhagan (2008) evaluated ERP projectsusing the DEA model for its software productivity using threevariables namely project efforts, function points and lines ofcode. Stefanou (2001) used a framework to evaluate the ERPsoftware product based on costs and benefit analysis. Teltumbde(2000) proposed a framework to evaluate ERP projects frommanagerial aspects, without considering any of their technicalfeatures.

Nozdrina (2009) proposed a model that uses the concept offuzzy logic to evaluate the efficiency of ERP projects withreference to software project management. A previous work byMyrtveit and Stensrud (1999) examined 30 SAP R/3 projectsbased on costs, users and EDI. Aversano and Tortorella (2013)proposed a framework to evaluate the quality and functionalityof open source ERP systems. Rothenberger and Srite (2009)investigated the ERP projects from the viewpoint of systemcustomisation and found that the requirements' uncertaintyimpacts system customisation during ERP implementation.Krishnan et al. (2000) undertook an empirical analysis to assessthe productivity and quality in software products. However,they have not discussed the scope of their study in the contextof offshore projects such as ERP. Kannabiran and Sankaran


(2011) conducted an empirical study to evaluate the keydeterminants of quality in the case of software projects deliveredthrough an offshoring model. Qin and Wang (2010) proposed aquality prediction modelling to measure and manage the qualityof the customised ERP product. This study infers that the qualityinspection to identify the defects in a customised ERP product isessential to deliver a customer centric and efficient ERP productto the enterprise.

Our review of the literature indicates that:

• Software quality contributes to efficiency of ERP (Qin andWang, 2010; Kannabiran and Sankaran, 2011; Paschalidouet al., 2013; Aversano and Tortorella, 2013;) and learningfrom such efficient ERP projects may lead the future ERPprojects to successful ERP implementation (Parthasarathyand Anbazhagan, 2008; Stensurd and Myrtveit, 2003).

• The evaluation of ERP projects is treated either from amanagerial perspective (Myrtveit and Stensrud, 1999;Nozdrina, 2009; Stefanou, 2001; Teltumbde, 2000) or in alargely organisational and cultural context (Krishnan et al.,2000; Rothenberger and Srite, 2009; Subramanian et al., 2007).

• Very few research studies are found in the literature that dealwith the evaluation of ERP projects from a technicalperspective, especially on software productivity measures(Parthasarathy and Anbazhagan, 2008; Stensurd and Myrtveit,2003). To the best of our knowledge, we could find no study onthe evaluation of ERP projects from its quality perspective(defect counts). Research on this dimension is essential, as mostof the previous studies relate the failure of ERP implementationwith that of its quality control, process maturity and defects(Aversano and Tortorella, 2013; Rajendran et al., 2006).

To approach this knowledge gap systematically, in thisstudy, we assess the ERP projects from the quality perspective.A conceptual research framework has been proposed in the nextsection to provide a high level view of the analysis of efficiencyof ERP projects.

3. Research objective

A successfully implemented ERP system can enhanceoperational efficiency by supporting a firm's business processesas well as creating competitive advantages by enabling innovativepractices (Al-Mashari et al., 2003). However, the majority of theERP implementations often end up in failure (Chen et al., 2009;Parthasarathy and Daneva, 2014), thereby crashing all the assuredbenefits of ERP software. Only the efficient ERP projects arefound to deliver the benefits of this integrated information systemto the implementing organisation. The rest of the projects stillremain in the ‘black box’ and need a review of their productdevelopment and quality processes, besides project management(Chen et al., 2009).

Current challenges facing ERP project managers are mostlytechnical and not largely managerial or organisational in nature(Rosa et al., 2013). Quantitative methods such as DEA are foundto be most suitable for analysing the efficiency of softwareprojects such as ERP (Mayrhauser et al., 2000). Hence, this paper




applies the DEA model along with statistical techniques to 10ERP projects to analyse its efficiency from a quality perspective.To fulfil this objective, based on the review of the previousresearch on the evaluation of ERP projects, a conceptual researchframework (Shown in Fig. 1) has been developed that consists ofthe input variables-project efforts (person-months) and the outputvariables-size of ERP software (FP) and quality measures (defectcounts). The input and the output variables are processed by theDEA CRS model to determine the efficiency of ERP projects.These variables are observed to contribute to the efficiency ofERP projects.

4. Data envelopment analysis (DEA) model

In this paper, we use the DEA CRS model to assess theefficiency of ERP projects. At first, in this section, we present abrief overview of the DEA model, followed by a summary ofprevious research works that involve the application of DEA toIT and IS areas. This will benefit the readers to understand thesuitability of the DEA methodology in the context of efficiencyanalysis of ERP projects.

DEA is an alternative nonparametric method of measuringefficiency that uses mathematical programming rather thanregression (Stensurd and Myrtveit, 2003). Charnes et al. (1978)made the initial publication on the DEA model that handledConstant Returns to Scale (CRS). Afriat (1972) took efforts toapply the DEA model with Variable Returns to Scale (VRS).When applying the DEA model, the researcher has to decidewhether to use CRS or VRS. For IT projects (small or medium),CRS is found suitable (Charnes et al.1978; Stensurd andMyrtveit, 2003), whereas for large-scale IS projects delivering agreater number of outputs, the VRS is found to be appropriate(Stensurd and Myrtveit, 2003). In this paper, we chose the DEACRS model for efficiency analysis. After selecting the CRS orVRS, the next task is to select the optimisation mode for theDEA model.

The DEA model consists of two optimisation modes namelythe minimising inputs (MIN-IN) (input reducing efficiencymeasure) and the maximising outputs (MAX-OUT) (outputincreasing efficiency measure). Both the measures are accept-able in the context of IT and IS projects (Boehm, 1981). TheMIN-IN aims to minimise the inputs to generate the sameoutputs, whereas the MAX-OUT aims to maximise the outputs

Fig. 1. Conceptual res


given the current inputs. The DEA is known as a multi-factorproductivity analysis model to measure the relative efficienciesof a homogenous set of decision-making units (DMUs). Theefficiency score considering multiple input and output factors isdefined as:

Efficiency ¼ Weighted sum of outputs=Weighted sum of inputs

ð1Þ

If there are n DMUs, each with m inputs and s outputs, thenthe relative efficiency score of a test DMU p is found bysolving the following model proposed by Charnes et al. (1978)and Ray (2004).

Max

XS

K¼1

Vkykp

Xm

j¼1

ujx jp

ð2Þ

such that

XS

k¼1

vkyki

Xm

j¼1

ujx ji

≤1 ∀i; vk ; uj≥0 ∀k; j;

where k = 1 to s, j = 1 to m, i = 1 to n;yki = amount of output k produced by DMU i,xji = amount of input j utilised by DMU i,vk = weight given to output k,uj = weight given to input j.The DEA model requires that the input and the output

variables are related to each other. To fulfil this property, thecorrelation analysis should be carried out between the selectedinput and output variables before the application of the DEA(Ray, 2004). The computational procedure to solve the DEAproblem is beyond the scope of this study and hence it has notbeen described here. Previous studies that have used the DEAmodel for efficiency analysis have utilised the automatedsoftware tools (Koch, 2007; Ray, 2004) developed by leadingIT vendors to process the inputs and the outputs of the DEA.

earch framework.



Table 2Profile of ERP team involved in the development of ‘Mars’.

Role Business sector/domain Avg. experience —IT/IS/ERP projects(in years)

Business/IT analyst Banking 05Education 06Logistics 07

Programmers/developers Supply chain 03Education 05Banking 06

Functional consultants Healthcare 07Education 04Insurance 03

Technical consultants Supply chain 05Banking 04Education 03


4.1. Application of DEA in IT and IS areas

In the past decade, we could witness the interest among theresearchers to apply the DEA model in the contexts of theefficiency analysis of IT and IS projects of small to midsize aswell as large organisations. For this phenomenal growth, onereason could be that the DEA has opened up possibilities foruse in cases which have been resistant to other approachesbecause of the complex (often unknown) nature of the relationsbetween the multiple inputs and multiple outputs involved inmany of these projects.

To identify recently published related work on the applicationof DEA in IT and IS projects, we searched the Scopus digitallibrary (www.scopus.com). We present a comprehensive, but notexhaustive, summary of DEA papers in IT and IS areas (Table 1).

5. Data collection and analysis

The data set used in this study consists of 10 ERP projectscompleted by a midsize IT company (given the pseudonym Sylaxin this paper) involved in the development and deployment ofconventional software as well as IS, such as ERP and CustomerRelationship Management (CRM) systems. Headquartered inIndia, the company Sylax has 12 IT development centres spreadacross India, USA, Canada and Europe. The company is strivingto provide IT solutions that solve complex business problems ofthe enterprises. Sylax has over 56,000 users from 300-plus clientorganisations, globally, since its inception. The company providessolutions to multiple verticals including higher education,banking, insurance, logistics, supply chain, and healthcare. Thecompany has achieved the rare distinction of ISO 27001:2005information security standard. The company has a currentworkforce of almost 1100.

As observed by software engineering researchers (Basili andZelkowitz, 2007; Wohlin, 2000), similar to other leading ITcompanies, Sylax maintained a project database of previouslycompleted ERP projects. This database would consist of the detailsof their projects such as software process models, project costs,efforts, quality assurance activities and project management. Allthe ERP projects in the sample implement the same ERP softwarepackage (referred to as ‘Mars’); therefore it is a homogeneous dataset. The projects were undertaken by the company between 2006and 2012.

Table 1Application of DEA in IT and IS areas.

Literature source Application of DEA

Paschalidou et al. (2013) Software quality assessParthasarathy and Anbazhagan (2008) Evaluation of ERP proKoch (2007) Estimation of project eLall and Teyarachakul (2006) Proposed an approachFisher et al. (2005) Evaluation of the implStensurd and Myrtveit (2003) Identification of high pMayrhauser et al. (2000) Estimation of softwareFisher and Sun (1996) Evaluation of the indivThore et al. (1996) Ranking the efficiencyDoyle and Green (1994) Benchmarking 22 micr


We collected the data in mid 2013 and it is an ongoingeffort. All the projects are related to the development of ERPsystems for educational institutions (small or midsize univer-sities) and the average duration of implementation rangesbetween 6 and 14 months. All these implementation projectshave followed a modular approach, i.e., a module of the ERPpackage has been implemented by the respective organisationand not the complete ERP package.

The capability of the developers (programmers) and otherERP team members such as consultants and IT analysts cangreatly influence the project efforts (Jalote, 2008). The domainknowledge and experience of these members in implementingIT and IS projects such as ERP should be considered whenestimating the efforts required to develop a module orcustomise an existing module of an ERP package. Table 2shows the profile of the ERP team members of Sylax involvedin developing the ERP package ‘Mars’. From Table 2, we findthat Sylax has carefully built its ERP team in such a way that itpossesses the required domain knowledge as well as rich ITexperience required for developing ERP packages for educa-tional institutions.

The ERP package ‘Mars’ comprised the following modules:academic, administration, admissions, industry interface, stu-dents activities, research, examination, and hostel and humanresources. The ERP product ‘Mars’ was developed usingMicrosoft technologies. It was built to enable the organisationto get a 360-degree view of its business, anytime, from anywhere

ment of traditional software projects.jects for effort prediction using DEA and regression analysis.fforts at an early stage of ERP implementation.for ERP software selection.ementation of ERP in SMEs.erformance ERP projects based on software productivity.productivity for traditional software products.idual performance of 22 e-mail packages.of 44 US IT companies using the datasets from 11 companies.ocomputers using DEA and regression analysis.


http://www.scopus.com


Table 3Descriptive statistics for ERP projects (Bi).

I/O units N MEAN STD MIN MAX MEDIAN

Project efforts (PE) (person-months) (input) 10 16.91 16.12 1.652 50.027 11.15Size of ERP software (FP) (output) 10 1036.5 631.42 345 2219 889.5Defect counts (DC) (output) 10 8.1 4.067 3 15 8.5


by integrating all its business functions onto one single platformand automating and integrating them, end-to-end.

In software engineering research, it would seem morereasonable to compare a project with other projects of similarsize (Stensurd and Myrtveit, 2003). It is observed that thecomplexity of implementation of a module of ERP (say,admissions) would vary from the implementation of anothermodule (say, academic). Hence, for productivity-based efficiencyanalysis of software projects, the level of complexity and the sizeof the module chosen for efficiency analysis should be the same.Thus, in this study, we have captured data for 10 ERP projectsthat were related to the development and deployment of the sameERP module — academic. The primary functionality of thismodule is to automate the teaching and learning processes ofundergraduate and postgraduate courses in educational institu-tions and also provide a seamless interface to other relatedprocesses such as examinations.

In this study, for the purpose of efficiency analysis, we consider10 ERP projects which deal with the implementation of the verysame ERP module, Academic. However, it should be noted fromTable 3 that the size of these modules (software size shown in FP)varies from one project to another. This is trivial in the case of ERPimplementation as the degree of system customisation variessignificantly from one project to another project.

A project database is a repository of the data of successfullycompleted software projects (Jalote, 2000). The concept ofdeveloping such database was floated two decades ago by softwareengineering researchers (e.g., Jalote (2000)). This database wasmeant to consist of data such as actual efforts used in the project,function points, lines of code of the developed product, datapertaining to project planning and management, data related toquality assurance processes (say, defects) and related data from riskinformation sheets (Jalote, 2000). Every IT company has beenmaintaining its own project database that was designed anddeveloped by its programmers/developers. A similar database wasalso maintained by Sylax and limited access was provided to theauthors to capture data to be used purely for this research work.

The candidate input and output metrics captured for this studyfrom these projects are: project efforts (PE) (person-months)(input), size of ERP software (FP) (output) and defect counts1

(DC) (output). The descriptive statistics for 10 ERP projects for

1 A defect indicates nonconformance to specific user requirements (Jalote,2008). We use various defect measurements as direct indicators of quality (Kan,1994). In this study, defect count (Jalote, 2008) is computed as (X + Y + Z)where X = extreme defects; Y = major defects and Z = minor defects. A minordefect makes the application unusable in some way (e.g., a modification isrequired to a screen shot or a report). A major defect creates a fault so that someof the software components become unusable. An extreme defect creates afailure in such a way that some part of the application becomes totally unusable.


these variables are shown in the following Table 3. There were nospecialised units in the data set.

As we discussed in Section 4.0, the correlation between theI/O units of the 10 ERP projects has to be verified before theapplication of the DEA CRS model and this was established inour study by means of correlation analysis (Levine, 1999) asshown in Table 4. This analysis shows that there is a correlationbetween the I/O units of the DEA CRS model, as the coefficientof correlation R lies between 0 and 1 for these units.

As we discussed in Section 4, we need to decide on thechoice of the DEA model (CRS or VRS) and the optimisationmode (MIN-IN or MAX-OUT) before the execution of theDEA. In this paper, we use the DEA CRS model for theefficiency analysis of 10 medium ERP projects. Also, we setthe optimisation mode as the input reducing efficiency measure(MIN-IN). The optimisation mode ‘MIN-IN’ seeks to minimiseinputs to produce the same outputs.

The DEA for 10 ERP projects (Bi) was carried out using themost prominently used DEA software package “FrontierAnalyst” (www.banxia.com). There are also other automatedtools available in the market for this analysis (Koch, 2007; Ray,2004).

6. Results

Table 5 shows the efficiency report of 10 ERP projectsanalysed from a quality perspective using the DEA CRS model.Table 5 consists of the following results:

(i) Potential improvement required for the I/O units (PE andFP) so as to enable the respective ERP projects (Bi) tobecome 100% efficient.

(ii) Efficiency of 10 ERP projects (Bi) based on the I/O units.

From the results, we see that the projects B1 and B7 arefound to be most efficient, followed by project B8 with anaverage efficiency score and the rest of the projects haveobtained an efficiency score of between 5% and 36%. Thisleads to the inference that 70% of the ERP projects chosen inthis study are weak in terms of their efficiency from a qualityperspective. However, it is also evident that the potentialimprovement required for the I/O units for some of theseprojects are achievable. It is interesting to look at the results forthe projects B8, B9 and B10. All these three projects possess thesame defect count, i.e., 3; however, the efficiency of theseprojects differs because of the potential improvement requiredfor its input unit, i.e., efforts.

We observe that, for a project to be reasonably efficient, theproject efforts, size of the software product and defects arising


http://www.banxia.com


Table 4Correlation analysis for input/output units of 10 ERP projects (Bi).

I/O units Project efforts (PE) (person-months) Size of ERP software (FP) Defect counts (DC)

Project efforts (PE) (person-months) 1 0.65 0.08Size of ERP software (FP) 0.65 1 0.05Defect counts (DC) 0.08 0.05 1


out of this product should be well balanced. As per the DEAanalysis the efficiency score of the projects B1 and B7 are veryhigh, despite their reasonable defect counts. This is possible asthese projects have used minimal project efforts to develop aproduct of size (FP ranging between 300 and 600). Hence,when we estimate its efficiency with MIN-IN mode of theDEA, i.e., evaluation, with the objective of minimising theproject efforts, then naturally these projects deserve to bedeclared as efficient. This provides an inference to the projectmanagers that the accurate effort estimation would help themreduce defects during the product development and sustain theefficiency of their project.

The significance of the DEA is that it only formalises andmakes explicit the expert judgement and provides quantitativefigures rather than qualitative expert assessments (Stensurd andMyrtveit, 2003). One might be interested to know as to how B1

and B7 acquired a very high efficiency score. Althoughin-depth investigating of each of 10 ERP projects chosen forthis study is beyond the scope of this study, our informaldiscussion with the consultants and several of the developersinvolved in these projects reveals that the projects B1 and B7

had the following unique features: (i) Both these projectsadapted a different strategy to capture business requirementsand align the same with the ERP system. This might havereduced their efforts for system customisation and probably thesubsequent defects expected from the customised ERP shouldalso be very low. In this way, the project efforts for theseprojects were under control. (ii) Quality processes, especiallyquality inspection and assurance, were strictly enforced and the

Table 5Software quality based efficiency analysis report of 10 ERP projects (Bi).

Project ID (Bi) I/O units (actual) I/O units (targ

PE a FP b DC c PE a

B1 1.698 555 10 1.698B2 8.938 699 15 2.55B3 11.513 1348 12 4.12B4 31.709 2219 8 6.79B5 34.25 599 8 1.83B6 50.027 1908 10 5.84B7 1.652 345 9 1.53B8 6.631 1080 3 3.3B9 11.899 482 3 1.47B10 10.8 1130 3 3.46a PE-project efforts (person-months) (input).b FP-function points (output).c DC-defect counts (output).


quality assurance group of the product development was giventhe liberty to decide the release of the product, after a formaltechnical review by their members.

6.1. Implications for practice and research

Development of ERP software differs from the analogy oftraditional software projects (Stensurd and Myrtveit, 2003).From the results, we could draw that the appropriate usage ofefforts in ERP projects may contribute to reduction of defects,which in turn subsequently would contribute to the efficiencyof ERP. In other words, projects with accurate estimationwould enable the managers to keep control of the defect countsrelative to the size of the software. We acknowledge that wehave provided only the preliminary results and that relatedvalidity concerns remain our most important issue.

With reference to project efforts and function points, ourresults agree with the observation made in the previous studiesby Stensurd et al. (2003) and Parthasarathy et al. (2008) onevaluation of ERP projects using DEA. Our results also supportthe inference drawn out in a previous work by Qin and Wang(2010), i.e., the defective ERP product will bring down thequality of the customised ERP and thus, in turn, its efficiency.The present study considers both the software productivitymeasures, i.e., the project efforts and the function points and thequality measure (defect counts) during the efficiency analysisof ERP projects. Thus, this study extends the work of Stensurdet al. (2003), Parthasarathy et al. (2008) and Qin et al. (2010).

et) Potential improvementrequired (in %)

Efficiency of ERP projects(Fi) (in %)

FP b PE a FP b

555 0 0 100832.50 −71.50 19.10 28.50

1348 −64.18 0 35.822219 −78.58 0 21.42599 −94.65 0 5.35

1908 −88.33 0 11.67499.50 −07.49 44.78 92.51

1080 −50.17 0 49.83482 −87.61 0 12.39

1130 −67.99 0 32.01




The results show that an increased understanding of theeffort estimation and its relationship to the size of the softwareand, in turn, the defects arising out of the software are essentialfor the managers to deliver an efficient ERP product. We havealso found that 2 of the 10 ERP projects are efficient and 6 ofthe remaining 8 projects have the potential for improvement byminimising their efforts. The project managers can react to thisscenario by carefully predicting the efforts for their future ERPprojects and bringing the defect measurement processes into theloop as suggested in the quality models of the efficient ERPprojects dotted in this study.

We offer an interesting practical implication from this study.That is, the project managers, the developers and the consultants ofthe ERP application development need to weigh all of the elements(efforts, FP and defect counts) as they change over time, in order toensure the efficiency of ERP. This is feasible if the projectmanagers recommend the implementing enterprise to considertheir pre-configured ERP instead of opting for customisation. Thiswould help them to keep their project efforts and FP intact and alsomanage the defect counts as part of their quality management.Reconciling all the diverse requirements from the customer andincorporating the same in the ERP will certainly require additionalefforts and review of the related quality processes, thereby delayingthe release of the customised product.

Many researchers and practitioners have observed that it iseasier and less costly to mould business processes to ERP systemsrather than vice versa (Luo and Strong, 2004; Parthasarathy andDaneva, 2014). When the implementing organisation chooses toinstal the pre-configured ERP system without customisation, thenthey should be prepared to customise their business processes to fitthe ERP system. This approach is referred to as ‘ProcessCustomisation’ in the ERP literature (Luo and Strong, 2004). Ifthe organisation fails to customise its business processes, but stillimplements the pre-configured ERP system and requests forsystem customisation during the post ERP implementation phase,then the project efforts required for such projects would be higherand the defect counts of the resulting customised product mightalso be significantly higher.

The practical benefit of this study is that the projectmanagers of the projects (say, B2 to B6 and B8 to B10) canidentify which projects (B1 or B7) they ought to consult andperhaps adopt their quality processes in order to improve theefficiency of their project when they upgrade their ERP infuture. We see a reasonable stability in our results and declarethat the study has revealed that the two projects (B1 and B7) aresuitable role models that are worthwhile being studied byproject managers of other, less efficient projects. The managersalong with their software quality assurance team shall utilise theresults of this study to review and map their project's effortestimation procedure, quality model in practice, and its relatedmanagement processes with that of what is being followed inthe efficient projects outlined in this study.

6.2. Evaluation of validity concerns

ERP software is typically developed and delivered as an ITsolution to the business problems that enterprises experience in


using proprietary software (Light, 2005). We deem thisresearch work as preliminary and a first step in analysing theefficiency of ERP projects that considers both the softwareproductivity and the quality measures. We believe that it wouldprovide a better understanding of the scope of the projectefforts, function points and defect counts in driving theefficiency of ERP projects.

As suggested by Yin (2008), we did the evaluation ofvalidity threats and presented below:

• There is a possibility for measurement errors in efforts, FPand defect counts (Boehm, 1981). This study has nothandled the errors in the datasets of 10 ERP projects for theI/O units. The procedure for estimation of these units mayalso vary from one IT firm to another.

• We observe that there could be errors in the DEA CRS modelused in this study. The software productivity measure, i.e., FP,is independent of the compressed project schedule and qualityrequirements (reliability, usability, portability, etc.). Hence asoftware project might yield a low FP score; however, it couldhave used more efforts to deliver the product in a short span aswell as to meet all the quality requirements. From a previousstudy (Stensurd and Myrtveit, 2003), we also record that themodel errors when applying the DEA for ERP projects aresmaller when compared to other conventional IT projects.

• The DEA CRS model is widely used in software engineeringresearch (for small or medium software projects) withoutany consideration for scale. When using the DEA, the scalecan be either CRS or VRS. A previous study by Stensurdand Myrtveit (2003) argues that the VRS best fits thelarge-scale IT projects such as ERP and another study byCharnes et al. (1978) states that for small or mediumsoftware projects, the CRS is well suited. As the presentstudy considers the projects that involve only a module ofERP and not the complete ERP package, our choice of CRSfor this study is found valid. However, when we extend thiswork to apply the DEA for 10 ERP projects that deal withthe implementation of a complete ERP package (withvarious ERP modules), then the DEA VRS might be themodel for efficiency analysis and accordingly the resultswould vary.

• The optimisation mode (MIN-IN or MAX-OUT) for the DEACRSmodel is another criterion to be decided by the researcherbefore commencing the execution of the model. In this study,we consider the project efforts as input, the FP and the defectcounts as the outputs. Hence, we chose the optimisation modefor this study as MIN-IN. We seek to minimise the effortswhile keeping the outputs constant and, in so doing, weevaluate the efficiency of ERP. The optimisation mode issubject to change depending on the input and output units wechoose for the DEA model. Accordingly, the results and itsinterpretation would certainly change.

• Finally, we would like to note that we do not plan to executean efficiency analysis of ERP projects involving bothstandard as well as customised product implementation.We are aware that the ERP projects often involve businessprocess or system customisation during the implementation




(Luo and Strong, 2004). The datasets of standard andcustomised ERP projects might give different results whenanalysed for their efficiency using the DEA model. We havenot extracted the datasets of 10 ERP projects used in thisstudy from a customisation perspective. This forms ourimmediate future research agenda.

7. Conclusions

This studywasmotivated to fill a void in the literature regardingthe efficiency of ERP projects from a software quality perspective.As the Indian IT industries have shifted their focus to the ‘quality’measures and management, the understanding of the efficiency ofERP from a quality perspective demands greater attention.Research studies in related areas such as effort estimation forERP projects, business process and system customisation, criticalsuccess and failure factors of ERP implementation, and softwareproductivity-based evaluation of ERP projects on IS offer littleguidance to the project managers aspiring to deliver efficient ERPproducts. Given these shortcomings, we have executed theefficiency analysis of ERP projects from a quality perspectiveusing the DEA CRS model.

To summarise, the efficiency analysis of ERP projects basedon software productivity (efforts and function points) andquality measures (defect counts) recognises an ERP project asan efficient one, if it has equilibrium between its productivityunits and also possesses low defect counts. In this study, fewERP projects with such characteristics are found to have highefficiency scores and a couple of other projects possess onlyaverage efficiency scores.

We suggest that the project managers involved in ERP projectsshould consider executing such efficiency analysis fulfilling thefollowing guiding principles: First, identification of the inputs andoutputs should be done carefully after brainstorming comprisingsoftware engineering and IS researchers. Second, the results of theefficiency analysis must be taken into confidence by the projectmanagers only as an initiative towards the software process andthe quality process improvement and not for radical redesign ofthe complete software development processes and practicesalready in place in the IT firms.

7.1. Future research

It is difficult to identify suitable factors for the efficiencyanalysis of software projects, and it may ultimately demandrelating an apple with a pear. It is a fact that an efficiencyanalysis of a software project should include its softwareproductivity as well as quality measures (say, defects counts)apart from other external factors such as schedule constraintsand risk parameters. Previous studies (Boehm, 1981; Brooks,1995) have observed that the software projects with com-pressed schedules require more effort to develop the very samesize of the software. Future research should investigate theefficiency assessment of software projects based on theirproductivity, quality measures, schedule and risks.

From a practitioner's point of view, future research shall createa benchmarking database consisting of accurate datasets of


various efficient ERP projects, thereby facilitating the projectmanagers to adopt the best software process model and qualitymodel to perform accurate effort estimation from past projectsand create a schema for dynamic risk assessment and manage-ment at an early stage of their project.

Conflict of interest

There is no conflict of interest to declare.

References

Afriat, S.N., 1972. Efficiency estimation of production functions. Int. J. Econ.Rev. 13 (3), 568–598.

Agarwal, M., Chari, K., 2007. Software effort quality and cycle time: a study ofCMM level 5 projects. IEEE Trans. Softw. Eng. 33 (3), 145–156.

Al-Mashari, M., Al-Mudimigh, A., Zairi, M., 2003. Enterprise resourceplanning: a taxonomy of critical factors. Eur. J. Oper. Res. 14 (1), 352–364.

Aversano, L., Tortorella, M., 2013. Quality evaluation of floss projects:application to ERP systems. Inf. Softw. Technol. 55 (7), 1260–1276.

Aydin, A.O., 2012. A new way to determine external quality of ERP software,computational science and its applications — ICCSA 2012. Lect. NotesComput. Sci 7336, 186–201.

Basili, V., Zelkowitz, M.V., 2007. Empirical studies to build a science ofcomputer science. Commun. ACM 50 (11), 136–144.

Boehm, B.W., 1981. Software Engineering Economics. Prentice-Hall, Engle-wood Cliffs, New Jersey.

Botta-Genoulaz, V., Millet, P.A., Grabot, B., 2005. A survey on the recentresearch literature on ERP systems. Comput. Ind. 56 (6), 510–522.

Brooks, F.P., 1995. The Mythical Man-Month-Essays on Software Engineering.Addison Wesley, Longman, Massachusetts, USA.

Carmel, E., Agarwal, R., 2002. The maturation of offshore sourcing ofinformation technology work. MIS Q. Exec. 1 (2), 65–78.

Charnes, A., Cooper, W.W., Rhodes, E., 1978. Measuring efficiency ofdecision making units. Eur. J. Oper. Res. 2 (6), 429–444.

Chen, C.C., et al., 2009. Managing ERP implementation failure: a projectmanagement perspective. IEEE Trans. Eng. Manag. 56 (1), 157–170.

Conte, S.D., Dunsmore, H.E., Shen, V.Y., 1986. Software Engineering Metricsand Models. The Benjamin Cumming Publishing Company, Michigan, USA.

Daneva, M., 2010. Balancing uncertainty of context in ERP project estimation: anapproach and a case study. J. Softw. Maint. Evol. Res. Pract. 22 (5), 329–357.

Davis, G.B., EinDor, P., King, W.R., Torkzadeh, R., 2006. IT offshoring:history prospects challenges. J. AIS 7 (11), 770–795.

Doyle, J., Green, R., 1994. Strategic choice and data envelopment analysis:comparing computers across many attributes. J. Inf. Technol. 9 (1), 61–69.

Fisher, D.M., Sun, D.B., 1996. LAN-based e-mail: software evaluation. J.Comput. Inf. Syst. 36 (1), 21–25.

Fisher, D., Kiang, S., Fisher, R.T., Chi, 2005. Evaluating mid-level ERPsoftware. J. Comput. Inf. Syst. 45 (1), 38–46.

Hwang, Y., Grant, D., 2011. Understanding the influence of integration on ERPperformance. Inf. Technol. Manag. J. 12 (3), 229–240.

Jalote, P., 2000. CMM in Practice. Addison Wesley, Longman, Massachusetts,USA.

Jalote, P., 2008. A Concise Introduction to Software Engineering. Springer-Verlag, London.

Kan, S.H., Basili, V.R., Shapiro, L.N., 1994. Software quality: an overviewfrom the perspective of total quality management. IBM Syst. J. 33 (1), 4–19.

Kannabiran, G., Sankaran, K., 2011. Determinants of software quality inoffshore development— an empirical study of an Indian vendor. Inf. Softw.Technol. 53 (11), 1199–1208.

Koch, S., 2007. ERP implementation effort estimation using data envelopmentanalysis. Technologies for Business Information Systems. Springer,Dordrecht, pp. 121–132.

Krishnan, M.S., Kriebel, C.H., Kekre, S., Mukhopadhyay, T., 2000. Anempirical analysis of productivity and quality in software products. Manag.Sci. 46 (6), 745–759.


http://refhub.elsevier.com/S0263-7863(14)00168-9/rf0010
























































Lall, V., Teyarachakul, S., 2006. Enterprise resource planning (ERP) systemselection: a data envelopment analysis (DEA) approach. J. Comput. Inf.Syst. 47, 123–127.

Levine, D.M., Berenson, M., Stephan, D., 1999. Statistics for Managers.Prentice Hall, New Jersey.

Light, B., 2005. Potential pitfalls in packaged software adoption, numerousorganizational considerations influence packaged software purchasing.Commun. ACM 48 (5), 119–121.

Luftman, J., Kempaiah, R., 2008. Key issues for IT executives 2007. MIS Q.Exec. 7 (2), 99–112.

Luo, W., Strong, D.M., 2004. A framework for evaluating ERP implementationchoices. IEEE Trans. Eng. Manag. 51 (3), 322–332.

Mayrhauser, A., Wohlin, C., Ohlsson, M., 2000. Assessing and understandingefficiency and success of software production. Empir. Softw. Eng. 5 (2),125–154.

Myrtveit, I., Stensrud, E., 1999. A controlled experiment to assess the benefitsof estimating with analogy and regression models. IEEE Trans. Softw. Eng.25 (4), 510–525.

Nozdrina, L., 2009. Applying of fuzzy logic modeling for the assessment ofERP projects efficiency. Proc. 5th Int. Sci. Conf. Project Management:Status and Opportunities, 1–2, NUS, Nikolaev.

Parthasarathy, S., 2011. Performance enhancement model for enterpriseresource planning (ERP) projects. Int. J. Inf. Syst. Chang. Manag. 5 (2),140–159.

Parthasarathy, S., 2012. Research directions for ERP projects. Int. J. Bus.Inform. Syst. 9 (2), 202–221.

Parthasarathy, S., Anbazhagan, N., 2008. Evaluating ERP projects using DEAand regression analysis. Int. J. Bus. Inf. Syst. 3 (2), 140–157.

Parthasarathy, S., Daneva, M., 2014. Customer requirements based ERPcustomization using AHP technique. Bus. Process. Manag. J. 20 (5),730–751.

Paschalidou, G., et al., 2013. An application of data envelopment analysis tosoftware quality assessment. Proceedings of the 6th Balkan Conference inInformatics, ACM, 228–235, 2013, New York, USA.

Pressman, S.R., 2010. Software Engineering — A Practitioner's Approach.McGraw-Hill International Edition, India, pp. 472–475.

Qin, Z., Wang, D., 2010. Quality prediction modeling for softwarecustomization in the absence of defect data. Int. J. Public Policy 6 (3–4),336–358.


Rajendran, C., Issac, G., Anantharaman, R.N., 2006. An instrument for themeasurement of customer perceptions of quality management in thesoftware industry: an empirical study in India. Softw. Qual. J. 14, 291–308.

Ray, S.C., 2004. Data Envelopment Analysis: Theory and Techniques forEconomics and Operations Research. Cambridge University Press, NewYork.

Rosa, W., et al., 2013. COTS integration and estimation for ERP. J. Syst. Softw.86 (2), 538–550.

Rothenberger, M.A., Srite, M., 2009. An investigation of customization in ERPsystem implementations. IEEE Trans. Eng. Manag. 56 (4), 663–676.

Rothenberger, M.A., Yi-Ching Kao, L.N., Wassenhove, V., 2010. Total qualityin software development: an empirical study of quality drivers and benefitsin Indian software projects. Inf. Manag. 47 (7–8), 372–379.

Sarrab, M., et al., 2014. Empirical study of open source software selection foradoption, based on software quality characteristics. Adv. Eng. Softw. 69,1–11.

Shaul, L., Tauber, D., 2013. Critical success factors in enterprise resourceplanning systems: review of the last decade. ACM Comput. Surv. 45 (4),1–39.

Stefanou, C.J., 2001. A framework for the ex-ante evaluation of ERP software.Eur. J. Oper. Res. 10, 204–215.

Stensurd, E., Myrtveit, I., 2003. Identifying high performance ERP projects.IEEE Trans. Softw. Eng. 29 (5), 398–415.

Subramanian, G.H., James, J.J., Klein, G., 2007. Software quality and IS projectperformance improvements from software development process maturityand IS implementation strategies. J. Syst. Softw. 80 (1), 616–627.

Sudhaman, P., 2011. Evaluating software quality using software metrics for asoftware product. Int. J. Inf. Syst. Chang. Manag. 5 (2), 180–190.

Teltumbde, A., 2000. A framework for evaluating ERP projects. Int. J. Prod.Res. 38 (17), 4507–4520.

Thore, S., Phillips, F., Ruefli, T.W., Yue, P., 1996. DEA and the managementof the product cycle: the U.S. computer industry. Comput. Oper. Res. 23(4), 341–356.

Tian, J., 2004. Quality-evaluation models and measurements. IEEE Softw. 21(3), 84–91.

Wohlin, C., 2000. Experimentation in Software Engineering. Springer, Berlin.Yin, R., 2008. Case Study Research. Sage, Thousand Oaks, CA.















































































Documents

Efficiency analysis of ERP projects—software quality perspective