ERP Software Evaluation and Comparative Analysis...198 ERP Software Evaluation and Comparative Analysis We believe that three weight values are suffi-cient: 1 – basic functionality,

Journal of Computing and Information Technology - CIT 12, 2004, 3, 195–209 195

ERP Software Evaluationand Comparative Analysis

Kresimir Fertalj and Damir KalpicFaculty of Electrical Engineering and Computing, University of Zagreb, Croatia

This paper presents the results of an investigation per-formed in 2001 under the title Comparative Analysis ofInformation Systems Software in Croatia. The focus wasset on the comparative analysis of domestic and foreignEnterprise Resource Planning �ERP� software, which ispresent in Croatia. The investigation was performedfrom the standpoint of ERP applicability, regardless ofthe development methods and information technology.In other words, the evaluation was performed primar-ily from the standpoint of users rather than designers,programmers or other persons engaged in the systemdevelopment and implementation.

System evaluation was performed in several phases andin multiple steps. A general list of relevant ERP char-acteristics was established first. This initial list wasupdated in co-operation with ERP suppliers and users.They completed the list by adding the characteristicsthey regarded as important or accentuated some featuresof their solutions that had not been mentioned initially.System evaluation was performed at the users’ sites,having insight to real applications. To increase theobjectivity and accuracy, the evaluating teams consistedof persons of different profile �independent field experts,e.g. an accounting expert, IT expert, end-user etc.�. Inspite of the attitude taken not to evaluate the concretebasic technology, some estimation of the computingarchitecture and functionality was performed, when itwas found relevant for the estimation of applicability.

Keywords: ERP software, software capability evalua-tion, software comparison.

1. Introduction

ERP system is an integrated information sys-tem to support the business within differentorganisational parts of an enterprise. Inter-national Data Corporation �IDC� defines theERP software as support for at least three outof the following four business segments �7�:Accounting, Manufacturing, Material manage-ment�distribution, Human resources manage-ment with payroll.

The termERPprimarily relates to customizationand usage of a “ready-made” software pack-age �in further text “ERP package”�, and onlysecondarily relates to software written on pur-pose for a specific customer. The leading globalproviders are SAP, ORACLE, BAAN, People-Soft and J. D. Edwards. ERP packages are de-signed to be customizable to concrete needs ofan organization and to its legacy systems. Thecustomization of ERP modules for a concreteuser is performed with special tools and usingspecific or standard programming languages.Due to already mentioned general ERP char-acteristics, customization and implementationusually require specialised knowledge and sig-nificant resources �6�. Practical experience andliterature agree that this is the crucial part forsuccess �21�.

This paper presents the results of an investiga-tion performed in 2001, under the title Compar-ativeAnalysis of Information Systems Softwarein Croatia. It received a grant by the CroatianMinistry of Science and Technology. �GRANTNo. 00–146�. The focus was set on the com-parative analysis of domestic and foreign Enter-prise Resource Planning �ERP� software, whichis present in Croatia. The aim was to determine:

� the actual contents which is being offeredwithin different solutions,

� the amount of necessary and sufficient in-vestments,

� difficulties and problems connected with im-plementation of ERP solutions,

� possible effects of application of the imple-mented solutions,

196 ERP Software Evaluation and Comparative Analysis

� the position of domestic solutions comparedto the foreign ones, opportunities of Croatiain this field.

The results of investigation are supposed to in-form potential and actual ERP customers aboutthe situation on the market, estimated costs ofpurchase, implementation, further developmentand maintenance. The ERP suppliers should getthe information regarding the desirable charac-teristics of these systems, as seen through users’eyes. The final purpose is to give the indus-try and administration answers to the followingquestions:

� What can actually be obtained from the so-lutions available on the market?

� What are the expected costs?

� What development strategies should be cho-sen?

It is understandable that our investigation paidsubstantial attention to the existing proprietaryERP solutions developed in Croatia. For theinternational reader, it may be of less interest,except if regarding Croatia as a paradigm for asmall, medium-developed country that has re-cently passed through an ugly war and not muchbetter privatisation. Although Croatia is betteroff than most developing countries of the oncecalled Third world, it is plagued by some com-mon troubles as those countries: weak legalsystem and moderate corruption. Both theseweaknesses surely affect the local ERP market.

2. Method for ERP System Evaluation

ERP systems are large and complex, so theirevaluation is also a very complex task. Whatis more, there are no generally accepted evalu-ation methods. Some widely accepted methods�18��19� relate to the evaluation of the softwaredevelopment within an organisation, or con-centrate on the choice of supplier of services.Some other investigations performed in differ-ent countries were mainly aimed to analyse themarket �3��7��8��9�.

Therefore, for the sake of this investigation, aproprietary method for evaluation and compari-son of ERP systemswas defined and elaborated.Essential differences to the known existing pro-cedures are the following:

� Definitions of desirable system characteris-tics are updated in co-operation with the sys-tem suppliers and users alike.

� Evaluation is performed by having insightinto the real-life system, rather than by ap-plying questionnaires.

Independent referees perform the evaluation inco-operationwith the staff,well acquaintedwiththe evaluated software. In this way, higher ob-jectivity is achieved than by filling-in question-naires where the suppliers or users “evaluatethemselves”. Disadvantage of this approach isthe increased time required to evaluate a largernumber of systems. Practical experience hasshown that the evaluation of a single subsystemmay take a couple of hours, while to evaluatea whole system can take a week �4–5 workingdays�.

2.1. Analysis of the Market and Applicationof ERP Systems

The preparatory phase consists of:

� Analysis of the world ERP market,

� Identification of ERP package suppliers,

� Identification of domestic ERP package pro-ducers,

� Identification of the existing ERP customers,

� Establishing the co-operation with ERP dis-tributors and ERP customers willing to par-ticipate in the evaluation.

� Evaluation planning.

2.2. Definition of the Domain of Analysisand the Working Procedure

In parallel with the market analysis and locationof ERP applications, the definitions are formu-lated for:

� Desirable subsystems and business functions�ideal ERP system�

� System components to be evaluated,

� Desirable characteristics for evaluation ofsingle ERP components,

� Procedures for evaluation and analysis of re-sults.

ERP Software Evaluation and Comparative Analysis 197

Desirable characteristics and procedures are pre-sented to the suppliers and users of ERP sys-tems. They are encouraged to criticise and theyare asked to present their opinion and sugges-tions to complete the initial material. If duringthe investigation new relevant characteristics areencountered, they can be added and evaluated.

In other words, the method can be dynamicallyadapted and enhanced, with the aim that theoutcome be the result of the overall knowledgeand experience of all stakeholders. To avoidpossible contradictory requirements for change,originating from different suppliers and ERPcustomers, the investigation team withholds theright to accept or dismiss the suggestions.

2.3. Evaluation Procedure

During the evaluation phase:

� Evaluation teams for specific business do-mains are formed,

� The evaluation plan is elaborated and up-dated for ERP systems and analysed do-mains,

� Analysis of existing ERP installations is per-formed.

Evaluation teams consist of a few experts foreach analysed business domain. It is desirablethat they are experienced users of software inthis field, but they need not be computing ex-perts, because only the user’s aspect is analysed.It is important that they are experts in the busi-ness domain and to increase the evaluation ob-jectivity it is desirable that they originate fromdifferent working environments.

The evaluation is performed within an organi-zation with installed ERP. If for some ERP so-lution such evaluation is not possible, the eval-uation can proceed through an insight into theevaluation copy of the software.

The supplier or the customer nominates the“best user”, meaning the person who is mostfamiliar with the system capabilities. This userbriefly presents the system capabilities, stress-ing its advantages and features. After preparingthe list of desirable characteristics the evalua-tors ask questions and write down the answers.They give positive marks to those characteris-tics that can be demonstrated on the computer.If, for any reason, it turns out to be impossible,

the evaluation proceeds based upon the avail-able user and�or program documentation. Theresults and possible additional notes are veri-fied and forwarded to the investigation team foranalysis.

2.4. Evaluation of Capabilities

For the purpose of evaluation, to each of thecapabilities a range of possible marks is ad-joined. The lowest mark is zero, denoting thatthe corresponding capability is not present. Thehighest mark is set depending on the desiredgranularity of results. For each capability, itsrespective importance is quantified by a weight.The lowest weight is unit, while the upper limitalso depends upon the desired granulation of theanalysis results.

The mark for a system is calculated after theformula:

O = Σsi/ mi* fi, for each capability i, where:

si is the mark for the capability i, mi is the max-imum possible mark for the capability i and fiis the weight of the capability i.

In the case when the highest mark for each ca-pability equals 1 and if all the weights are equalto 1, the system evaluation reduces to the sumof capabilities present in the system.

The relative mark for a system X, in proportionto the ideal system 0, is calculated as

RX= OX/ O0* 100

An example for possible marks is:

0 – capability is not present,

1 – partly present, available as option,

2 – capability is present,

3 – additional features present,

4 – especially useful capability or a significantinnovation.

It ismore difficult to define theweights, becausethe same capability does not have the same im-portance for various users. It should be left toeach supplier and each customer to attach theweights and to use the arithmetical average inthe analysis. Alternatively, weights may be pro-portional to the complexity required to achievethe respective capability. However, the impor-tance of a capability for the user should not beneglected.


We believe that three weight values are suffi-cient:

1 – basic functionality,

2 – advanced functionality,

3 – sophisticated functionality.

Regardless of the ranges and weights applied,the analysis and ranking of results can proceedon the level of single components �subsystems�and on the system level, because the capabilitiesare grouped around various domains.

2.5. Analysis of Results

In the last phase:

� A comparative analysis of the gathered re-sults is performed,

� A rank list of ERP solutions is produced,

� The gathered information is synthesized andconclusions are deduced.

2.6. An Abridged Evaluation Process

The evaluation by direct insight, if applied fora larger number of systems, might request anexcessive engagement of human and financialresources and take too much time. Therefore,an abridged approach is also envisaged.

The method can be modified by decreasing thedepth of analysis. For the purpose of evaluation,shortened lists of capabilities are checked. Fur-ther simplification evaluates groups of businessfunctions or whole subsystems. Finally, onlythe existence of groups of functions or subsys-tems can be checked. Naturally, the quality ofanalysis decreases with simplifications.

3. Results of the Comparative Analysis

3.1. Analysed Systems and Components

The call for co-operation within Croatia wasaddressed to 14 ERP package suppliers and de-velopers and to 10 ERP customers who had pur-chased an ERP solution or had developed a pro-prietary system. For the purpose of evaluation,the worldwide known ERP packages present inCroatia were selected. Selection of domestic

ERP products was based on the previous know-ledge of the investigation team and completedby references of some IT companies.

The initial response rate to our call for co-operation was 63%, where 38% participated inpreliminary phases of investigation, but onlya quarter of them agreed to participate in theevaluation process. It is interesting to mentionthat the initially high percentage �63%� of ERPpackage suppliers willing to co-operate, signif-icantly dropped during the investigation. At theend, only five systems were analysed: three do-mestic ones, where two of them had previouslybeen awarded for good quality, and the two lead-ing ERP packages on the world market. Somesubsystems were not analysed or not analysedin details, partly because they were not imple-mented, but also because of limited investiga-tion resources and lack of time. A comparativeanalysis was performed for the following ERPpackages, listed in alphabetic order:

� BAAN, by the BAAN Company, Herndon,VA, USA, at the customer who had been us-ing it for 4 years,

� IS in the company LURA GROUP, Croatia,a proprietary solution,

� SAP, by the SAP A.G., Waldorf, Germany,at the customer who had been using it for 5years,

� ERP package by the company ENEL d.o.o.,Split, Croatia,

� SUPERby the companySUPER-KINGd.o.o.,Zagreb, Croatia, installation at a representa-tive customer.

We were aware of the diversity of moduleswithin commercially available ERP packagesand of the diversity in the organization of thoseERP subsystems that were developed on pur-pose, for known customers. We defined thedesirable capabilities, for the components. Theevaluationwas completed for the following com-ponents:

� General characteristics,

� Accounting,

� Manufacturing monitoring,

� Projects monitoring,

� Sales,

� Purchase,


� Stocks management,

� Manufacturing management,

� Quality control.

For the following components, the evaluationproceeded only partly, i.e. only the existence ofcomponents was checked:

� Project management,

� Maintenance management,

� Human resources with payroll.

Three of the evaluated systems comprised allthe components, while the rest contained Ac-counting, Sales, Purchase, Stocks managementand Human resources with payroll. All the par-ticipants in the investigation had been informedin advance about the evaluation process and re-ceived the evaluation forms to be better preparedand to have their say regarding the proposed listsof capabilities. Members of the Croatian Ac-countants Association joined the investigationteam and helped to create the evaluation formsduring the preparatory phase.

Respecting the fact that some values may bea confidential, in our call we proposed thatthe concrete customer and supplier informationbe hidden, or revealed only after their writtenpermission. Until the end of the investiga-tion, only a domestic company SUPER-KINGalowed revelation of its name. All other partici-pants withheld their permissions. We apologise

to our readers for having the permission to re-veal only one of the analysed systems, while therest must remain coded. This is partly under-standable, because, as stated before, our investi-gation resources were scarce and, therefore, ourconcrete findings could even become a subjectof litigation.

3.2. Summary of Subsystems Evaluation

Table 1 and Figure 1 compare the counts ofcomponents within subsystems that were eval-uated for capabilities. The last column �Ideal�contains the maximum count of capabilities foreach component, representing an ideal system.The last row �Relative system mark� containsthe percentage of implemented capabilities inrespect to the ideal system.

The systems coded as 10100, SUPER, 10400have all the components, while for the system10400 three of them were not evaluated due toobjective reasons. Only the systems 10100 andSUPER contain more than half of all desirablecapabilities. The system SUPER achieved thebest marks and the system 10100 follows. Thesystem 10400 is a complete set of average valu-ed components. The system 10200 achievedvery good marks for the components it contains.The system 10500 definitely requires a seriousredesign.

SUMMARY MARKS SystemComponent 10100 10200 SUPER 10400 10500 IdealAccounting 67 96 120 58 46 127Quality control 11 0 23 0 0 23Purchase 16 31 37 18 15 43General characteristics 32 20 58 25 14 61Sales 46 51 66 18 21 67Manufacturing management 105 0 168 88 0 173Stocks management 22 17 32 12 10 34Manufacturing monitoring 13 0 12 0 0 14Projects monitoring 15 0 13 0 0 15Summary system mark 327 215 529 219 106 557Relative system mark 58.71% 38.60% 94.97% 39.32% 19.03% 100%

Table 1. Summary marks of ERP packages.


Figure 1. Summary marks of ERP packages.

3.3. Relative Mark Values

Table 2 and Figure 2 represent the relative pres-ence of capabilities compared to the ideal sys-tem. The last column contains the average valueof the examined ERP systems. The last rowpresents the average components mark.

On average, Accounting and Sales are best sup-ported. Purchase and Stocks management fol-low the suit, but their completeness is onlyslightly above 50%. General characteristicshardly reach half of the Ideal system. Supportsto the Manufacturing and to the Quality controlare the weakest points.

RELATIVE MARKS SystemComponent 10100 10200 SUPER 10400 10500 AverageAccounting 52.76% 75.59% 94.49% 45.67% 36.22% 60.94%Quality control 47.83% 0.00% 100.00% �4.35% 0.00% 30.43%Purchase 37.21% 72.09% 86.05% 41.86% 34.88% 54.42%General characteristics 52.46% 32.79% 95.08% 40.98% 22.95% 48.85%Sales 68.66% 76.12% 98.51% 26.87% 31.34% 60.30%Manufacturing management 60.69% 0.00% 97.11% 50.87% 0.00% 41.73%Stocks management 64.71% 50.00% 94.12% 35.29% 29.41% 54.71%Manufacturing monitoring 92.86% 0.00% 85.71% �7.14% 0.00% 37.14%Projects monitoring 100.00% 0.00% 86.67% �6.67% 0.00% 38.67%Average component value 64.13% 34.07% 93.08% 28.86% 17.20% 47.47%

� For the system 10400 only the existence of subsystem was taken into account.

Table 2. Relative marks of ERP packages.


Figure 2. Relative marks of ERP packages.

4. Summary of the Analysis of the LeadingForeign ERP Packages

After a detailed analysis of the ERP packagesused in Croatia �SAP and BAAN� and after ac-quiring a partial insight into the software andavailable documentation for the packages byORACLE, J. D. Edwards and PeopleSoft andafter an additional review of the literature pub-lished worldwide, we came to the followingconclusions:

� ERP packages are technologically modern,comprehensive and reliable software.

� They cover many business domains.

� They support more than a thousand busi-ness functions or working procedures.

� They support many variants of businessprocedures.

� They support the operational parts of busi-ness procedures, i.e. everyday transac-tions which generate data.

� They are based on modern technologies�multy-tier client�server architecture,OLE, HTML, SMTP, MAPI, EDI,ALE,...�.

� Although there are sporadic objections totheir reliability, they perform at least so

well as the proprietary customer designedsolutions.

� ERP packages are sets of interconnected par-tial solutions, rather than integrated solu-tions.

� They had initially been developed to sup-port a certain function �e.g. Accounting,Manufacturing� and other modules wereadded afterwards.

� Single modules or subsystems had beendeveloped in different software compa-nies and were acquired by the owner ofthe “integral” ERP package later. Theseacquired subsystems are interconnectedin differentways �commondatabase, datatransfer, and even manual data entry fromone module to the other�.

� Within single subsystems, the variants forsomebusiness functions coexist as partialsolutions, mostly without any generalisa-tion.

� Functional quality and sophistication of im-plemented solutions is mediocre.

� They principally support everyday trans-actions with a low level of analytic re-porting.


� Sophisticated functions, like optimiza-tion or simulation, are mostly absent andthey are substituted by combinations ofmore complex queries and reporting.

� ERP packages are very large and inefficient.

� The software dwells over huge databasesand the program code is huge. For ex-ample, SAP handles 14,000 tables, thereare 650 fields to describe a material, andPeopleSoft contains 14,000 predefinedscreen forms.

� An average customer uses only a smallfraction of the features available, so themajor part of the system is a useless bur-den in application and maintenance.

� Software was principally designed for ef-ficient transaction processing and simplereporting. Due to the large number ofhighly normalized tables, more sophisti-cated reporting is resource consuming.

� ERP packages are complex and rigid for im-plementation and maintenance.

� Implementation and customization requirenon-standard fourth generation languagesand specialised routines for transactionhandling.

� The systems have been designed as alarge set of predefined functions, ratherthan easily customizable solutions.

� Accordingly, the customer is sometimesforced to change the organization andbusiness procedures to adapt to the soft-ware.

� The systemadministration requires know-ledge of a large number of parametersand procedures, which are described inmanuals of more than thousand pages.

� Customization and implementation are timelyand complex. They comprise the followingactivities:

� Definition of the customer’s informationrequirements.

� Definition of input and output for the in-formation system.

� Identification of those embedded compo-nents, which are useful to the customer.This is usually a small fraction of the

package. Retrieving of the desired datastructure and desired function is difficultbecause of the large number of prede-fined items.

� Design and customization of the data-handling screen forms.

� Implementation of advanced reporting,because only simple ones are at hand �24�.

� Lack of the basic reporting system can besolved in the following ways:

� ERP database is used and only report-ing modules are added. Reporting slowsdown the system and assignment of lowerpriority to reporting can be a remedy.An example is SNAPpack by InformationBuilders, New York, applied for report-ing from the leadingERPpackages �SAP,J.D. Edwards and PeopleSoft�.

� ERP database is copied on another me-dium and the reporting proceeds fromthere. For example, EMC Corp., Mas-sachusetts, offers a hardware and soft-ware solution called Time Finder, to cre-ate a copy of the database on a separateserver. It is used for reporting, but alsoas a backup for the database.

� Data from the ERP base are written into adifferently structured new database. It isthe most common Data Warehousing ap-proach where usually separate hardwareis used.

� Combination of the above solutions,where some reports are generated by ad-ditional programs from the ERP databaseand some other come from a separate sys-tem containing the copy. An example isCrystal Decisions, by Seagate Software,which generates reports from ERP sys-tems and from theDataWarehousing sys-tems.

� Regardless which of the above proce-dures is applied, copying of ERPdata intospreadsheets �e.g. MS Excel� is mostusual.

� ERP packages are not a stable basis to de-velop an information system:

� Soon after the purchase and at the lat-est after three years, the pressure of thesupplier to buy a new version intensifies.


It is sometimes enforced with the threatthat any support to the existing versionwill cease otherwise �13�.

� Due to all the considerations enumeratedabove we find that the mentioned, worldrenown ERP packages do show some seriousdisadvantages:

� The structure of these systems had notbeen designed on purpose, it had “hap-pened” through time, as sediments of �ac-quired� subsystems and partial solutions.

� They are difficult to understand, to im-plement and to maintain �6�.

� They require constant consulting.

� They excessively consume the comput-ing resources.

What a customer really gets with an ERP ac-quisition is the contents and structure of thedatabase and a large number of partial solutionsto support the operational functions, whichmay,but need not, comprise the solutions the cus-tomer really needs. The missing functional-ity is repaired with additional components �so-called concepts� like Customer Relations Man-agement �CRM�, Supply Chain Management�SCM�, Application Link Enable �ALE�, DataWarehousing and Data Mining. To the set of allthese partial solutions a preposterous name bestof breed is given. Unfortunately, instead of anintegral system, based on a solid concept, add-ons are applied leading to fragmentation, theyincrease complexity and inefficiency of the in-formation system.

� In spite of all the setbacks andmuch criticismaddressed to the famous ERP systems, ourinformal sources of information, which un-fortunately cannot be properly documented,lead us to a speculation that the market ac-cepts them for the following reasons:

� The number of predefined functions isso large that it leaves the potential cus-tomer with the impression that within thepackage “all has been solved”. However,much effort to reach full functionality andapplicability is required.

� ERP suppliers impress the potential cus-tomers with the numbers of existing in-stallations and reference lists with mostfamous world companies.

� Significant knowledge and experience hasbeen gathered and embedded during years-long existence of some ERP packages.

� The customer tries to compensate the lackof own IT personnel by hiring so called“Certified consultants” who in some ca-ses, informally but accountably reportedto us, achieve that title regardless of theprevious education and after only a shorttraining.

� In industrialized countries, the massively“produced”, meaning cheap, semi-quali-fied but highly specialised work forceprovides the customization of the pack-age fragments.

� As the initial investment represents abouta third of overall costs, the ERP costsmayappear lower than a proprietary develop-ment.

� Customers are exhausted by technologi-cal progress in IT and they like to delegatethe problem to ERP suppliers.

� While integrating islands of computerisa-tion, to avoid internal arguments whosesolution should prevail, an ERP packageis acquired as a neutral solution.

� If you buy the “world’s best” solution,you do not get fired.

� Some customerswho have dearly paid fortheir ERP do not dare to admit failure.

� In some developing countries, big invest-ment may imply big bribe.

� In developing countries, the loans fromhighly developed world mostly have tobe spent on purchases back from thesame highly developed world. There-fore, “world-renown” consulting com-panies are engaged who favour “world-renown”ERPproducts and “ready-made”solutions, regardless how low is the pos-sibility to implement them locally. Mostof the money returns to the highly de-veloped world, local elites enjoy somefraction of it, the project often fails andin the years to come the poor populationpays back the loan increased by interestrates. According to the Croatian dailypress, the need to rely on local expertise


in order to succeed has recently been ad-mitted by the World Bank’s President Mr.Wolfensohn.

5. Position of Local Solutions in Respect tothe Leading ERP Packages

We found that good proprietary ERP solutionsexist in Croatia. According to the results ofour investigation 2 out of 3 examined ERP sys-tems are comparable or in some �at least locallyappreciated� aspects even better than the worldrenown packages.

Logically, there are not many local ERP pack-ages on the market, especially having in mindthat development of such software takes hun-dreds of person-months, or even person-years.However, there are a few other good Croa-tian packages, which were not evaluated in thisstudy.

General characteristics of these locally pro-duced and favourably evaluated ERP packages:

� They are offered as complete solutionsencompassing complete customer require-ments, with prepared screen forms and re-porting systems.

� Practically no customization is required forthe implementation. Some organizationalmeasures and education of users are the onlyrequirement.

� They are compact and homogenous, becausethey do not consist of acquired and later in-terconnected components.

� They are implemented much easier and aremuch cheaper.

� They have a very weak marketing.

� They find it very difficult to follow the tech-nical advancements.

6. Opportunities for Improvement

6.1. Situation in Croatia

According to the available data and some previ-ous investigation, the situation can be describedin following statements:

� The majority of existing information sys-tems are technologically obsolete or do notcorrespond to the customer’s needs.

� After the initial boom, where most of thosewho had some money bought a renownedERP, the ERP market has shrunken. It canbe attributed to political changes, increasedcaution and lack of money �9�.

� The Croatian market potential is estimatedon 300–400 larger customers andmany smalland medium size �with 10 or more employ-ees� companies �9�.

Apart from the above-mentioned exceptions,the main characteristics of most other domes-tic products are:

� Incomplete solutions: Most domestic com-panies do not offer an integral solution. Usu-ally they are only partly integrated and coveronly certain domains, usually Accounting,Materials handling and Human resourceswith payroll.

� Inadequate solutions: Software developed10 years ago is obsolete now. Some domes-tic companies, specialised for domains likebanking or insurance, tend to widen theiractivity as outsourcers, regardless of their�in�competence in new domains.

Main setbacks of acquisition of foreign ERPpackages are:

� World market prices are too high for localconditions. Very high initial investments arenecessary. It would be fair that the foreignvendors adapt their pricing policies to the lo-cal per capita BDP. It might be not applicablefor the foreign consultants’ fees, but for theprice of licences, it should be feasible. Suchpolicy would reduce the differences in bene-fit of buying an ERP system between highlydeveloped countries on one side and thoseless developed on the other.

� Incompatibility of foreign solutions with lo-cal conditions, resulting in significant cus-tomisation and organizational efforts �20�.A hidden cost caused by higher restructur-ing efforts in less developed world remains,but it would be too much to expect from ERPvendors to cater for that.

� Big discrepancy between the price of the ba-sic product �30–40% of the overall costs�


compared to the customization costs �60–70% of the overall costs�.

� Too long period of implementation for a“ready-made” solution. It may take fromone to three years and after 3-4 years, a newdevelopment cycle is already due. Some im-plementations extend even longer and�or therequired functionality is never achieved.

� Long-term dependency upon the supplierand expensive consultancy. Fees of 1,200-1,800 USD per day are paid for a consultant,which exceeds a good local monthly salary.

� Occasional incompetence of �especiallysome foreign� consultants and instability ofsuppliers. Poor treatment of customers bythe ERP producers derives from the percep-tion that Croatia is a small and unimportantmarket where third class consultants can besent. On the contrary, more knowledge isoften required in such a country than in ahighly organized one.

Cost and Benefit

Depending on the field of their activity, poten-tial customers can count on some usual positiveeffects of computerization �12�:

� integration of financial data,

� standardization of human resource data,

� standardization of business processes,

� standardization of manufacturing processes.

In USA, an average investment in a projectamounts to 15 million USD, average end-usercosts are up to 53.320 USD and return of in-vestment can be expected after 31 months, atthe earliest �12�. Additional costs of supportafter implementation amount to 2,304 USD peruser in large organizations and up to 7,870 USDfor the small ones �15�. It is not always easy todetermine whether the investment in computer-ization pays off �14��16��22�, but there is a gen-eral awareness that a company not investing inits information system would be out of businessvery soon.

Domestic software is much cheaper. A repre-sentative product can be bought for 2.350 USDper workstation, with one-year guarantee andsupport. General cost benefit analysis was not

possible, because the necessary data have notbeen revealed.

It is nearly impossible to estimate in advancesome hidden costs like:

� educational costs including the adaptation ofusers to new working ways,

� integration and testing, which is difficult toforesee, depending upon the connectionwithlegacy systems,

� data conversion often represents an unpleas-ant surprise, due to un-normalized relationsor inadequate codification in legacy systems,

� data analysis becomes costly if the ERP dataneed to be combined with data from othersources,

� protracted engagement of consultants,

� unexpected efficiency decline as the staffcannot proceed in the usual way.

The cost of owning an enterprise-wide applica-tion goes far beyond an application’s purchaseprice.�1�. The largest costs result from imple-mentation efforts. Failed efforts have broughtsome companies to sue the ERP supplier �4�.

6.2. Proposal for IS Development Policy

Recommendations to Customers and Users

Potential customers should perform the follow-ing actions:

� Estimation whether the intended ERP pur-chase really corresponds to their require-ments �5��2�.

� Market-oriented complex companieswhotend to be and stay in some way specificand recognizable and wish to maintainthe edge over competition should rely onsome proprietary development, at leastfor support of their highly competitivefunctions.

� Manufacturing companies should con-sider the applicability of the offered ready-made solutions. For example, production-planning approach heavily depends onthe type of manufacturing: assembly in-dustry versus high series or continuousproduction, assembly tables versus pro-duction processes descriptions �11� etc.


� Cost benefit analysis, especially the ratio be-tween adaptation of business procedures tosoftware versus the costs of software adap-tation to business processes.

� Definition of appropriate metrics for out-sourcers�consultants, where their resultsshould prevail over their physical presence.

� An additional criterion to evaluate theoutsourcers can derive from independenttesting of the staff ability to use the im-plemented solution.

� Definition of metrics to evaluate and manageown IT staff, which includes:

� education and maximum usage of inter-nal human resources,

� awards according to successful imple-mentation,

� readiness to changes in staff, in order tosubstitute overpriced employees,

� avoiding of possible blackmails and mis-use after the implementers get acquaintedwith the weak points of the company,

� Assurance of acceptability for solutions de-livered from outsourcers, achieved by pre-cise contracts, definition of acceptance testsand major payments only after successfulimplementation.

� Planning of implementation and maintenan-ce, including feasibility studies and riskman-agement.

� Establishing of a pilot project or installationto prove the functionality.

� Estimation of alternative strategies:

� Simultaneous implementation, so calledBig Bang requires excessive efforts; it of-ten fails and it should be avoided.

� Franchise Strategy comprises the instal-lation of independent subsystemswith in-tegration only in their common process�e.g. in Accounting�.

� Slam-dunk strategy is oriented to somecrucial business processes. It is conve-nient for smaller companies that are notready to invest in comprehensive ERPs.

Information system owners who have just com-pleted the implementation cycle or who intendto start a new cycle should perform:

� System quality analysis.

� Revision of the computerization project.

� System cost benefit analysis.

Regardless of the customer type, we suggest:

� Engagement of independent consultants, es-pecially if an impartial quality analysis isrequired.

� Engagement of independent �third party�outsourcers for support and maintenance, ifapplicable.

� Establishing of at least a minimum local ITexpertise. Local experience has shown thatERP installations became successful after lo-cal expertise had been gained.

� The most important aspect is the proper mo-tivation and high commitment of the topmanagement. Establishing of an informa-tion system cannot be completely purchasednor delegated. It cannot be introduced bybrute force, because there are too many pos-sibilities to compromise the project if it isperceived as being against some of the stake-holders’ interests �10�.

Recommendations to ERP Suppliers andDevelopers in a Developing Country

In addition to some widely recognized mea-sures �17� suppliers-representatives of foreignERP solutions should:

� In a country with moderate to high corrup-tion, resist the temptation to apply bribery asa shortsighted method, which should back-fire eventually.

� Increase the quality of support and customiza-tion instead of pure reselling.

� Educate local consultants instead of relyingon foreign ones who are expensive and ofteninept to understand the local conditions.

� Include local experts in the developmentand customization.

� Negotiate joint approach to thirdmarkets.

� Adapt the licence price to the local percapita BDP.


� Forward users’ remarks to the software pro-ducers to improve the quality and customers’satisfaction.

� Accept that major payment is due afterthe user-acknowledged functionality hasbeen achieved.

Domestic developers should:

� Technically update and complete the exist-ing applications.

� Offer application hosting and provide appli-cation services as soon as possible.

� Direct themselves towards small andmediumsize customers.

� Direct themselves towards specialized soft-ware, which is not present in major ERPs.

7. Conclusion

The response rate in this investigation left uswith an impression that domestic companies aremostly uninterested in improvements, which israther strange, considering the potential of cur-rent unsatisfactory state of computerization inthe country. The suppliers of ERP solutions,who trust their products, accepted the challengeof evaluation and so did the customers who dohave some achievements but who are also awareof the deficiencies in their systems.

The leading world-renowned ERP packages aretechnologically modern, comprehensive and re-liable software. The functional quality andsophistication of the implemented software ishowever surprisingly mediocre. Customizationand implementation of a “ready-made” solutioncan extend through years, creating a large dis-crepancy between the price of the basic productand the costs of customization. Themain reasonfor high costs is the necessity that, simultane-ously, the software needs to be adapted to theorganization and the organization and the busi-ness processes have to be adapted to the ERPpackage. The weakest point might be the con-sultants who often lack the knowledge abouttheir complex ERP system and, on the otherhand, they lack experience in information sys-tems development and do not understand the tar-get system. The presumed, and in ERP systemsembedded high level of organization and obe-dience to the rules, cannot always be respected

in developing countries. Potential ERP buyersshould count on a dependency upon the supplierand on expensive consultancy for a long time.Reliance upon a pool of own experts might bethe best solution in a country with relatively lowsalaries.

The main characteristics of domestic ERP prod-ucts, with few exceptions, are incompletenessand inadequacy and�or technical obsolence.Best solutions can be found for Accounting,Human resources and Payrolls. Material han-dling is less present. Support for manufactur-ing is lacking. On the other hand, investigationhas detected some cheap local solutions, whichcould in a fair competition challenge, accordingto locally appreciated functionality, the world-renowned products.

Regardless whether an information system isbought or developed, computerization shouldlead to the integration of data and to standard-ization of business processes. The total pricefor a certain customer can hardly be properlyforecasted. Many hidden costs must be takeninto account. The largest cost results form pos-sible failure or even worse, protracted efforts ina failed project.

Software development is work intensive. Theprice of human work in highly developed coun-tries can be up to 10 times higher than in amedium developed country like Croatia. Forthe latter countries, price performance ratio oftheir proprietary solutions can be more accept-able than in highly developed countries. World-renowned ERP solutions are aimed for highlyorganized societies. The assumption that in-troduction of such systems into less developedenvironments would introduce order and dis-cipline and improve organization, may soundplausible but it is not necessarily true. If thegap is too wide, the failure is granted. Withoutinvestment in local expertise, the technologygap only widens. Our suggestion for countrieslike Croatia is not to expect that wonderful solu-tions can be simply bought. Own effort is indis-pensable and only this can make such countriesrespected partners on the world market.

We cannot offer a clear answer to an intriguingquestion,whether to buy an expensive renownedERP or a cheaper domestic counterpart. A com-plex ERP might take too much effort and timeto implement. Domestic product can be put into


function quicker and requires less restructuring.When, with time, the user finds the functionalityof such a systemunsatisfactory and the deliverercannot meet the growing requests for enhance-ment, itmight be the right time for acquisition ofa renown ERP. In such case acceptance shouldproceed easier and quicker. Which approach ismore favourable has to be deliberated on caseby case basis.

It is natural that any less developed countrytends to become part of the highly developedworld. Pure reliance on foreign expertise canmake only standing dependency and maintain,or even widen the gap. Illustrative is the exam-ple of theRepublic ofKorea. It could notmake asubstantial progress regardless of its successfulelectronic industry as long as it was dependanton foreign supplies. The better the Korean finalproducts were sold, the more expensive becamethe imported inputs. When Korea took over theexpertise, it became a well-known success story�25�.

References

�1� Applix Enterprise, Total Cost of Ownership Studyfor Enterprise Applications, white paper, Applix.Inc, 2000.

�2� R. B. CAMPBELL, ERP: Show me the Money!, Axis-logic Consulting Inc., 2001,http��www�axislogic�com�erpstory�htm

�3� Ernst & Young and uni Wien, Optimierungsbedarfvon ERP-Software in Osterreich, Ernst & YoungUnternehmensberatung Ges.m.b.H und Institut furKreditwirtschaft an der Wirtschaftsuniversitat Wien,Vienna, 1999.

�4� S. GOODWIN, Software Risk Management MakesGood Business Sense,http��www�stickyminds�com�sitewide�asp�ObjectId��Function�DETAILBROWSE�ObjectType�ART, 2004-07-06.

�5� B. HECHT, Choose the right ERP software, Data-mation, 43�3�, pp. 56–58.

�6� P. C. HOLLAND, B. LIGHT, A Critical Success Fac-torsModel for ERP Implementation,IEEESoftware,16�3�, pp. 30–36.

�7� IDC, The Integrated Enterprise Application (ERP)Market in Croatia, IDC East Central Europe, Praha,1998.

�8� IDC, Management Summary, 1999 Croatian ERPMarket, IDC East Central Europe, Praha, 1999.

�9� IDC, The Integrated Enterprise Resource Man-agement Software Application Market in Croatia,1999–2004, IDC East Central Europe, Praha, 2000.

�10� D. KALPIC, K. FERTALJ, V. MORNAR, Analysis ofReasons for Failure of a Major Information SystemProject, BITWorld 2001 Conference Proceedings,CD-ROM, ISBN 0 905304 36 5, Kamel, Sherif�ed.�. Cairo: The American University in Cairo,2001, pp. 1–8.

�11� D. KALPIC, M. BARANOVIC, V. MORNAR, CaseStudy Based on a Multi-Period Multi-Criteria Pro-duction Planning Model,European Journal of Oper-ational Research 87, �1995�, pp. 658–669, ElsevierScience B. V., Netherlands.

�12� C. KOCH, The ABCs of ERP, 2002.http��www�cio�com�research�erp�edit�erpbasics�html

�13� M. LEVINSON, Let’s Stop Wasting $78 Billion aYear, CIO Magazine, 15.10.2001.http��www�cio�com�archive��wasting�html

�14� A. C. MACADA, M. A. MAHMOOD, J. L. BECKER,Development and Validation of an Instrument toEvaluate the Impact of Information Technology In-vestments: A Cross-Country Analysis, Presentedat the IRMA 2000 — Challenges of InformationTechnology Management in the 21st Century, �May21–24, 2000� Anchorage, AK, USA, pp. 893–896.

�15� METAGROUP, A Platform-Related Cost Analysisof ERP Applications on-Going Support Costs inthe Mid-Tier, MetaGroup Consulting, 11. February2000.

�16� M. A. MAHMOOD, Amerireal Corporation: Informa-tion Technology and Organizational Performance”,Presented at the IRMA 2000 — Challenges of Infor-mation Technology Management in the 21st Cen-tury, �May 21–24, 2000� Anchorage, AK, USA, pp.1037–1038.

�17� A.-W. SCHEER, F. HABERMANN, Enterprise resourceplanning: making ERP a success, Communicationsof the ACM, 43�4�, pp. 57–61.

�18� SEI, Software Capability Evaluation Version 3.0Method Description CMU/SEI-96-TR-002, Soft-ware Engineering Institute, Carnegie Mellon Uni-versity, Pittsburgh, Pennsylvania, 1996.

�19� SEI, Software Capability Evaluation Version 3.0Implementation Guide for Supplier SelectionCMU/SEI-95-TR-012, Software Engineering Insti-tute, Carnegie Mellon University, Pittsburgh, Penn-sylvania, 1996.

�20� C. SOH, S. S. KIEN, J. TAY-YAP, Enterprise resourceplanning: cultural fits and misfits: Is ERP a univer-sal solution?, Communications of the ACM, 43�4�,pp. 47–51.

�21� T. M. SOMERS, K. NELSON, The Impact of CriticalSuccess Factors across the Stages of EnterpriseResource Planning Implementations, Proceedingsof the 34th Hawaii International Conference onSystem Sciences, �2001�, Hawaii, USA.


�22� C. STEDMAN, What’s Next for ERP?, Computer-world, 33 �1999�, pp. 48–49.

�23� C. STEDMAN, ERP projects cost more than their‘measurable’ payback, Computerworld, �OnlineNews, 03�29�99 06:29 PM�.

�24� J. ZYGMONT, ERP system performance — rethinkreporting, Datamation IT management Update,19.5.1999.

�25� �� Invited lecture of a Korean expert on the 18th

International Conference on Computers & Indus-trial Engineering, ICC&IE’95, Shanghai, China,October 25–27, 1995.

Received: February, 2004Revised: July, 2004

Accepted: August, 2004

Contact address:

Kresimir FertaljFaculty of Electrical Engineering and Computing

University of ZagrebUnska 3

10000 ZagrebCroatia

Phone: �385 1 6129 918Fax: �385 1 6129 915

e-mail: kresimir�fertalj�fer�hr

Damir KalpicFaculty of Electrical Engineering and Computing

University of ZagrebUnska 3

10000 ZagrebCroatia

Phone: �385 1 6129 919Fax: �385 1 6129 915

e-mail: damir�kalpic�fer�hr

KRESIMIR FERTALJ is an assistant professor at the Department of Com-puter Science of the Faculty of Electrical Engineering and Computing,University of Zagreb, Croatia. He graduated and received his M.Sc. andPh.D. degrees in computing from the same institution. Since graduationin 1988, he has been working at the Department of Applied Mathemat-ics, where he currently lectures a couple of undergraduate and post-graduate courses in computing. He also held courses in data modelling,information systems development and fourth generation languages. Hisprofessional and scientific interest is in computer-aided software engi-neering and complex information systems. He has defined a propri-etary software development method based on iterative prototyping and,based on it, developed a proprietary source code generator. He ran a fewprojects supported by the Croatian Ministry of Science and Technology.Currently he is the principal researcher on two such projects, “Improve-ment of Information System Development Methods” and “BusinessInformation System Feasibility”. He has authored and co-authoredmany scientific and professional publications and participated in con-ferences, locally and abroad. Professionally, he participated in a dozeninformation system projects for business, industry and administration.

DAMIR KALPIC received his Ph.D. degree in 1982 in computer sciencefrom the Faculty of Electrical Engineering, University of Zagreb. Since1971 he has been working in the same institution, nowadays calledFaculty of Electrical Engineering and Computing. He is full professor,teaching information systems and software-related courses in under-graduate and postgraduate studies. He has mentored more than 180graduation theses, and a number of M.Sc. theses and doctorates. Hehas written over 160 texts and R&D papers, mostly reflecting practicalimplementation of IT in many different fields. He is the senior memberof the R&D team at the Department of Computer Science. Togetherwith his colleagues, he has completed more than 10 complex projectsaccepted in practice by industry and administration. They cover in-formation systems development, operational research applications anddatabases. He was the deputy dean and vice dean, editor of the CITJournal and member of many international programme committees. Foreight years he was chairing the International Programme Committee ofthe Conference “Information Technology Interfaces”.

Documents

ERP Software Evaluation and Comparative Analysis...198 ERP Software Evaluation and Comparative Analysis We believe that three weight values are suffi-cient: 1 – basic functionality,