A case study for project and portfolio management ... recent survey (Raymond and Bergeron, 2008) demonstrates that project managers are starting to realise the benefits obtained from

Int. J. Project Organisation and Management, Vol. 2, No. 4, 2010 361

Copyright © 2010 Inderscience Enterprises Ltd.

A case study for project and portfolio management information system selection: a group AHP-scoring model approach

Vassilis C. Gerogiannis* and Panos Fitsilis Department of Project Management, Technological Research Centre of Thessaly, Technological Education Institute of Larissa, 41110, Larissa, Greece E-mail: [email protected] E-mail: [email protected] *Corresponding author

Dimitra Voulgaridou Department of Mechanical Engineering, Sector of Industrial Management and Operational Research, National Technical University of Athens, Iroon Polytechniou 9, 15780 Athens, Greece E-mail: [email protected]

Konstantinos A. Kirytopoulos Financial and Management Engineering Department, University of the Aegean, Kountourioti 41, 82100 Chios, Greece E-mail: [email protected]

Evi Sachini Strategic Planning and Development Department, Hellenic National Documentation Centre, National Hellenic Research Foundation, Vas. Constantinou 48, 11635 Athens, Greece E-mail: [email protected]

Abstract: Features variety, variations among organisations’ needs and the plethora of powerful project and portfolio management (PPM) information systems in the market, make the selection process among various PPM information systems a complicated process. The large number of the evaluation criteria involved during the decision aggravates the problem. This paper addresses the limitation of the number of criteria in such complex decision problems. It develops a hybrid model that exploits the benefits of the group analytic hierarchy process (GAHP) along with the simplicity of the scoring model for justifying the final selection. The proposed approach permits consideration of much more criteria than the typical AHP approach can

362 V.C. Gerogiannis et al.

reasonably handle and takes into account experts’ opinions for the comparison of the PPM information systems as well as users’ opinions on specific preferences/ needs. The proposed approach is applied on a real case in a public Greek organisation.

Keywords: project and portfolio management information systems; multi-criteria decision-making; MCDM; group decision-making; analytic hierarchy process; AHP; hybrid model.

Reference to this paper should be made as follows: Gerogiannis, V.C., Fitsilis, P., Voulgaridou, D., Kirytopoulos, K.A. and Sachini, E. (2010) ‘A case study for project and portfolio management information system selection: a group AHP-scoring model approach’, Int. J. Project Organisation and Management, Vol. 2, No. 4, pp.361–381.

Biographical notes: Vassilis C. Gerogiannis holds a Diploma and a PhD in Software Engineering from the University of Patras, Greece. At present, he is a Fulltime Assistant Professor at the Project Management Department in the Technological Educational Institute (TEI) of Larissa, Greece. He is also a Lecturer at the Greek Open University and a Module Leader in a Post Graduate Program of Studies in Management Science, organised by TEI of Larissa in cooperation with the Staffordshire University, UK. He has been participating as a Technical Consultant and Project Manager in EU funded R&D projects. Up to now, he has published more than 40 papers in international journals and conference proceedings. His research interests include project management, software project management and software engineering.

Panos Fitsilis is a Fulltime Professor at the Department of Project Management of the Technological Education Institute of Larissa, Greece. He holds a Diploma and a PhD from the Department of Computer Engineering and Informatics from the University of Patras, Greece. He has extensive experience in managing IT projects for large organisations of the private and public sector. His research interests include project management methodologies, project software management, software development methodologies and business information systems.

Dimitra Voulgaridou holds an Electrical and Computer Engineer Diploma and a PhD on Decision Support and Supply Chain Management. She works as a Research and Teaching Associate at National Technical University of Athens and at University of the Aegean. Her main research interests cover decision support, supply chain management, project management and ERP systems.

Konstantinos A. Kirytopoulos holds a Mechanical Engineering Diploma (Bachelor plus MSc equivalent) and PhD on Project Risk Management. He is specialising in industrial management and is currently an Assistant Professor at the University of the Aegean in Greece. His main research activity and publications are in the fields of risk management, project management and decision-making. More information about his academic activity may be reached at www.kirytopoulos.eu.

Evi Sachini has been the Head of the Strategic Planning and Development Department of the National Documentation Centre of Greece since 1997. She holds a PhD in Chemical Engineering from the National Technical University of Athens. She has been the manager of the project ‘National Information System on Research and Technology’ aiming to develop open access infrastructures addressed to the academic and research communities in Greece.

A case study for project and portfolio management information system 363

1 Introduction

The exploitation of an appropriate project and portfolio management (PPM) information system provides a lot of advantages for an organisation that undertakes projects and/or portfolios of projects to implement organisational changes, business process redesign and develop new products and/or services. A recent survey (Raymond and Bergeron, 2008) demonstrates that project managers are starting to realise the benefits obtained from PPM information systems, in terms of achieving increased productivity, effectiveness and efficiency of their managerial tasks. Other empirical studies (Liberatore and Pollack-Johnson, 2003) report that larger and more complex projects are pushing project managers to use PPM software more. Contemporary PPM information systems have embarked from single-user/single-project management systems to distributed, multi-user, multi-functional software packages which offer integrated project management solutions, not limited to scope management, budget and planning control. Market analyses in the area (Wang, 2007; Light and Stang, 2008) present that modern PPM information systems are trying to offer support for all knowledge areas described in the ‘Project Management Body of Knowledge’ of the Project Management Institute (PMI, 2008) through covering an expansive view of the ‘integration management’ knowledge area that includes investment analysis, alignment of strategic programmes and support for business case creation. Commercial PPM information systems provide support for a range of an organisation’s activities by offering a number of features such as time, resource and cost management, workgroup capabilities, reporting features, risk and contract management support. More interested readers, not familiar with the technical characteristics of commercial PPM systems, are referred to Ahlemann (2007), where detailed information is given for 34 commercial PPM systems. In that report, each PPM system is presented based on approximately 70 functional and 30 non-functional features.

This variety of features along with the variation among each organisation’s needs and the plethora of PPM information systems in the market make the selection process of a PPM system very difficult and complicated. The problem is often approached by ad hoc procedures based only on personal preferences of the decision-maker or marketing information/hype of PPM information systems and this may lead to a final selection that does not precisely reflect the actual organisation’s needs or, even worse, to an unsuitable PPM system. Empirical research shows that project managers often experience limitations when using commercial project management software and may find it inadequate for managing complex projects (White and Fortune, 2002). Therefore, a well-established and systematic technique, from the multi-criteria decision-making (MCDM) domain, can be useful to support a PPM system selection process. Although, there is no a generic methodology that can be adopted for selecting a software package of any type, literature reviews on evaluating software products suggest that users and decision-makers can receive a lot of support, if they decide to adopt an MCDM method (Jadhav and Sonar, 2009).

In particular, the findings of review studies (Vaidya and Kumar, 2006; Jadhav and Sonar, 2009) present that the analytic hierarchy process (AHP) has been widely and successfully used in evaluating several types of software packages (e.g., MRP/ERP systems, simulation software, CAD/CASE systems, KM systems, etc.). The AHP method


was introduced by Saaty (1980) and its primary objective is to classify a number of alternatives (e.g., candidate software packages) by considering a given set of qualitative and/or quantitative criteria, according to pairwise comparisons/judgements provided by the decision-makers. AHP application results in a hierarchical levelling of the selection criteria, where the upper hierarchy level is the goal of the decision process, the next level defines the selection criteria, which can be further subdivided into subcriteria at lower hierarchy levels and finally, the bottom level presents the alternative decisions to be evaluated.

Although, AHP has been extensively used for the evaluation of various software products and systems, little work has been done in the field of evaluating PPM information systems. This can be attributed to the fact that despite the many advantages of the AHP, scholars seem to agree that its main limitation is the large number of pairwise comparisons needed, which result in significant time and effort requirements. The time needed increases geometrically with the increase of the criteria involved, making AHP practical prohibitive for complicate decision environments, such as the selection of a PPM system.

In addition, PPM information systems selection, as every complicated problem, is recommended to be held by groups rather than individuals (Tindale, 2003). This is mainly because groups can represent a larger and more diverse set of perspectives and constituencies, thus being more ‘fair’ (Tyler and Smith, 1998). Moreover, the idea that ‘two heads are better than one’ is widespread and typically accurate, based on the empirical record (Bonner, 2004).

The research question for this paper is how can a decision-making process overcome the limitation of a large number of criteria in a complex decision problem, such as the selection of a PPM information system. As a response, in this paper a practical and easy to use hybrid model for the evaluation of a set of PPM systems is developed, to help an organisation to select a PPM system that reflects the organisation’s requirements. The proposed approach exploits the benefits of MCDM and, in particular, encompasses all the advantages of the AHP method and group-based AHP decision techniques, while it manages to maintain the simplicity required by integrating the scoring model for the evaluation of the criteria. Finally, the involvement of both users and experts in the decision process achieves to exploit the expertise/interests of each one of them and to strengthen the results. In a nutshell, the contribution of the proposed approach focuses on a twofold innovation:

• it uses a hybrid model that permits consideration of much more criteria than the typical AHP approach can reasonably handle

• it takes into account experts’ opinion for the comparison of the PPM information systems as well as users’ opinion on specific preferences/needs.

The rest of this paper is organised as follows. Section 2 presents the relevant literature background for AHP and group-based decision techniques. Software package selection problems and existing PPM evaluation frameworks are also critically discussed. Section 3 provides a detailed description of the proposed hybrid model. Section 4 describes the model application in a real case study, while conclusions and possible extensions of the research work are addressed in Section 5.


2 Literature review

2.1 MCDM and AHP applications in software package selection problems

An MCDM method (like AHP) overcomes some limitations of a conventional weighting scoring method (WSM). On one hand, in a WSM, criteria weights and rating scales are assigned arbitrary and real numbers are produced as final results. So, final comparative values are often realised to represent the true difference among the decision alternatives, rather than their relative ranking. Furthermore, assigning representative criteria weights can be a very difficult task, when the number of selection criteria becomes large (Kontio, 1996).

On the other hand, among the main advantages of applying the AHP method one could consider that (Jadhav and Sonar, 2009):

1 it is capable to provide a hierarchical decomposition of a decision problem that helps in better understanding of the overall decision-making process

2 it handles both quantitative and qualitative criteria

3 it is based on relative, pairwise comparisons of all decision elements; instead of arbitrarily defining a percentage score and a weight for each decision element, AHP allows the decision-maker to focus on the comparison of two criteria/alternatives, at a time, thus, it decreases the possibility of defining ratings based only on personal perceptions of the evaluators or other external influences

4 AHP is applicable to both individual and group-based decision-making (this is often achieved by considering the geometric mean of all comparison values)

5 it enables consistency checks upon pairwise decision-comparison judgements

6 it supports sensitivity analysis to examine the effects of changing values of criteria weights on the final ranking of the decision alternatives

7 the method applicability and its extended features (e.g., sensitivity analysis) are supported by software systems – e.g., expert choice (www.expertchoice.com), super decisions (www.superdecisions.com) – although, pairwise comparisons and calculations/consistency checks can be easily implemented in a spreadsheet application software.

All the above advantages have influenced the wide application of AHP to MCDM problems, in many different sectors, including project management and software engineering project management. As far as project management is concerned, Al Harbi (2001) presented a field survey of applications of AHP, focusing, in particular, on the selection problem of the best contractor to undertake a construction project. The case study in this research uses a group decision-making approach that involves characteristics of candidate contractors as well as project priorities defined by the project owner. Prioritising software requirements and selecting component off the self-systems (COTS) are two representative examples of software engineering project management problems which have gained attention to be supported by AHP. In these problems, AHP has been used to compare software requirements (Karlsson and Ryan, 1997) or COTS products (Kontio, 1996; Lozano-Tello and Gomez Perez, 2002) by taking into account the relative importance between value and cost of each requirement/COTS product.


The application of AHP has been reported in many other case studies related to software package selection problems. The interested reader is referred, e.g., to (Ahmad and Laplante, 2007) where the authors propose the use of AHP to evaluate three imaging software packages by combining seven types of subjective data (software complexity metrics) and experts’ opinions. Their approach helps in deciding an appropriate imaging software when there are contradictory complexity metrics (quantitative data) for each candidate software package, by incorporating experts’ judgements and relying on their experience within the subject. Another representative example is the application of AHP in the selection of a suitable knowledge management (KM) tool (Ngai and Chan, 2005). The authors list various candidate decision criteria and subcriteria for justifying the selection process but, in their case study, they actually show the use of only three major criteria (cost, functionality and vendor) to evaluate three alternative KM systems. Considering additional criteria or subcriteria, often leads to a combinatorial explosion of the number of individual pairwise comparisons, thus, making AHP application very time and cost ineffective (Maiden and Ncube, 1998). A similar problem appears in another approach that follows AHP for the selection of an enterprise resource planning (ERP) information system (Karaarslan and Gundogar, 2008). The authors suggest eight selection criteria (seven of them correspond to different modules of an ERP system and the ninth criterion covers the general system functionality) and a very detailed list of subcriteria for the evaluation of two ERP systems. The evaluators in the presented case study were asked to rate explicitly (with an integer value from zero to five) each subcriterion, an approach that improved the practical application of the decision-making method.

Despite its popularity, there are two main difficulties related with the practical application of AHP. First, when determining ‘crisp’ comparative values, any uncertainties on judgements of decision-makers cannot be easily handled (Saaty, 1980). A proposed solution for this limitation is to use a combination of AHP with fuzzy logic (Chang, 1996). For example, in Cebeci (2009), a combined fuzzy logic – AHP based approach is proposed to select the most suitable ERP system for a textile manufacturing company. However, the complexity of fuzzy logic approaches often raises difficulties for their practical application; thus, the problem is practically handled with sensitivity analysis. A second difficulty in AHP application appears when there are dependencies among the selection criteria. In such a case, the analytic network process (ANP) should be used (Kirytopoulos et al., 2008a). ANP is an AHP extension that handles both intra- and inter-dependencies among clusters of selection criteria (Saaty, 1996). The application of the ANP method can be, however, very time and cost intensive, since it often requires a great number of pairwise comparisons (Voulgaridou et al., 2009). For example, a case study of ANP use in selecting among three ERP systems, with respect to 12 selection criteria organised into system and vendor related clusters, concludes that considering all possible interactions among criteria requires much more additional time and effort (Percin, 2008) than the typical AHP approach.

Although, AHP has been extensively used for the evaluation of various software products, surprisingly little work has been done in the field of evaluating PPM information systems with AHP or even by considering its extensions, such as ANP. Even those who tried to use the method for project management systems selection (Ahmad and Laplante, 2006) admitted that their work is rather indicative with main objective to expose a representative case for illustrating the selection process and not to create a definitive set of criteria/features that should be taken into account in practice. The reasoning behind this shortcome in bibliography should be probably assigned to the


inability of the AHP and ANP methods to handle problems with many criteria. Selecting a sophisticated PPM information system may entail more than 100 criteria/functional characteristics (Ahlemann, 2007; NASA PMTWG, 2004). Thus, the use of a hybrid model such as the one proposed in this paper, could be efficient for practically using AHP in a real practical context.

On the other hand, the selection of factors involved in decision-making for selecting a PPM information system is crucial for an effective outcome. Sophisticated PPM information systems no longer emphasise only on scheduling and resource management issues. They are software tools that can support the life-cycle management of a project or a portfolio of projects. Therefore, the selection of an appropriate PPM system is often a complex process that requires systematic support. There are research papers in the relevant literature which aim to compare PPM information systems through the evaluation of quality features [such as user friendliness and documentation; De Wit and Herroelen (1990)] or conventional technical characteristics, such as scheduling and resource management capabilities (Rushinek and Rushinek, 1991; Maroto and Tormos, 1994). However, it seems to be a lack of comparative approaches that consider both experts’ judgements and PPM system users’ requirements, in an actual organisational context. In a real world context, it should be stressed that the users should decide on what is important or not for their work and experts decide on which PPM information system is better for each criterion addressed.

The proposed approach in this paper follows a practical, hybrid, group-based method that was applied in an actual context of an organisation that required support for selecting an appropriate PPM system to fulfil specific project/portfolio management requirements. The selection criteria were specified according to a comprehensive, functional oriented evaluation framework (NASA PMTWG, 2004). The result was a detailed set of criteria/functional features grouped into clusters.

2.2 Group analytic hierarchy process (GAHP)

Group decision-making is considered to be a process for deriving a single group preference from a number of individual preferences regarding a finite set of criteria and alternatives. The group decision-making process in the AHP involves the construction of pairwise comparison matrices at each level of network either by consensus voting or by aggregating the individual preferences (Saaty, 1989). Forman and Peniwati (1998) showed that the group prioritisation methods in the AHP method apply two basic techniques for aggregating the individual preferences into a group preference, depending on whether the group wants to act together as a unit or as separate individuals, and specify two aggregation approaches:

1 aggregating individual judgements (AIJ)

2 aggregating individual priorities (AIP).

AIJ is a synergistic aggregation approach, where the judgements of the decision-makers for each set of pairwise comparisons are aggregated into a new set of aggregated group judgements at each level. The aggregated group judgements are considered as judgements of a ‘new individual’ and the priorities of this individual are derived as a group solution. In the AIP approach, the group is considered as a collection of individuals, with different value systems. This approach requires priorities to be obtained from each individual, and


then aggregated into final group priorities. Both aggregation approaches in the AHP involve two independent stages, performed in different successions: group synthesis and prioritisation (Bryson and Joseph, 1999). Group synthesis is based on an additional aggregation procedure that combines the individual judgements (AIJ) or priorities (AIP) into group preferences, while the prioritisation stage requires an appropriate prioritisation method for deriving a single group priority (AIJ) or a number of individual priorities (AIP).

The weighted arithmetic mean (WAM) and the geometric mean (GM) mathematical procedures are commonly used to determine group preferences for both the AIJ and AIP aggregation approaches. Aczel and Saaty (1983) claim that only the GM is an appropriate procedure as it preserves the reciprocal properties of the aggregated pairwise comparison matrices. Forman and Peniwati (1998) also state that the GM should be used for AIJ, as for AIP either the WAM or GM are meaningful procedures, satisfying the Pareto optimality principle. Lately, a third approach, the weighted geometric mean (WGM) is considered to be the optimal procedure (Levy and Taji, 2007). To illustrate the group prioritisation process in the AHP, let aij,z represent the comparison of elements i and j, with respect to the element z, in a pairwise comparison matrix A for n decision-makers. The group judgement by taking the weighted geometric mean of all comparisons is equal to:

1

,1 ,1

, ,1

1

lnexpi

n npiaij zi i ij znp i

ij z ij z ni

ii

p aw a

p

∑=

=

=

=

⎛ ⎞⎜ ⎟

⎛ ⎞ ⎜ ⎟= =⎜ ⎟ ⎜ ⎟⎜ ⎟⎝ ⎠ ⎜ ⎟

⎜ ⎟⎝ ⎠

∑∏

∑

where aij,z is each person’s paired comparison, n is the number of decision-makers, and pi (0, 0 ≤ pi ≤ 1, 0) is the importance priority for the person’s paired comparison.

2.3 Evaluation frameworks for PPM information systems

Raymond and Bergeron (2008) reported that a number of project managers indicate strong impact of the PPM system upon the successful completion of their projects, while others do not. Their research indicates that ‘unsatisfied’ project managers are depended upon a PPM system that produces lower quality information; hence they use the system less and they are less supported in their project management tasks. From this research, we can conclude that it is important for an organisation to select a proper PPM system that covers technical, managerial and organisational needs.

The PPM system selection process can be supported by referencing to published survey results. In the past, e.g., PMI had published an extensive survey (PMI, 1999) that compared more than 200 different PPM information systems in dimensions like scheduling, cost management, risk management, resource management and communication management. However, such detailed comparisons focus rather on technical factors which represent vendors’ perspectives and they should be always utilised with care by considering specific project management needs within the context of individual organisations. Support in setting up a PPM system can be also gained by considering the users’ perceptions from a PPM system usage. This often requires the application of a technology acceptance model to provide some empirical evidence on the


relationships that exist, e.g., between a system’s usefulness and its ease of use (Davis, 1989). A representative example of this approach is the one presented in (Ali and Money, 2005). This work surveyed 497 PPM system users and the final result was a general index for measuring the effectiveness of PPM information systems according to four, user-oriented, dimensions (i.e., information quality, system functionality, ease of use, performance impact). However, the theoretical background of such an index lacks the mathematical formality of an MCDM approach. Formal support can be proven useful in cases that selection criteria contradict each another and respective PPM system users are actually expressing their preferences and not their actual knowledge on potential benefits derived from a PPM system.

To reach a more detailed list of selection factors, evaluators-decision-makers may utilise criteria offered by a conceptual reference information model for PPM information systems, like e.g., the RefModePM model (Ahlemann, 2009). A conceptual reference model provides a holistic approach for a PPM system selection process, since it handles the selection problem from a business process reengineering viewpoint. A reference model can be followed, first of all, to categorise all respective PPM system users (i.e., members of an organisation personnel), then to identify the organisation’s project management processes and, finally, to provide descriptions for the organisation’s project data. RefModePM is mainly a process oriented framework that classifies project management processes within a project-based organisation into the following general areas, which, in a broad sense, follow the phases of a project life-cycle:

1 project idea generation

2 project idea evaluation

3 portfolio planning

4 project preparation

5 detailed planning

6 project execution

7 project controlling

8 portfolio controlling

9 internal project termination

10 external project termination.

Each of these areas is further subdivided into specific process steps which finally, form the selection criteria for a candidate PPM system.

Apart from RefModePM, to decide upon PPM information systems selection criteria, we have considered another evaluation framework, proposed by the NASA PPM information systems working group (NASA PMTWG, 2004). This framework (NASAPM) is more oriented towards functional features of a PPM system. It lists explicitly a number of technical/performance/reporting requirements to be used as selection criteria, where each requirement is further analysed into a set of functional features. The following nine evaluation clusters of criteria, based on the NASAPM evaluation framework, have been used in our selection process:


C1 open database connectivity and architecture – workgroup and networking capabilities

C2 ease of use

C3 project scheduling support – definition of project task/field features

C4 baselining – tracking project progress and calendar features

C5 resource and cost management features

C6 risk management features

C7 reporting features

C8 support for supplier and contract management

C9 portfolio management features.

Each cluster includes a set of specific criteria/functional features. For instance, cluster C3 (project scheduling support – definition of project task/field features) summarises 18 criteria/functional features, namely:

1 perform basic scheduling/PERT functionality

2 allow variable scaling for task duration

3 perform full critical path method (CPM) functionality, including capability of showing multiple critical paths in output reports

4 allow users to designate logical relationships (i.e., start-to-start/SS, start-to-finish/SF, finish-to-finish/FF, and finish-to-start/FS)

5 allow users to customise tables and views

6 allow users to create project templates

7 generate an organisational breakdown structure (OBS) and a work breakdown structure (WBS) or allow user to impose a WBS

8 allow users to assign positive or negative lag/lead times on logical relationships

9 perform resource levelling and smoothing

10 have the capability of ‘de-linking’ percent complete from remaining duration

11 allow users to define and assign constraints to tasks-milestones

12 allow users to specify tasks or milestones to be rolled-up

13 allow users to define fields for each project/task/resource

14 roll-up multiple projects into a master schedule

15 define a task with the duration being automatically calculated based upon its dependency with another task (hammock task)


16 incorporate a large comment/notes field for entry of soft information

17 define task start/end dates as fixed, resource-driven or effort-driven

18 allow users to create a read-only version of project (fields, tables, resources, calendars).

In total, for the nine listed above, evaluation clusters, a total number of 105 criteria have been identified to be evaluated.

This vast number of criteria prevents decision-makers from using the typical AHP approach. It should be mentioned here that the use of ANP is not considered at all, first because there are no dependencies among criteria of different clusters (the problem can be indeed modelled as a hierarchical problem) and second because its use would be even more complicated than the AHP for this number of criteria (Kirytopoulos et al., 2008b).

3 The proposed hybrid model

The proposed framework consists of a ‘four-steps’ hybrid model, as shown in Figure 1. The first step employs the GAHP method in order that the users classify the clusters of the criteria based on their relative importance. The second step is performed by the experts who, based on the scoring model, assign weights to the specific criteria/functional features of the PPM information systems. Then, the overall importance of each alternative is calculated by aggregating the results of the GAHP and the scoring model. Finally, sensitivity analysis is performed in order to evaluate the stability of the final decision. A detailed description of each step follows hereafter:

Step 1 The users evaluate the clusters of the criteria by implementing the AHP according to the needs of their organisation. The users are divided into two groups. Team I users’ group consists of project managers involved in planning, executing and monitoring of single projects and Team II users’ group consists of project officers, responsible for strategic management, contract management, multi-project coordination and planning the organisation’s project portfolios. Each team assigns weights to the clusters according to its value system and the weighted geometric mean is utilised in order to calculate final weights of this step. As far as it concerns the clusters, they are adopted from the NASAPM evaluation framework:

C1 open database connectivity and architecture – workgroup and networking capabilities

C2 ease of use

C3 project scheduling support – definition of project task/field features

C4 baselining – tracking project progress and calendar features

C5 resource and cost management features

C6 risk management features

C7 reporting features


C8 support for supplier and contract management

C9 portfolio management measures.

The aggregation of the two teams’ judgements is performed using the GAHP in order to obtain the final weights of the clusters according to the users.

Step 2 The experts implement the scoring model in order to evaluate the alternatives (PPM information systems) with respect to each of the criteria. The criteria selected correspond to the functional features, as proposed by the NASAPM evaluation framework.

Step 3 The weights of the decision-makers (both users and experts) are aggregated in order to provide the final result. The following formula describes the mathematical expression of that aggregation:

1 1 2 2 9 9...N c c c c c cW w N w N w N= × + × + + × (1)

where

WN the final weighted score of the alternative N

Wci the weight of cluster i (i = 1, 2, ..., 9) according to users’ judgements, and

Nci the score of the alternative N for the criteria included in cluster i (i = 1, 2, …, 9) according to the experts.

Step 4 This final step performs sensitivity analysis to the final weights obtained during the previous step in order to determine if the final ranking of the alternatives is stable to changes in the inputs (judgements or weights).

The following section applies the proposed hybrid model to a real case of selecting a PPM system for satisfying the needs of a public organisation in Greece that undertakes projects on science research and technology issues.

Figure 1 The proposed hybrid model (see online version for colours)


4 Case study

4.1 General description

This section presents the application of the proposed hybrid GAHP-scoring model within the context of a project that took place in 2008 with the overall goal of selecting the most appropriate PPM system for the Hellenic National Documentation Centre (NDC) (www.ekt.gr). NDC is the public organisation in Greece responsible for providing documentation and information support on science research and technology issues. Since its establishment, in 1980, NDC has been involved in a number of projects, funded by national and EU operational programmes. The organisation operates the National Science and Technology Digital Library, develops digital content (such as the National Archive of PhD theses), supports the networking of the Hellenic University Digital Libraries and operates as national contact point for some EU funded R&D programmes like ‘ideas’, ‘cooperation’ and ‘capacities’.

NDC does not maintain an integrated project/portfolio management infrastructure. A team of 16 employees (users) with project management responsibilities undertakes and manages the organisation’s projects. Each of these employees utilises a stand-alone PPM system according to his/her own personal preferences/experience. In order to increase project management effectiveness and productivity, NDC has decided to adopt a collaborative PPM information system. NDC appointed the Technological Research Center of Thessaly in Greece to act as an expert and aid the decision-making process. Three experts (evaluators) from the Technological Research Center, with an average of five years teaching/professional experience in PPM systems, were involved in this process, aiming to identify NDC requirements from a PPM system and to select an appropriate system that will cover these requirements.

4.2 Implementation of the proposed hybrid model

Model construction

Three joint meetings between the users and the experts group were conducted to agree upon selection criteria and candidate PPM information systems. In the first meeting, the experts group from the Technological Research Center of Thessaly presented to the users group from the NDC the details of the NASAPM systems evaluation framework. Then the users and the experts agreed to base the selection process on technical/performance/reporting requirements listed in the NASAPM framework. In the second meeting, the users were asked to provide answers to a detailed questionnaire with the aim to identify the specific project management responsibilities of each user. Through analysing their answers, two teams of users were identified: Team I users group that consisted of 11 project managers involved in planning, executing and monitoring of single projects and Team II users group that included five project officers, responsible for strategic management, contract management, multi-project coordination and planning the organisation project portfolios. In the third meeting, a discussion took place upon possible candidate PPM information systems to be evaluated. Though there is a large number of an available PPM information system, both experts and users were queried to express their general opinion on ten PPM information systems which in market survey results (Light and Stang, 2008) are characterised as leaders and challengers in this


segment of enterprise software market. Five from these PPM information systems were excluded for two reasons. First, since they do not have presence in the national market and second, because experts were persuaded that their usage was inappropriate for the specific case, mainly due to lack of technical support and the non-availability of training services. This first-level screening resulted in a list of five powerful, widespread PPM information systems with portfolio management and workgroup capabilities and strong presence (i.e., technical/training support) in the national market. For project confidentiality reasons and aiming at the non-commercial promotion of any software package, we will refer to these PPM information systems as A, B, C, D and E. Based on the above, the decision model was constructed, as shown in Figure 2.

Figure 2 Hierarchical structure of the PPM information systems selection problem

Step 1 Users’ evaluation on clusters

After the construction of the model, the first step of the proposed hybrid model, concerning the evaluation of the clusters, was implemented. To analyse the users’ requirements from a PPM system, the authors designed and disseminated to the two teams of the NDC personnel an AHP-structured questionnaire, asking from each one to comparatively evaluate the nine clusters and express their perception on the relative importance of each cluster with respect to the overall performance and the benefits provided from a candidate PPM information system. Each of the 16 users was requested to perform 36 pairwise comparisons. The judgements of the 11 members of Team I (project managers) were aggregated using the geometric mean in order to obtain the final group decision result. Similarly, the geometric mean was implemented in order to obtain the group weight of Team II. The final clusters’ weights were calculated using the weighted geometric mean (WGM) as shown in formula 2:


ln lnexp

TeamI TeamIITeamI Ci TeamII Ci

CiTeamI TeamII

p w p ww

p p⎛ ⎞+

= ⎜ ⎟⎜ ⎟+⎝ ⎠ (2)

where, Wci, the final weight of the cluster i (i = 1, 2, ..., 9), WciTeamI and Wci

TeamII, the weights of the cluster i according to Teams I and II, respectively and pTeamI and pTeamII, the importance priorities of Team I and Team II, respectively. For the present case the importance priorities were assigned as pTeamI = 0.6 and pTeamII = 0.4. These priorities comprise a strategic decision by the senior management that reflect its perception about the importance of each group (team) towards the final decision-selection. The final weights, as obtained using the WGM, are presented in Figure 3.

Figure 3 Final weights of clusters (see online version for colours)

Step 2 Experts’ assessment on PPM information systems

The final model, as described above, resulted in the identification of five candidate PPM information systems and nine clusters which include 105 criteria (functional features of the NASATM evaluation framework). In order to evaluate the candidate systems in a manageable and reliable way, the experts scored the performance of the PPM information systems with respect to the criteria previously identified. In particular, they utilised an ordinal scoring scale, similar to the one proposed by RefModePM (Ahlemann, 2007). The three experts collectively evaluated each PPM system by providing a general accepted score value on each one of the 105 functional features and they utilised a ‘two-stars’ score (zero-stars: no functionality, one-star: adequate functionality, two-stars: advanced functionality). For example, considering the feature ‘perform resource levelling and smoothing’ of cluster C3 (‘project scheduling support – definition of project task/field features’), all tools received by experts one star (adequate functionality), except from tool B that scored two stars (advanced functionality). Experts in that case justified their ranking as tool B was the only one that allowed the user to define an unlimited number of priority rules for resource levelling. All experts provided a short written justification for every score they gave. Any deviations/missing scores were resolved in a joint meeting held by the three experts and a final consensus was reached on the evaluation of each of the 105 functional features.


The result of the experts’ assessment was an aggregate total score for each system, in each one of the nine clusters. For example, according to cluster C3 (project scheduling support – definition of project task/field features), an aggregate score of system A equal to 21/36 means that the experts perceived that system A scores 21 stars in total of 36 stars (C3 includes 18 features * 2 (maximum number of stars) = 36 stars).

The final scores, as they were given by the experts, for each system in each cluster are shown in Table 1. Thus, Table 1 shows the degree to which each PPM system scores in criteria of each cluster. For instance, for cluster C1 (open database connectivity and architecture – workgroup and networking capabilities), system B ‘gathered’ 17 stars out of 28.

The values of Table 1 were normalised (i.e., in [0...1] scale) by dividing each score by its column total. The normalised matrix is presented in Table 2. This normalisation is necessary in order for the final weighted score of each alternative to be attained. For example, the values of column C1 in Table 1 are 0.75 (= 21/28); 0.61; 0.86; 0.68; 0.79 and its sum equals to 3.68. Thus, the normalised value for alternative A in Table 2 is 0.2039 (= 0.75/3.68). This table depicts the final normalised scores of all PPM information systems according to experts’ judgements. Table 1 Final scores given by the experts

Clusters PPM information systems C1 C2 C3 C4 C5 C6 C7 C8 C9

A =21/28 =5/20 =21/36 =13/24 =13/22 =9/14 =20/42 =3/12 =7/12

B =17/82 =13/20 =23/36 =14/24 =13/22 =2/14 =23/42 =4/12 =7/12

C =24/28 =12/20 =22/36 =8/24 =14/22 =10/14 =10/42 =6/12 =6/12

D =19/28 =10/20 =23/36 =13/24 =14/22 =10/14 =21/42 =4/12 =6/12

E =22/28 =10/20 =23/36 =12/24 =12/22 =10/14 =23/42 =7/12 =6/12

Table 2 Normalised scores of dPPM information systems

Clusters PPM information systems C1 C2 C3 C4 C5 C6 C7 C8 C9

A 0.2039 0.1000 0.1875 0.2167 0.1970 0.1915 0.2062 0.1250 0.2188

B 0.1650 0.2600 0.2054 0.2333 0.1970 0.1702 0.2371 0.1667 0.2188

C 0.2330 0.2400 0.1964 0.1333 0.2121 0.2128 0.1031 0.2500 0.1875

D 0.1845 0.2000 0.2054 0.2167 0.2121 0.2128 0.2165 0.1667 0.1875

E 0.2136 0.2000 0.2054 0.2000 0.1818 0.2128 0.2371 0.2917 0.1875

Step 3 Aggregation of users’ and experts’ judgements

In order to aggregate users’ and experts’ judgements into one final solution, the formula (1) presented in Section 2 is utilised. For the alternative A the final weighted score is calculated as follows:

1 1 2 2 3 3 4 4 5 5 6 6 7 7

8 8 9 9

A c c c c c c c c c c c c c c

c c c c

W w A w A w A w A w A w A w Aw A w A

= × + × + × + × + × + × + ×

+ × + × (3)


where WA, the final weighted score of the alternative A, wci, the weight of the cluster i

(i = 1, 2, ..., 9) according to users’ judgements (obtained from Figure 3), Aci, the score of the alternative A for the criteria included in cluster i (i = 1, 2, …, 9) according to the experts (obtained from Table 2). And by replacing values we get WA = 0,185.

The final weighted scores for all the alternatives are shown in Figure 4, from where it can be deduced that alternative E is the most dominant PPM information system for the present case study.

Figure 4 Final weighted scores of the alternatives (see online version for colours)

0.185

0.2020.197 0.197

0.215

0.1700.1750.1800.1850.1900.1950.2000.2050.2100.2150.220

A B C D E

Figure 5 Final weights of the alternatives after sensitivity analysis (see online version for colours)

Step 4 Sensitivity analysis

Sensitivity analysis is concerned with ‘what-if’ kind of questions to determine if the final answer is stable to changes (experiments) in the inputs, either judgements or priorities. In the present case, sensitivity analysis was performed by changing the priorities. Of special


interest is to see if these changes alter the order of the alternatives. In order to do so the weights of all clusters have been changed by a factor of 30%. Figure 5 depicts the ranking of the alternatives in a scenario where wc1 (the priority of the dominant cluster) would be reduced by a factor of 30%. The graph shows that even during this extreme scenario, information system E remains the dominant alternative. Performing similar sensitivity tests for other clusters led to the conclusion that the outcome is very stable and does not change the overall ranking despite that all the alternatives receive a very close final weight.

5 Conclusions

This paper presented a hybrid model, which utilises both the GAHP and the scoring method, as an answer to the basic research question provided in the abstract, namely ‘how can a decision-making process overcome the limitation of a large number of decision criteria when selecting a PPM information system?’ The proposed hybrid model is argued to be suitable for decision problems that comprise a significant number of criteria (105 for the studied problem) and thus, cannot be handled with typical MCDA methods as AHP because of the great number of pairwise comparisons needed. The proposed method encompasses all the advantages of the GAHP, while overcoming its major limitation concerning the time needed for the comparisons, as the scoring model has been utilised for this purpose. The innovativeness of the proposed approach is twofold. On one hand, it uses a hybrid model that permits consideration of much more criteria than the typical AHP approach can reasonably handle and, on the other hand, it takes into account both experts’ and user’s opinion for making the right decision. Experts can utilise the scoring model to express their judgements on the selection criteria/functional features of PPM systems, while users evaluate pairwise the selection criteria with respect to what they perceive as more important for their project success. Thus, the final selection is based on the aggregation of both experts’ and users’ judgements.

The method presented here, through discussing a real case study, exposes how a group-based, multi-criteria decision analysis approach can be applied to facilitate the selection problem of an appropriate PPM system. All problem stakeholders (i.e., both PPM system experts and PPM system users) have been actively involved in the selection process, so that their expertise could be exploited at a maximum degree. Specifically, the users provided the model with their corporate knowledge concerning the needs of their organisation, while the experts fed the model with the knowledge concerning the information systems functionality.

The results of the model (Figure 3) indicate that system E offers a powerful environment with integrated support for both organisation resource planning and project management services. System B encompasses a widespread project management system that operates at an organisation-level platform, offering portfolio and multi-user management capabilities. It appears that the least promising system is system A. System A is a PPM information system specialised mainly in IT service portfolio and financial management functionalities, with low technical support responsiveness in the national market. As far as the evaluation of system C is concerned, its main advantageous characteristic is the strong integration with database management applications (i.e., it received, from the experts, the highest score in cluster C1 ‘open database connectivity


and architecture – workgroup and networking capabilities’). Finally, one of the strongest advantages of system D, which was weighted equal to C, is the support for quantitative risk management techniques. Finally, the sensitivity analysis performed for the specific case study implies that the output of the model is rather stable, since the ranking of the alternatives is preserved.

This study raises several important issues that could spark further research. An interesting idea could be the exploration of the sensitivity concerning the ‘star’ scoring model. Treating with uncertainties (e.g., by applying a fuzzy AHP approach) would strengthen the proposed approach in deriving more precise results. The authors also plan to further validate the method’s implications and applicability in other case problems addressing the selection of other types of software packages. Although not applicable in the case study presented here, dependencies among criteria may be present in other cases, thus such an issue could be handled by applying a more extended decision-making technique (e.g., the ANP).

Acknowledgements

The authors would like to thank Mr. George Kakarontzas and Mr. Athanasios Tzikas for being supportive in the application of the approach described in this paper. The authors also acknowledge the enthusiastic involvement of the personnel of the Hellenic National Documentation Centre in the empirical part of the research. Finally, the authors wish to thank the editor and the anonymous reviewers for their constructive critique that helped the improvement of this paper.

References

Aczel, J. and Saaty, T. (1983) ‘Procedures for synthesising ratio judgements’, Journal of Mathematical Psychology, Vol. 27, No. 1, pp.93–102.

Ahlemann, F. (2007) Project Management Software Systems – Requirements, Selection Processes and Products, A study by the European Business School, 5th ed., Würzburg, BARC.

Ahlemann, F. (2009) ‘Towards a conceptual reference model for project management information systems’, International Journal of Project Management, Vol. 27, No. 1, pp.19–30.

Ahmad, N. and Laplante, P. (2006) ‘Software project management systems: making a practical decision using AHP’, Proceedings of the 30th Annual IEEE/NASA Software Engineering Workshop, pp.76–84.

Ahmad, N. and Laplante, P. (2007) ‘Reasoning about software using metrics and expert opinion’, Innovations in Systems and Software Engineering, Vol. 3, No. 4, pp.229–235.

Al Harbi, K.M. (2001) ‘Application of AHP in project management’, International Journal of Project Management, Vol. 19, No. 4, pp.19–27.

Ali, A.S.B. and Money, W.H. (2005) ‘A study of project management system acceptance’, Proceedings of the 38th Hawaii International Conference on System Sciences, IEEE Press, pp.234–244.

Bonner, B. (2004) ‘Expertise in group problem Solving: recognition, social combination and performance’, Group Dynamics: Theory, Research, and Practice, Vol. 8, No. 4, pp.277–290.

Bryson, N. and Joseph, A. (1999) ‘Generating consensus priority point vectors: a logarithmic goal programming approach’, Computers & Operations Research, Vol. 26, No. 6, pp.637–643.


Cebeci, U. (2009) ‘Fuzzy AHP-based decision support system for selecting ERP systems in textile industry by using balanced scorecard’, Expert Systems with Applications, Vol. 36, No. 5, pp.8900–8909.

Chang, D.Y. (1996) ‘Application of extent analysis method on fuzzy AHP’, European Journal of Operational Research, Vol. 95, No. 3, pp.649–655.

Davis, F.D. (1989) ‘Perceived usefulness, perceived ease of use and user acceptance of information technology’, MIS Quarterly, Vol. 13, No. 3, pp.319–340.

De Wit, J. and Herroelen, W.S. (1990) ‘An evaluation of microcomputer-based software packages for project management’, European Journal of Operational Research, Vol. 49, No. 1, pp.102–139.

Forman, E. and Peniwati, K. (1998) ‘Aggregating individual judgements and priorities with the AHP’, European Journal of Operational Research, Vol. 108, No. 1, pp.165–169.

Jadhav, A.S. and Sonar, R.M. (2009) ‘Evaluating and selecting software packages: a review’, Information and Software Technology, Vol. 51, No. 3, pp.555–563.

Karaarslan, N. and Gundogar, E. (2008) ‘An application for modular capability-based ERP software selection using AHP method’, The International Journal of Advanced Manufacturing Technology, Vol. 42, Nos. 9–10, pp.1433–3015.

Karlsson, J. and Ryan, K. (1997) ‘A cost-value approach for prioritizing requirements’, IEEE Software, Vol. 14, No. 5, pp.67–74.

Kirytopoulos, K., Leopoulos, V. and Voulgaridou, D. (2008a) ‘Supplier selection in pharmaceutical industry: an analytic network process approach’, Benchmarking: An International Journal, Vol. 15, No. 4, pp.494–516.

Kirytopoulos, K., Leopoulos, V. and Voulgaridou, D. (2008b) ‘Integrating multicriteria decision making and multiobjective optimization methods for supplier selection’, Proceedings of the 15th International Working Seminar on Production Economics, Vol. 3, pp.211–222.

Kontio, J. (1996) ‘A case study in applying a systematic method for COTS selection’, Proceedings of the 18th International Conference on Software Engineering, IEEE Press, pp.201–209.

Levy, J. and Taji, K. (2007) ‘Group decision support for hazards planning and emergency management: a group analytic network process (GANP) approach’, Mathematical and Computer Modelling, Vol. 46, Nos. 7–8, pp.906–917.

Liberatore, M.J. and Pollack-Johnson, B. (2003) ‘Factors influencing the usage and selection of project management software’, IEEE Transactions on Engineering Management, Vol. 50, No. 2, pp.164–174.

Light, M. and Stang, D.B. (2008) ‘Magic quadrant for IT project and portfolio management’, Gartner RAS Core Research Note, G00157924, Gartner Research.

Lozano-Tello, A. and Gomez Perez, A. (2002) ‘BAREMO: how to choose the appropriate software component using the analytic hierarchy process’, Proceedings of the International Conference on Software Engineering and Knowledge Engineering (SEKE), pp.781–788.

Maiden, N.A. and Ncube, V. (1998), ‘Acquiring COTS software selection requirements’, IEEE Software, Vol. 15, No. 2, pp.45–56.

Maroto, C. and Tormos, P. (1994) ‘Project management: an evaluation of software quality’, International Transactions in Operational Research, Vol. 1, No. 2, pp.209–221.

NASA PMTWG (2004) NASA Project Management Tool Analysis and Recommendations White Paper, Project Management Tool Working Group, Principal Center for Workgroup HW and SW, NASA Glenn Research Center, available at km.nasa.gov/pdf/54927main_pm-tool-paper.pdf.

Ngai, E.W.T. and Chan, E.W.C. (2005) ‘Evaluation of knowledge management systems using AHP’, Expert Systems with Applications, Vol. 29, No. 4, pp.889–899.

Percin, S. (2008) ‘Using the ANP approach in selecting and benchmarking ERP systems’, Benchmarking: An International Journal, Vol. 15, No. 5, pp.630–649.


PMI (1999) Project Management Software Survey, Project Management Institute, Newtown Square.

PMI (2008) A Guide to the Project Management Body of Knowledge, Project Management Institute, Newtown Square.

Raymond, L. and Bergeron, F. (2008) ‘Project management information systems: an empirical study of their impact on project managers and project success’, International Journal of Project Management, Vol. 26, No. 2, pp.213–220.

Rushinek, A. and Rushinek, S. (1991) ‘A product evaluation and selection system for project management software’, Computers in Industry, Vol. 16, No. 3, pp.289–301.

Saaty, T. (1980) The Analytic Hierarchy Process, McGraw Hill, New York. Saaty, T. (1989) ‘Group decision making and the AHP’, in Golden, B., Wasil, E. and Harker, P.

(Eds.): The Analytic Hierarchy Process: Applications and Studies, Springer-Verlag, New York.

Saaty, T.L. (1996) Decision Making with Dependence and Feedback: The Analytic Network Process, RWS Publications, Pittsburgh.

Tindale, R. (2003) ‘Group decision making’, in Hogg, M. and Cooper, J. (Eds.): Sage Handbook of Social Psychology, Sage Publications, London.

Tyler, T. and Smith, H. (1998) ‘Social justice and social movements’, in Gilbert, D., Fiske, S. and Lindsey, G. (Eds.): The Handbook of Social Psychology, McGraw Hill, Boston.

Vaidya, O.S. and Kumar, S. (2006) ‘Analytic hierarchy process: an overview of applications’, European Journal of Operational Research, Vol. 169, No. 1, pp.1–29.

Voulgaridou, D., Kirytopoulos, K. and Leopoulos, V. (2009) ‘An analytic network process approach for sales forecasting’, Operations Research: an International Journal, Vol. 9, No. 1, pp.35–53.

Wang, R. (2007) Introducing Project-based Solutions – Enterprise Solutions for Project-based and People-Centric Businesses, Forrester Research, Inc.

White, D. and Fortune, J. (2002) ‘Current practice in project management – an empirical study’, International Journal of Project Management, Vol. 20, No. 1, pp.1–11.

Documents

A case study for project and portfolio management ... recent survey (Raymond and Bergeron, 2008) demonstrates that project managers are starting to realise the benefits obtained from