R&D evaluation methodology basedon group-AHP with uncertainty

Alberto Garinei1,*, Emanuele Piccioni2, Massimiliano Proietti3, Andrea Marini3, Stefano Speziali3,Marcello Marconi1, Raffaella Di Sante4, Sara Casaccia5, Paolo Castellini5, Milena Martarelli5, NicolaPaone5, Gian Marco Revel5, Lorenzo Scalise5, Marco Arnesano6, Paolo Chiariotti7, Roberto Montanini8,Antonino Quattrocchi8, Sergio Silvestri9, Giorgio Ficco10, Emanuele Rizzuto11, Andrea Scorza12, MatteoLancini13, Gianluca Rossi2, Roberto Marsili2, Emanuele Zappa7, Salvatore Sciuto12

1Department of Engineering Sciences, Guglielmo Marconi University, Rome, Italy2Department of Engineering, University of Perugia, Perugia, Italy3Idea-re S.r.l., Perugia, Italy4Department of Industrial Engineering-DIN, University of Bologna, Forlı, Italy5Universita Politecnica delle Marche, Dipartimento di Ingegneria Industriale e Scienze matematiche(DIISM), Ancona, Italy6Universita Telematica eCampus, Novedrate (CO), Italy7Department of Mechanical Engineering, Politecnico di Milano, Milan, Italy8Department of Engineering University of Messina, Messina, Italy9Research Unit of Measurements and Biomedical Instrumentation, Campus Bio-Medico University of Rome,Rome, Italy10Department of Civil and Mechanical Engineering (DICEM), University of Cassino and Lazio Meridionale,Cassino (FR), Italy11Department of Mechanical and Aerospace Engineering, Sapienza, University of Rome, Rome, Italy12Department of Engineering, University of Roma Tre, Rome, Italy13Department of Mechanical and Industrial Engineering, University of Brescia, Brescia, Italy*Corresponding author: a.garinei@unimarconi.it

Abstract

In this paper, we present an approach to evaluate Research & Development (R&D) performancebased on the Analytic Hierarchy Process (AHP) method. Through a set of questionnaires submittedto a team of experts, we single out a set of indicators needed for R&D performance evaluation. Theindicators, together with the corresponding criteria, form the basic hierarchical structure of theAHP method. The numerical values associated with all the indicators are then used to assign ascore to a given R&D project. In order to aggregate consistently the values taken on by the differentindicators, we operate on them so that they are mapped to dimensionless quantities lying in a unitinterval. This is achieved by employing the empirical Cumulative Density Function (CDF) for eachof the indicators. We give a thorough discussion on how to assign a score to an R&D projectalong with the corresponding uncertainty due to possible inconsistencies of the decision process. Aparticular example of R&D performance is finally considered.

Keywords: AHP, Multi-Criteria Decision Making, R&D performance, R&D measures

arX

iv:2

108.

0259

5v1

[cs

.CY

] 5

Aug

202

1

1. Introduction

The Analytic Hierarchy Process (AHP) is a Multi-Criteria Decision Making (MCDM) method developedby Saaty in the 1970’s (Saaty (1977)). It provides asystematic approach to quantifying relative weights ofdecision criteria.

Its strength relies on the fact that it allows to de-compose a decision problem into a hierarchy of sub-problems, each of which can be analyzed independentlyin a similar manner. It is used in a wide variety of deci-sion situations, in fields like education, industry, health-care and so on.

In this paper, we propose a method to evaluate Re-search and Development (R&D) performance, based ongroup-AHP, through the introduction of a “score” as-signed to each R&D projects in a given set.

R&D represents the set of innovative activities un-dertaken by companies and/or governments to developnew and more efficient services or products as well as toimprove the existing ones. It has become somewhat cru-cial to have a systematic method to evaluate the perfor-mance a given project or research activity (Lazzarottiet al. (2011)). See, among others, also (Kerssens-vanDrongelen & Bilderbeek (1999)), (Moncada-Paterno-Castello et al. (2010)), (Tidd et al. (2000)), (Griffin(1997)), (Bremser & Barsky (2004)), (Jefferson et al.(2006)), (Kim & Oh (2002)), (Kaplan et al. (1996)),(Chiesa et al. (2009)) and references therein for the im-portance of R&D performance assessment. Quantita-tive methods coupled with qualitative assessments areused in decision support systems, for example by projectfunding commissions.

However, there are currently no standards for measur-ing the performance of an R&D project. The methoddeveloped in this paper stems from a critical approachto the measurement problem concerning complex sys-tems (such as Research and Development). With thehelp of group multi-criteria methodologies, we tried tofaithfully represent the evaluations of R&D projectsthrough the involvement of stakeholders. As a matterof fact, the latter represent diverse interests, and belongto different domains of knowledge.

We used three questionnaires addressed to stakehold-ers at different stages of the process with the ideal goalof developing a shared decision support tool that iseasy to use and whose operation can be directly ex-plained. In view of adopting the logic of the metrolog-ical method, we defined a model capturing the subtlefeatures of R&D performance evaluation and keepingtrack of measurements uncertainties.

In order to introduce the standard AHP decisionstructure, we need to define precisely what our criteriaand sub-criteria will be. Criteria (or perspective in our

parlance) are selected following the existing literatureand, more in detail, have been identified to be: Inter-nal Business perspective, Innovation and Learning per-spective, Financial perspective, Network and Alliancesperspective.

We then single out a set of sub-criteria (indicators)through a set of questionnaires submitted to a team ofexperts selected from academia or private research hubsin Italy. The indicators, along with the correspondingcriteria, will form in our analysis the basic hierarchicalstructure of the AHP method.

In order to have a sensible way to aggregate the valuesthe different indicators take on, we operate on them insuch way they share the same scale, namely they are alldimensionless quantities varying over the same range,which for convenience we choose to be 0 to 1. This isattained by employing as transformation map for eachthe indicator the corresponding empirical CumulativeDensity Function (CDF). In this way, all the resultingvariables are approximately uniformly distributed overthe unit interval.

It is well-known that decision processes in complexsystems carry along judgmental inconsistencies. Awareof the fact that some inconsistencies are difficult to getrid of, we propose a rigorous method to quantify the un-certainty affecting the “score” of a given R&D project.In order to better show how our method works, we givean example of application in the last section of this pa-per. The method has been employed to evaluation ofR&D projects whose data are stored in the DPR&DI(Digital Platform for R&D and Innovation Projects).

This paper is organized as follows. In Section 2 wediscuss in detail the basics of the AHP method as de-veloped originally. In Section 3 we propose a methodto choose the criteria and sub-criteria to evaluate R&Dperformance through a set of questionnaires. We thengive a detailed and precise account on how to evaluateR&D performance of a given project, and finally we dis-cuss the consistency of the proposed method. We givean example of R&D performance evaluation in Section4 and our conclusions in Section 5.

2. Theoretical Background: the AHPmethod

In this section, we discuss the basics of the AHP (An-alytic Hierarchy Problem) method as developed origi-nally by Saaty in the 1970’s. More details can be found,for example, in the book (Saaty (2010)) or in the review(Ishizaka & Labib (2011)).

1

2.1 Decision problems

We face many decision problems in our daily lives. Theycan be as simple as deciding what jeans we want to buyor more involved, like what person to hire for a post-docposition. Whatever decision problem we are facing, asystematic way to deal with it can be useful, and thisis where AHP comes to play a role.

In AHP, each decision problem can be broken down inthree components, each with the same basic structure:

• The goal of the problem, namely the objective thatdrives the decision problem.

• The alternatives, namely the different options thatare being considered in the decision problem.

• The criteria, namely the factors that are used toevaluate the alternatives with respect to the goal.

Moreover, if the problem requires it, we can asso-ciate sub-criteria to each criterion, adding extra layersof complexity. We will see an example of this in Section4.

The three levels (or more if we consider sub-criteria)define a hierarchy for the problem, and each level canbe dealt with in a similar fashion to the others. Thisis essentially the basic structure of the AHP method indecision problems. The rest of this section is devoted tospelling out the details of how a decision is eventuallymade.

2.2 Weighting the problem

A crucial ingredient in any decision problem is the map-ping of notions, rankings etc. to numerical values. Ba-sic examples of mappings are scales of measurements,like the Celsius-degree for the temperature or dollars formoney. In these cases we have what are called standardscales, where standard units are employed to determinethe weight of an object.

However, it often happens that the same number (say100°) means different things to different people, accord-ing to the situation, or different numbers are as good(or as bad) for a given purpose (e.g. when trying tofind the right temperature for a fridge 100° is as bad as−100°). Moreover, it might be the case that we need toanalyze processes for which there is no standard scale.Thus, we need to find a way to deal with these situationsconsistently.

It turns out that what really matters is pairwise com-parisons between different options. In this way we cancreate a relative ratio scale and, in fact, here is the cruxof the AHP method, as we will see in a moment.

In the case we are dealing with a standard scale, wecan assign to n objects n weights w1, . . . , wn. Then, we

can create a matrix1 A ∈ Rn×n of pairwise comparisonsin the following way

A =

w1/w1 w1/w2 · · · w1/wn

w2/w1 w2/w2 · · · w2/wn

......

. . ....

wn/w1 wn/w2 · · · wn/wn

. (2.1)

The matrix A is an example of a reciprocal matrix, i.e.a matrix where each entry satisfies aij = 1/aji. Thisis indeed what we would expect when there is an un-derlying standard scale. For example, if we are are todetermine which among two apples is the reddest and,according to a given scale, apple a is twice as red asapple b, it necessarily follows that apple b is one-half asred as apple a.

Note the following interesting fact, that will be rel-evant for us later. If we define the vector w =(w1, . . . , wn)T it is easily seen that

A · w = nw , (2.2)

where the dot-product is just matrix product, i.e. wis an eigenvector of A with eigenvalue n. In fact, it israther easy to convince ourselves that the matrix A ineqn. (2.1) has rank 1 and a theorem in linear algebratells us that it must have only one non-zero eigenvalue.On the other hand, the trace of a matrix gives the sumof it eigenvalues which in our case turns out to be 1 +· · · + 1 = n. It is therefore coherent to conclude thata consistent matrix like A above has only one non-zeroeigenvalue, n. In this case n is also called the principaleigenvalue, i.e. the largest of the eigenvalues of a squarematrix.

As we said before, sometimes we have to deal withdecision processes where a standard scale does not existand thus we are not given a priori a weight vector w.What is really meaningful in this case is the matrix ofpairwise comparisons between alternatives, similar tothat in eqn. (2.1)

A = (aij) =

1 a12 · · · a1na21 1 · · · a2n...

.... . .

...an1 an2 · · · 1

. (2.3)

Here aij tells us how the i-th object compares to thej-th object according to a give criterion/goal. Notice

1In this paper we deal mainly with finite dimensional real vec-tor spaces. In particular, if V and W are vector spaces of dimen-sions n and m respectively, a choice of bases v = {v1, . . . , vn} andw = {w1, . . . , wm} determines isomorphisms of V and W with Rn

and Rm, respectively. Any linear operator from V to W has a ma-trix presentation A ∈ Rn×m with respect to the given bases. Inthis respect, the eigenvalue eqn. (2.2) is a linear transformationfrom a space to itself.

2

that also in this case we should impose aij = 1/aji, i.e.we should have a reciprocal matrix, but now each entryis not given by a ratio of two quantities.

In order to make the pairwise-comparison coefficientsaij as explicit as possible, the Saaty’s 1-9 scale is oftenused (see Figure 1). The scale should be read in thefollowing way: If an object i is as important as theobject j, then we should set aij = 1. If, instead objecti is more important than the object j, then aij shouldbe set to 3, 5, 7 or 9, following the scheme in Figure1. Also the intermediate even values (2, 4, 6, 8) can beused and allow for finer assessments.

Lev

elof

imp

ort

ance

1 ← Equal importance

3 ← Moderate importance

5 ← Essential or strong importance

7 ← Very strong importance

9 ← Extreme importance

2

4

6

8

Figure 1: Saaty’s 1-9 scale.

What if we considered an eigenvalue equation also forthe matrix A defined in (2.3)? And what would be themeaning of the weights (priorities) wi in this case? Letus begin by answering the first question first.

The Perron-Frobenius theorem tells us that there ex-ists one principal eigenvalue, λmax, and that it is unique.We then find an equation of the form

1 a12 · · · a1na21 1 · · · a2n...

.... . .

...an1 an2 · · · 1

w1

w2

...wn

= λmax

w1

w2

...wn

(2.4)

It is a theorem (Saaty (1990)) that for a reciprocal n×n matrix with all entries greater than zero, the principaleigenvalue λmax is always greater or equal to n, λmax ≥n. In particular, λmax = n if and only if A is a consistentmatrix.

What is it meant by consistent matrix? If we reckonthat alternative i is aij times better than alternativej, and the latter is ajk times better than alternativek, we should have, for consistency, aik = aijajk. Thisis know as multiplicative consistency. It is easily seenthat multiplicative consistency implies reciprocity, butthe converse is not true.

It is often the case that multiplicative consistency isnot respected, introducing some form of inconsistency

in the evaluation process. One major drawback, forexample, is that the fundamental scale ranges from 1/9to 9 and a product of the form aijajk might very wellbe outside the scale, making it impossible to respectmultiplicative consistency.2 In the next subsection, wewill see how to manage possible inconsistencies.

In order to have a (nearly) consistent matrix of pair-wise comparisons A, λmax should not differ much fromthe dimension of A, n. In particular, finding theeigenvector w = (w1, . . . , wn)T amounts to finding theweights (or priorities) of the n objects (alternatives),and we are assured that, if the matrix A is sufficientlyconsistent, aij ≈ wi/wj . Note that multiplying bothsides of eqn. (2.4) by an arbitrary constant is harmless,and therefore the vector w can be conveniently normal-ized as we please. We will have to say a little more onthis below.

2.3 How to compute weights

We now find ourselves in the position where we shoulddetermine the priority vector w, eqn. (2.4), once a pair-wise comparison matrix is given. The easiest way to doso is to solve eqn. (2.4) using standard methods in lin-ear algebra. However, general procedures are not alwaysexempt from inconsistencies (in AHP). For example, forinconsistent matrices with dimension greater than 3,there is a right-left asymmetry, i.e a right-eigenvectoris not a left-eigenvector.

In order to avoid this issue, a common alternative tocompute the priority vector w makes use of the loga-rithmic least squares (LLS) method (De Jong (1984)),(Crawford & Williams (1985)). The relation betweenthe matrix pairwise comparison A and the relative pri-ority vector w can be expressed as

aij =wi

wjεij , i, j = 1, . . . , n , (2.5)

where εij are positive random perturbations. It is com-monly accepted that for nearly consistent matrices theεij factor is log-normal distributed3. Thus, to deter-mine the weights wi one can take the logarithm of (2.5)

2There are different approaches to deal with the problem ofthe scale range. One approach could be to change the linear scalegiven before to a more convoluted one. For example in (Doneganet al. (1992)) an asymptotic scale is employed so that we neverget out of a prefixed scale range. However, in the literature, thelinear scale of Saaty seems to be the most widely used scale.

3Indeed, the authors of (Shrestha & Rahman (1991)) foundout that the error factors εij best describe the inconsistency inthe decision process when they are log-normal distributed

log εij ∼ N (0, σ2ij) , (2.6)

where N (µ, σ2) is the normal Gaussian distribution function withmean µ and variance σ2. In particular, note that the mean valueof the error factor εij is 1 and its range can be varied by choosingσ2ij accordingly with the degree of expertise.

3

and then apply the least square principle, namely min-imizing the sum of squares of log εij ,

E(w) =

n∑i,j=1

(log aij − log(wi) + log(wj))2. (2.7)

An easy computation reveals that E(w) is minimizedwhen

wi =

n∏j=1

aij

1n

, i = 1, . . . , n . (2.8)

This is also called the geometric mean, and from nowon we will adopt this method to compute weights. Notethat for consistent matrices, wi as in (2.8) is an eigen-vector with eigenvalue n. The weights wi in eqn. (2.8)are defined up to a multiplicative constant (see eqn.(2.7)). We have normalized them so that

∏nj=1 wj = 1.

2.4 Aggregation

The final step is to aggregate local priorities across allcriteria to in order to determine the global priority ofeach alternative. This step is necessary to determinewhich alternative will be the preferred one.

In the original formulation of AHP, this is done in thefollowing way. If we denote lij the local priority (weight)of the alternative i with respect to the criterion j andwj the weight of the criterion j, the global priority pifor the alternative i is defined to be

pi =∑j

wj lij . (2.9)

Criterion weights and local priorities can be normal-ized so that they sum up to 1. In this way, we find∑

i pi = 1. The alternative getting the highest priority(modulo inconsistencies to be discussed later) will bethe favorite one in the decision process.

Let us now move on to discussing (some of the) pos-sible inconsistencies of the AHP method.

2.5 Consistency of the AHP method

As we remarked before, the AHP method is based onthe idea that there is always some underlying scale ina decision problem. This is encoded in the fact thatwhen we have calculated our weight matrix – which bydefinition is a consistent ratio matrix built out of theweight ratios – this one should not be too far off theoriginal pairwise comparison matrix.

In order to determine how far off we are, we need tofind a way to determine the inconsistency of our deci-sion matrices. To this purpose, it is useful to recall acouple of facts (Saaty (1990)). Saaty noticed that for a

reciprocal n × n matrix A with all entries bigger thanzero, the principal eigenvalue is always equal or greaterthan n. This is easily proved with some simple linearalgebra.

Moreover, it turns out that A is a fully consistentmatrix if and only if the principal eigenvalue is strictlyequal to n.

Given these facts, it is possible to define a set of in-dices to measure the consistency of our decision matri-ces. In particular, we can define the Consistency Index(CI) as

CI =λmax − nn− 1

. (2.10)

Note that CI ≥ 0, as a consequence of what we saidabove. Also, the more CI is different from zero themore inconsistent we have been in the decision process.

We can also define the Random Index RI of size nas the average CI calculated from a large number ofrandomly filled matrices. For a discussion on how thesematrices are created see (Alonso & Lamata (2006)).

Finally, we define the Consistency Ratio CR as theratio CI(A)/RI(A) for a reciprocal n×n matrix, whereRI(A) is the random index for matrices of size n.

Usually, if the CR is less than 10% the matrix is con-sidered to have an acceptable consistency. Nonetheless,this consistency index is sometimes criticized as it al-lows contradictory judgments. See the review (Ishizaka& Labib (2011)) for a discussion about this.

In the literature, several other methods to measureconsistency have been proposed. See (Ishizaka & Labib(2011)) for an account of the existing methods. Forexample, the authors (Alonso & Lamata (2006)) havecomputed a regression of the random indices and pro-posed the following formula

λmax < 1.17699n− 0.43513 , (2.11)

where n is the size of the pairwise comparison matrix,while (Crawford & Williams (1985)) propose to use theGeometric Consistency Index GCI

GCI =2

(n− 1)(n− 2)

n−1∑i=1

n∑j=i+1

[log

(aij

wi/wj

)]2.

(2.12)In the coming sections, we will make extensive use ofthe GCI for the computation of consistency of decisionprocesses as we believe it is more apt to capture thepropagation of inconsistencies.

3. Methodology

In this section, we propose a methodology to evaluateR&D performance. In particular, we discuss in detailhow criteria and sub-criteria are to be chosen in ourproposed method.

4

3.1 Criteria and sub-criteria in R&D perfor-mance evaluation

3.1.1 Perspectives to measure R&D performance

Determining R&D performances usually relies on theidentification of indicators (or metrics) relative to somecriteria (perspectives). Giving the same importance toall indicators and/or criteria can lead to an oversimplifi-cation of the R&D measuring process and this, in turn,may lead to misinterpretation to the actual performanceof an R&D project (Salimi & Rezaei (2018)).

Thus, it is crucial to correctly identify criteria andsub-criteria and subsequently determine relative impor-tance. The latter step can be carried out by asking ateam of experts to make pairwise comparisons betweenalternatives for both perspectives (criteria) and indica-tors (sub-criteria).

Following the literature, for example (Kaplan et al.(1996)), (Bremser & Barsky (2004)), (Lazzarotti et al.(2011)) (Salimi & Rezaei (2018)), we lay out the fourperspectives which are relevant for measuring R&D per-formance:

• Internal Business perspective (IB)

• Innovation and Learning perspective (I&L)

• Financial perspective (F)

• Network and Alliances perspective (N&A)

Let us spell out what each perspective is about.The Internal Business perspective refers to internal re-sources, such as technological capabilities or human re-sources, that influence directly the performance of aproject. The Innovation and Learning perspective refersto the development of new skills as the result of projectactivities. Financial perspective, instead, aims at cap-turing financial aspects of a project, with a focus on fi-nancial sustainability of a project. Finally, the Networkand Alliances perspective refers to the interaction withdifferent partners, such as external companies involvedin project activities and realization of the results.

The authors (Salimi & Rezaei (2018)) consider alsothe “Customer perspective”, which refers to the extentthat R&D satisfies the needs of customers. In the fol-lowing sections, we will be interested mainly in projectswhich do not involve customers. Thus, we will stickwith the four criteria identified above.

The four perspectives presented here will be the fourcriteria of our decision process. Indicators, i.e. sub-criteria, will be associated with each of the criteria in away that we now describe.

3.1.2 Selection of Indicators

Let us briefly outline the three steps we propose areto be taken in order to determine indicators for eachcriterion. These will be labeled Step 0, 1 and 2 and canbe summarized as follows:

• Step 0: Selection of relevant raw data, i.e. thebuilding blocks for the final indicators, through aquestionnaire given to a team of experts.

• Step 1: Identification of the right indicators fromdata selected at Step 0 through a second question-naire.

• Step 2: Pairwise comparisons between perspectives(criteria) and indicators (sub-criteria) according tothe AHP method described in the previous sectionwith some modifications that we describe later.

More in detail, in Step 0 we prepare a list of param-eters (raw data) that will be used to identify the indi-cators for the decision process. The list, an example ofwhich is given in Section 4, is submitted to a team ofexperts who are asked to identify the parameters thatare usually available in the projects they are involved in.This step is necessary to understand which parameters,among the proposed ones, are more versed to capture aproject performance.

In Step 1, we ask the same team of experts to build,out of the raw data selected at Step 0, the indicatorsfor the different perspectives. In particular, each of theparticipants is asked to form a number of normalizedindicators for each perspective. For example, jumpingahead to the example of R&D performance evaluationgiven in Section 4, if we think that the number of find-ings in a given project (each given in a publication orpresented at a conference) in the shortest time is a rel-evant indicator for Innovation and Learning, then wemight propose as indicator: # of findings/total time ofthe project.

If, for any reason, the experts think that some quan-tities do not need to be normalized and can stand ontheir own, they are allowed to choose no denominator.Finally, a set of indicators for each perspective is formedaccording to the consensus they received from the ex-perts.

In Step 2, the team of experts is eventually askedto form pairwise comparison matrices, both betweenall criteria and sub-criteria. Nevertheless, there is animportant caveat. Differently from the original AHPmethod, we require no strict reciprocity: aij shouldnot be necessarily equal to 1/aji, but small (and spo-radic) deviations are allowed. The reason for introduc-ing such an inconsistency is that we would like to de-velop a method capable to capture and bypass possible

5

inconsistencies that often influence decision processes inR&D performance evaluation.

3.2 AHP for evaluating R&D performance

As it should be by now clear, in our method, the cri-teria for R&D performance evaluation are representedby the four perspectives mentioned in the last section,while indicators – relative to each criterion – are thesub-criteria. Different projects in an evaluation sessionmake up the alternatives. In brief, the alternative whichscores the biggest global priority will correspond to themost impactful – as for the chosen criteria – project forR&D.

3.2.1 Pairwise comparisons of perspectives and indi-cators

Let us define the pairwise comparison matrix amongcriteria C ∈ R4×4 in the following manner

C =

c11 c12 c13 c14c21 c22 c23 c24c31 c32 c33 c34c41 c42 c43 c44

. (3.1)

Of course, cii = 1 for i = 1, . . . , 4. The priority vectorv∗ of C can be easily computed as the geometric meanover the columns of C, see eqn. (2.8),

v∗ =

(c11 c12 c13 c14)

14

(c21 c22 c23 c24)14

(c31 c32 c33 c34)14

(c41 c42 c43 c44)14

. (3.2)

It turns out to be useful to our purposes to normalizeit in such a way the sum of its components is 1

v =v∗∑4i=1 v

∗i

. (3.3)

In the same fashion, we can define the pairwise com-parison matrix among sub-criteria A(c) ∈ Rmc×mc ,

A(c) =(a(c)ij

)=

a(c)11 · · · a

(c)1mc

.... . .

...

a(c)mc1

· · · a(c)mcmc

, (3.4)

where c is an index that labels the different criteria (inour case there is 4 of them). We can define, just as inthe case of criteria, the priority vector w(c) for each A(c)

w(c)∗ =

(a11 a12 · · · a1mc)

1mc

(a21 a22 · · · a2mc)1

mc

...

(amc1 amc2 · · · amcmc)1

mc

, (3.5)

and normalize it so that

w(c) =w(c)∗∑mc

i=1 w(c)∗i

. (3.6)

It turns out to be useful to repack the vectors w(c)intoa matrix W ∈ R4×Nind , with Nind =

∑cmc the total

number of indicators, in the following fashion

W =

w(1)T 0 0 0

0 w(2)T 0 00 0 w(3)T 00 0 0 w(4)T

. (3.7)

We can now compute the global weight of the i-thindicator as

Pi = (vTW )i =

4∑j=1

vj Wji , i = 1, . . . , Nind . (3.8)

Note that∑Nind

i=1 Pi = 1 in our normalization. Whenthere is more than one expert the global weight vec-tors for each expert have to be combined so to obtain aunique global weight P (group). We will do this again byconsidering the geometric mean over the experts, i.e weemploy the AIP (Aggregation of Individual Priorities)method rather than the AIJ (Aggregation of IndividualJudgments), see (Dong et al. (2010)),

P(group)i =

∏Nexp

k=1

(P

(k)i

) 1Nexp

∑Nind

j=1

∏Nexp

k=1

(P

(k)j

) 1Nexp

, (3.9)

where k runs over the number of experts, Nexp, and

P(k)i is the global weight vector of the k-th expert.

3.2.2 Evaluating R&D performance

Finally, we need to find a way to determine the prior-ity (or score) of each of the alternatives, i.e. differentprojects in our case.

Each of the indicators in a given project can be mea-sured, in general, by means of a standard scale. Forinstance, “time of a project” (see Section 4) can be eas-ily extrapolated once we know the date of beginningand end of that given project. So it seems natural, inorder to compute the score of each project, to multi-ply the indicator-global-priorities by the correspondingR&D measurement and, in fact, here lies the centralpoint of our method.

Once we have determined the global weight of eachindicator, we should multiply it by its “performance”parameter. For instance, going back to the exampleof # of findings/total time of the project mentioned

6

in the previous section, the higher this number is, ina given project, the better the project itself will per-form in the final evaluation. This will ensure that theproject, among those taken into considerations, with themost performing indicators will be the most valuable forR&D.

However, the alert reader has surely noticed that thiscan lead to a nonsense, as R&D measurement are oftendimensionful quantities and it makes no sense to sumthem up. Thus, what we propose is to “map” each R&Dmeasurement to a dimensionless parameter lying in therange 0 to 1 using the empirical Cumulative Distribu-tion Function (CDF).

We remind the reader that the CDF of a real-valuedrandom variable X is the function given by

FX(x) = P (X ≤ x) , (3.10)

where P (X ≤ x) is the probability that the randomvariable X takes on a value less than or equal to x.Among its properties, we have that the CDF is a non de-creasing function of its argument and right-continuous.In particular, if X is a continuous random variable

limx→−∞

FX(x) = 0 , limx→∞

FX(x) = 1 . (3.11)

In integral form the CDF can also be expressed as

FX(x) =

∫ x

−∞fX(t) dt , (3.12)

where fX(x) can be interpreted as a probability densityfunction for the variable X. It is quite trivial to provethat for a continuous random variable X, the randomvariable Y = FX(X) has a standard uniform distribu-tion.4 Indeed,

FY (y) = P (Y ≤ y) = P (FX(X) ≤ y)

= P (X ≤ F−1X (y)) = FX(F−1X (y)) = y .(3.13)

In practice, we map each R&D measurement variableXi using the corresponding empirical CDF, in place ofthe true unknown CDF, so to obtain variables havingan approximately uniform distribution in the range 0 to1.

4If X is a discrete random variable, then its CDF is given by

FX(x) =∑

xi≤X

P (X = xi) ,

where P (X = xi) is the probability for X to attain the valuexi. Clearly, in this case the map Y = FX(X) does not yield avariable with standard uniform distribution: The resulting vari-able is still discrete and P (Y = y) = P (X = F−1

X (y)). However,if X can take sufficiently many values, it can be approximatelyseen as a continuous variable and also the aforementioned resultapproximately holds.

Thus, the final R&D performance can be computedby means of the following formula5

SR&D =

Nind∑i=1

P(group)i FXi(xi) (3.14)

Note that FXi(xi) ≤ 1 for any i = 1, . . . , Nind. There-

fore, SR&D ≤∑Nind

i=1 P(group)i = 1. Thus we conclude

that the R&D performance for each project is alwaysnormalized to lie in the range 0 to 1:

0 ≤ SR&D ≤ 1 . (3.15)

3.3 Consistency of the method

In AHP we are asked to make comparisons between eachpair among the alternatives. Even though in ideal sit-uations there would not be any inconsistencies, in realsituations our decisions are subject to judgmental errorsand conflicting with each other to some extent.

In the following we will stick with the assumption thaterror factors are log-normal distributed with 0 mean.Let us then proceed to estimate what the variance in ageneric R&D performance evaluation is going to be forus.

3.3.1 Uncertainty in R&D performance

As just remarked, it is commonly accepted that in-consistencies are log-normal distributed. For example,(Shrestha & Rahman (1991)) found that for a pairwisecomparison matrix of dimension n the variance of theerror σ2 is well approximated by the formula (2.12),that we report here for clarity,

σ2 =2

(n− 1)(n− 2)

n−1∑i=1

n∑j=i+1

[log

(aij

wi/wj

)]2,

(3.16)where aij is the pairwise comparison matrix and wi thecomponents of the corresponding priority vector.

In our case, at the level of the four criteria (thefour perspectives mentioned in the previous section) wewould find an error of the form

σ2 =1

3

3∑i=1

4∑j=i+1

[log

(cijvi/vj

)]2(3.17)

5We have assumed throughout that the larger an indicatorperformance is the more it will contribute to R&D performance,SR&D. It might very well be that exactly the opposite happensfor a given indicator: the smaller an indicator is the better it isin terms of performance. In that case, it is enough to replaceFXi

(xi) by 1− FXi(xi).

7

while for each of the sub-criteria we find

σ(c)2 =2

(mc − 1)(mc − 2)

mc−1∑i=1

mc∑j=i+1

[log

(a(c)ij

w(c)i /w

(c)j

)]2(3.18)

where, again, c is an index labeling each of the criteriaand mc is the number of sub-criteria for the criterion c.In a similar fashion, in (Eskandari & Rabelo (2007)) itis argued that the variances associated with each localweight are given by

σ2vi =

15

16

4∑j=1

v2j − v2i

σ2v2i (3.19)

for the case of the four criteria, while it is of the follow-ing form

σ2

w(c)i

=m2

c − 1

m2c

mc∑j=1

w(c)j

2− w(c)

i

2

σ(c)2w2i (3.20)

for the case of the sub-criteria. Note that we are as-suming no correlation among different criteria or sub-criteria. In this way we can also repack the errors ofeqn. (3.20) in the following 4×Nind matrix

σ2W =

(σ2w(1)

)T0 0 0

0(σ2w(2)

)T0 0

0 0(σ2w(3)

)T0

0 0 0(σ2w(4)

)T

(3.21)Given that we are interested in estimating the final

error affecting SR&D for each of the projects, it is nec-essary to see how the uncertainties propagate. In par-ticular, the variance error for the global weight of anindicator (for each of the experts) is found to be

σ2Pi

=4∑

j=1

(σ2vjW

2ji + v2j (σW )2ji

), i = 1, . . . , Nind .

(3.22)Note that, in order to derive eqn. (3.22), we assumedthat the uncertainty affecting the criteria and sub-criteria are independent of each other. Finally, in orderto estimate the error affecting the global weight of an in-dicator for the total group of experts we use the generalformula (see for instance (Bevington et al. (1993)))

σ2

P(group)i

=

Nexp∑l=1

Nind∑j=1

(∂P

(group)i

∂P(l)j

)2

σ2

P(l)j

, (3.23)

where the derivatives are easily computed from eqn.(3.9) to be

∂P(group)i

∂P(l)j

=P

(group)i

(δij − P (group)

j

)NexpP

(l)j

. (3.24)

Here δij is the kronecker delta: δij = 1 if i = j and 0otherwise. The uncertainty on the final outcome SR&D

is easily evaluated to be (the x’s are assumed to haveno associated statistical error)

σ2SR&D

=

Nind∑i=1

σ2

P(group)i

F (xi)2 . (3.25)

4. Application of the method andResults

In this section, we apply our methodology to R&D per-formance of 34 projects stored in the DPR&DI (DigitalPlatform for R&D and Innovation Projects).

The DPR&DI is a PaaS (Platform as a Service)for the management of R&D and industrial innovationprojects. It allows to monitor in real time the progressof any project, the storage of information and sharing ofdata. It can also be used to create connections betweenthe various parties involved in the innovation process,creating a shared space for collaboration that connectsresearchers, innovators, institutions and funding agen-cies6.

We discuss in detail the various steps to find a projectperformance by applying the general procedure ex-plained in the previous sections.

4.1 Step 0

First of all, we lay out the raw data (Step 0) that wereckoned were necessary to build meaningful indicatorsto evaluate the R&D performance of the projects in theDPR&DI.

Let us start off by giving all the quantities that webelieve are relevant to characterize the magnitude of aproject

6It has been developed by Idea-re S.r.l. under the grant de-livered by the Umbria Region “POR FESR 2014-2020. Asse IAzione 1.3.1. Sostegno alla creazione e al consolidamento di start-up innovative ad alta intensita di applicazione di conoscenza e alleiniziative di spin-off della ricerca”

8

– Duration of the project

– Number of calls for tenders

– Number of partners involved in the project

– Number of project activities

– Number of people involved in the project

– Number of people with an education ap-propriate for the given topic

– Time spent on the project

– Equipment usage time

Project

Second, we believe the impact on R&D is driven alsoby the amount of findings for a given project. Thus, weproposed to consider also the following quantities:

– Number of findings (papers, books, confer-ences, exhibitions, others)

– Number of papers for a given project

– Number of books for a given project

– Number of conferences attended to presenta given result

– Number of exhibitions attended to presenta given result

– Number of patents for a given project

Findings

Moreover, it is crucial to have indicators measuringthe total costs of a given project, especially in order toquantify the sustainability of the project itself. Thus,we introduce raw data also for detailing financial re-porting:

– Total cost of the project

– Total cost of the project team

– Total cost of equipment

– Total cost of external suppliers

– Total cost of consultants

Financial Reporting

and financial support:

– Grant eligible expenses

– Tax credit eligible expenses

Financial Support

At this point, a team of experts was asked to givea ranking of the raw data just given in order to forma coherent set of indicators. In particular, this led usto Step 1, where raw data are combined to form theindicators, as explained in Section 3.1.

4.2 Step 1

As already anticipated, a statistical analysis made overthe experts’ opinions has led to a set of indicators thatcan be used to evaluate R&D performance. These arereported in Table 1. As we can see, there are 5 in-dicators for the Internal and Business perspective, 6for Innovation and Learning, 5 for the Financial per-spective and 4 for Alliances and Network perspective.We have thus created a layer of 20 indicators (sub-criteria), each associated with a given perspective (cri-terion). This, along with the 34 project considered inthis study, makes up the basic AHP structure in theR&D performance evaluation.

4.3 Step 2

We are now ready, as for Step 2, to compute the R&Dperformance for the 34 selected projects using formulasspelled out in Section 3.2.

In particular, the distribution for the R&D perfor-mance scores is depicted in Fig. 2a. We can see thatthe distribution is quite uniform, and all scores lie (ap-proximately) in the range 0.6 to 0.8 (remember that theSR&D is normalized to be in the range 0 to 1).

As for the consistency of our results we can employthe formula (3.25). Scores with errors are shown inFig. 2b.

9

Table 1: Indicators selected by the team experts consulted to evaluate R&D performance.

Perspective Indicators

Internal Business Perspective

Number of findings / Cost of the projectNumber of people in the project / Project durationGrant eligible expensesTime spent on the project / Number of people involvedTime spent on the project / Number of activities

Innovation and Learning perspective

Number of papers / Number of people in the projectNumber of books / Number of people in the projectNumber of patents / Total cost of the projectNumber of findings / Duration of the ProjectNumber of papers / Total cost of the projectNumber of findings / Time spent on the project

Financial perspective

Total cost of the team / Total cost of the projectTotal cost of the suppliers / Total cost of the projectTotal cost of equipment / Total cost of the projectGrant eligible expenses / Total cost of the projectNumber of patents / Total cost of the projectTime spent on the project / Total cost of the project

Alliances and Networks perspective

Number of partnersNumber of partners / Time spent on the projectNumber of project activities / Total cost of suppliersNumber of patents / Number of suppliers

0.60 0.65 0.70 0.75 0.80Score

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Prob

abilit

y

(a) Histogram of the score distribution. On the y-axis we havethe relative probability of finding a given score (x-axis).

0 5 10 15 20 25 30 35Project id

0.0

0.2

0.4

0.6

0.8

1.0

Scor

e

(b) Scores with error bars.

Figure 2: Score distribution for 34 projects stored in the DPR&DI.

As we can see, the σ2 on any given project is quitesignificant, making it hard to identify precisely whichproject performs best in this particular analysis. Thisis essentially due, as we would expect, to the degree ofinconsistency allowed when forming the pairwise com-parisons. What could be nice to do is to compute the

probability of inversion of two given projects in the finalranking. We leave issues like this for future studies.

10

5. Discussion and Conclusions

In this paper we considered a new approach to de-termine R&D performance based on the group-AHPmethod. As explained thoroughly in the main text, theAHP method is a powerful method that allows to quan-tify relative weights of criteria in a decision problem. Inparticular, any decision process is suitably decomposedinto a hierarchy of sub-problems that are usually rathereasy to deal with.

In this paper the decision process of the AHP methodcorresponds, roughly speaking, to determining whichamong a list of R&D projects has the best performanceaccording to a number of criteria (perspective) and sub-criteria (indicators) selected by a team of experts.

The need for a systematic and quantitative analysis ofthe performance of R&D projects relies on the fact that,nowadays, R&D is one of the most significant deter-minants of the productivity and growth of companies,organizations, governments etc. Thus, it has becomesomewhat crucial to have at our disposal an intuitive,easy, efficient yet systematic and analytical method toquantify R&D performance.

More in detail, we started off in Section 2 by describ-ing the basics of AHP method as originally developedby Saaty, outlining all the important steps to follow ina decision process in order to determine the best amonga set of alternatives.

In Section 3 we laid out the general procedure ofour proposed method in order to define the basic AHPstructure for R&D performance evaluation. As we haveseen in the main text, this is essentially based on a setof questionnaires handed to a team of experts who areasked, through a number of steps, to define a consistenthierarchical structure of the AHP-based method. Thenwe gave more mathematical details on how a quantita-tive evaluation of R&D performances and relative in-consistencies can be carried out. Finally, in Section 4we presented an example of our method for the case ofa number of projects stored in the DPR&DI platform.

We believe that our results might have important im-plications for those companies, organizations and publicadministrations interested in determining R&D perfor-mance. First of all, we provided a method for a firm tomake comparisons between its R&D projects. In thisway managers are facilitated in understanding whichproject is more deficient and in which area (perspec-tive) or even in formulating more effective strategies toimprove the R&D performance of low-scoring projectsaccording to their own objectives. Second, our methodoffers a way of comparing a company’s R&D global per-formance to the performance of other firms.

To sharpen our work further, it could be interestingto study and quantify the compatibility of the different

experts (i.e. how far off they are with respect to one an-other) involved in the decision process. See for instance(Aguaron et al. (2019)). Another interesting directionmight be that of gathering data from different experts(eqn. (3.9)) using a weighted geometric mean. For ex-ample, we could set up a computation where the moreconsistent an expert has been in writing down pairwisecomparison matrices, the more weight she/he will havein the computation of priorities. Moreover, it would beinteresting to find a way of discussing more perspectivesthan those considered in this paper (Internal business,Innovation and Learning, Financial and Network andAlliances perspectives). In this way, we may hope tobuild a more general method suitable to many more or-ganizations. We hope to tackle all these problems inthe near future.

Acknowledgments

It is a great pleasure to thank the Italian Association ofUniversity Professors of Mechanical and Thermal Mea-surement for its support during the realization of thepresent paper.

References

Aguaron, J., Escobar, M. T., Moreno-Jimenez, J. M. &Turon, A. (2019), ‘Ahp-group decision making basedon consistency’, Mathematics 7(3), 242.

Alonso, J. A. & Lamata, M. T. (2006), ‘Consistencyin the analytic hierarchy process: a new approach’,International journal of uncertainty, fuzziness andknowledge-based systems 14(04), 445–459.

Bevington, P. R., Robinson, D. K., Blair, J. M.,Mallinckrodt, A. J. & McKay, S. (1993), ‘Data re-duction and error analysis for the physical sciences’,Computers in Physics 7(4), 415–416.

Bremser, W. G. & Barsky, N. P. (2004), ‘Utilizingthe balanced scorecard for r&d performance measure-ment’, R&D Management 34(3), 229–238.

Chiesa, V., Frattini, F., Lazzarotti, V. & Manzini, R.(2009), ‘Performance measurement in r&d: explor-ing the interplay between measurement objectives,dimensions of performance and contextual factors’,R&d Management 39(5), 487–519.

Crawford, G. & Williams, C. (1985), ‘A note on theanalysis of subjective judgment matrices’, Journal ofmathematical psychology 29(4), 387–405.

De Jong, P. (1984), ‘A statistical approach to saaty’sscaling method for priorities’, Journal of Mathemati-cal Psychology 28(4), 467–478.

11

Donegan, H., Dodd, F. J. & McMaster, T. (1992), ‘Anew approach to ahp decision-making’, Journal of theRoyal Statistical Society: Series D (The Statistician)41(3), 295–302.

Dong, Y., Zhang, G., Hong, W.-C. & Xu, Y. (2010),‘Consensus models for ahp group decision making un-der row geometric mean prioritization method’, De-cision Support Systems 49(3), 281–289.

Eskandari, H. & Rabelo, L. (2007), ‘Handling uncer-tainty in the analytic hierarchy process: A stochas-tic approach’, International Journal of InformationTechnology & Decision Making 6(01), 177–189.

Griffin, A. (1997), ‘Pdma research on new product de-velopment practices: Updating trends and bench-marking best practices’, Journal of Product Innova-tion Management: An International Publication ofThe Product Development & Management Associa-tion 14(6), 429–458.

Ishizaka, A. & Labib, A. (2011), ‘Review of the main de-velopments in the analytic hierarchy process’, Expertsystems with applications 38(11), 14336–14345.

Jefferson, G. H., Huamao, B., Xiaojing, G. & Xiaoyun,Y. (2006), ‘R&d performance in chinese industry’,Economics of innovation and new technology 15(4-5), 345–366.

Kaplan, R. S., Norton, D. P. et al. (1996), ‘Using thebalanced scorecard as a strategic management sys-tem’.

Kerssens-van Drongelen, I. c. & Bilderbeek, J. (1999),‘R&d performance measurement: more than choosinga set of metrics’, R&D Management 29(1), 35–46.

Kim, B. & Oh, H. (2002), ‘Economic compensationcompositions preferred by r&d personnel of differentr&d types and intrinsic values’, R&D Management32(1), 47–59.

Lazzarotti, V., Manzini, R. & Mari, L. (2011), ‘Amodel for r&d performance measurement’, Interna-tional journal of production economics 134(1), 212–223.

Moncada-Paterno-Castello, P., Ciupagea, C., Smith,K., Tubke, A. & Tubbs, M. (2010), ‘Does europe per-form too little corporate r&d? a comparison of eu andnon-eu corporate r&d performance’, Research Policy39(4), 523–536.

Saaty, T. L. (1977), ‘A scaling method for prioritiesin hierarchical structures’, Journal of mathematicalpsychology 15(3), 234–281.

Saaty, T. L. (1990), ‘How to make a decision: the an-alytic hierarchy process’, European journal of opera-tional research 48(1), 9–26.

Saaty, T. L. (2010), Mathematical principles of decisionmaking (Principia mathematica decernendi), RWSpublications.

Salimi, N. & Rezaei, J. (2018), ‘Evaluating firms’ r&dperformance using best worst method’, Evaluationand program planning 66, 147–155.

Shrestha, G. & Rahman, S. (1991), ‘A statistical repre-sentation of imprecision in expert judgments’, Inter-national journal of approximate reasoning 5(1), 1–25.

Tidd, J., Bessant, J. & Pavitt, K. (2000), ‘Managinginnovation’.

12