1

Evaluation of Research Quality 2011-2014 (VQR 2011-2014)

Criteria for the Evaluation of Research Outputs

Group of Evaluation Experts

for Area 05 -Biological Science

(GEV05)

30 November 2015

2

1. INTRODUCTION .............................................................................................................................. 3

2. DELIMITATION OF THE GEV AREA ........................................................................................... 3

3. ORGANIZATION OF THE GEV ...................................................................................................... 5

3.1 Sub-GEV composition ......................................................................................................................... 5

3.2 Assignment of research products within the GEV .................................................................. 6

3.3 Operating rules of the GEV ............................................................................................................... 6

4. THE EVALUATION OF RESEARCH PRODUCTS ...................................................................... 7

5. PEER REVIEW EVALUATION ....................................................................................................... 7

5.1 The selection of external peer reviewers ................................................................................... 7

5.2 Peer evaluation ..................................................................................................................................... 8

6. BIBLIOMETRIC ANALYSIS ........................................................................................................... 9

6.1 Data Bases .............................................................................................................................................. 9

6.2 The time windows of citations ....................................................................................................... 9

6.3 Self-citations .......................................................................................................................................... 9

6.4 Bibliometric indicators ................................................................................................................... 10

6.5 The algorithm for classifying products .................................................................................... 10

6.6 Calibration procedure ..................................................................................................................... 11

8. PRODUCTS FOR EVALUATION ....………………………………………………………………………18

7. OTHER PRODUCTS ...................................................................................................................... 18

8. CONFLICTS OF INTEREST ......................................................................................................... 19

3

1. Introduction

This document describes the organization of the Group of Evaluation Experts for the

Biological Science Area (from now on, the GEV 05) and the criteria the Group will use in

evaluating research outputs. The document is divided in eight parts. Section 2 lists the Settori-

Scientifico-Disciplinari, the Settori-Concorsuali and the ERC Sectors that are relevant for the

GEV. Section 3 summarizes the internal operating rules of the GEV. Section 4 describes the

evaluation criteria for the research products. Section 5 describes the peer review process and

the guidelines for the selection of external reviewers. Section 6 describes the bibliometric

criteria: the journal databases, the bibliometrics indicators, the algorithm and the calibration

procedure. Section 7 describes the evaluation criteria for specific products. Finally, Section 8

describes how the GEV plans to solve potential conflicts of interest between GEV members

and authors of research products.

2. Delimitation of GEV Area

The GEV will take care of the evaluation of the products submitted by researchers belonging to the

Settori-Scientifico-Disciplinari (SSD), Settori-Concorsuali (SC) and ERC Sectors (ERC) listed in

Tables 1-3.

Table 1. The relevantSettori Scientifico-Disciplinari (SSD) for Area 05

Area 05 - Biological Science Relevant Settori Scientifico-Disciplinari (SSD)

BIO/01 GENERAL BOTANY

BIO/02 SYSTEMATIC BOTANY

BIO/03 ENVIRONMENTAL AND APPLIED BOTANY

BIO/04 PLANT PHYSIOLOGY

BIO/05 ZOOLOGY

BIO/06 COMPARATIVE ANATOMY AND CYTOLOGY

BIO/07 ECOLOGY

BIO/08 ANTHROPOLOGY

4

BIO/09 PHYSIOLOGY

BIO/10 BIOCHEMISTRY

BIO/11 MOLECULAR BIOLOGY

BIO/12 CLINICAL BIOCHEMISTRY AND MOLECULAR BIOLOGY

BIO/13 EXPERIMENTAL BIOLOGY

BIO/14 PHARMACOLOGY

BIO/15 PHARMACEUTICAL BIOLOGY

BIO/16 HUMAN ANATOMY

BIO/17 HISTOLOGY

BIO/18 GENETICS

BIO/19 MICROBIOLOGY

Table 2. The relevant Settori-Concorsuali (SC) for Area 05

Area 05 - Biological Science Relevant Settori Concorsuali (SC)

05/A1 BOTANY

05/A2 PLANT PHYSIOLOGY

05/B1 ZOOLOGY AND ANTHROPOLOGY

05/B2 COMPARATIVE ANATOMY AND CYTOLOGY

05/C1 ECOLOGY

05/D1 PHYSIOLOGY

05/E1 GENERAL BIOCHEMISTRY

05/E2 MOLECULAR BIOLOGY

05/E3 CLINICAL BIOCHEMISTRY AND MOLECULAR BIOLOGY

05/F1 EXPERIMENTAL BIOLOGY

05/G1 PHARMACOLOGY, CLINICAL PHARMACOLOGY AND PHARMACOGNOSY

05/H1 HUMAN ANATOMY

05/H2 HISTOLOGY

05/I1 GENETICS

05/I2 MICROBIOLOGY

Table 3. The relevant ERC sectors (ERC) for Area 05

Area 05 - Biological Science Relevant ERC sectors

PE4_1 Physical chemistry

PE4_2 Spectroscopic and spectrometric techniques

PE4_3 Molecular architecture and Structure

PE4_4 Surface science and nanostructures

PE4_5 Analytical chemistry

PE4_6 Chemical physics

PE4_11 Physical chemistry of biological systems

PE4_13 Theoretical and computational chemistry

5

PE4_15 Photochemistry

PE5_11 Biological chemistry

PE5_14 Macromolecular chemistry

LS1_1 Molecular interactions

LS1_2 General biochemistry and metabolism

LS1_3 DNA synthesis, modification, repair, recombination and degradation

LS1_4 RNA synthesis, processing, modification and degradation

LS1_5 Protein synthesis, modification and turnover

LS1_6 Lipid synthesis, modification and turnover

LS1_7 Carbohydrate synthesis, modification and turnover

LS1_8 Biophysics (e.g. transport mechanisms, bioenergetics, fluorescence)

LS1_9 Structural biology (crystallography and EM)

LS1_10 Structural biology (NMR)

LS1_11 Biochemistry and molecular mechanisms of signal transduction

LS2_1 Genomics, comparative genomics, functional genomics

LS2_2 Transcriptomics

LS2_3 Proteomics

LS2_4 Metabolomics

LS2_5 Glycomics

LS2_6 Molecular genetics, reverse genetics and RNAi

LS2_7 Quantitative genetics

LS2_8 Epigenetics and gene regulation

LS2_9 Genetic epidemiology

LS2_10 Bioinformatics

LS2_11 Computational biology

LS2_12 Biostatistics

LS2_13 Systems biology

LS2_14 Biological systems analysis, modelling and simulation

LS3_1 Morphology and functional imaging of cells

LS3_2 Cell biology and molecular transport mechanisms

LS3_3 Cell cycle and division

LS3_4 Apoptosis

LS3_5 Cell differentiation, physiology and dynamics

LS3_6 Organelle biology

LS3_7 Cell signalling and cellular interactions

LS3_8 Signal transduction

LS3_9 Development, developmental genetics, pattern formation and embryology in animals

LS3_10 Development, developmental genetics, pattern formation and embryology in plants

LS3_11 Cell genetics

6

LS3_12 Stem cell biology

LS4_1 Organ physiology and pathophysiology

LS4_2 Comparative physiology and pathophysiology

LS4_3 Endocrinology

LS4_4 Ageing

LS4_5 Metabolism, biological basis of metabolism related disorders

LS5_1 Neuroanatomy and neurophysiology

LS5_2 Molecular and cellular neuroscience

LS5_3 Neurochemistry and neuropharmacology

LS5_4 Sensory systems (e.g. visual system, auditory system)

LS5_6 Developmental neurobiology

LS5_7 Cognition (e.g. learning, memory, emotions, speech)

LS5_8 Behavioural neuroscience (e.g. sleep, consciousness, handedness)

LS5_9 Systems neuroscience

LS5_10 Neuroimaging and computational neuroscience

LS6_1 Innate immunity and inflammation

LS6_2 Adaptive immunity

LS6_3 Phagocytosis and cellular immunity

LS6_4 Immunosignalling

LS6_5 Immunological memory and tolerance

LS6_6 Immunogenetics

LS6_7 Microbiology

LS6_8 Virology

LS6_9 Bacteriology

LS6_10 Parasitology

LS6_11 Prevention and treatment of infection by pathogens (e.g. vaccination, antibiotics, fungicide)

LS6_13 Veterinary medicine and infectious diseases in animals

LS7_2 Diagnostic tools (e.g. genetic, imaging)

LS7_3 Pharmacology, pharmacogenomics, drug discovery and design, drug therapy

LS7_5 Toxicology

LS7_6 Gene therapy, cell therapy, regenerative medicine

LS8_1 Ecology (theoretical and experimental; population, species and community level)

LS8_2 Population biology, population dynamics, population genetics

LS8_3 Systems evolution, biological adaptation, phylogenetics, systematics, comparative biology

LS8_4 Biodiversity, conservation biology, conservation genetics, invasion biology

LS8_5 Evolutionary biology: evolutionary ecology and genetics, co-evolution

LS8_6 Biogeography, macro-ecology

LS8_7 Animalbehaviour

7

LS8_8 Environmental and marine biology

LS8_9 Environmental toxicology at the population and ecosystems level

LS8_10 Microbialecology and evolution

LS8_11 Speciesinteractions

LS9_1 Non-medical biotechnology and genetic engineering (including transgenic organisms, recombinant proteins, biosensors, bioreactors, microbiology)

LS9_2 Synthetic biology, chemical biology and bio-engineering

LS9_3 Animal sciences (including animal husbandry, aquaculture, fisheries, animal welfare)

LS9_4 Plant sciences (including crop production, plant breeding, agroecology, soil biology)

LS9_6 Forestry and biomass production (including biofuels)

LS9_7 Environmental biotechnology (including bioremediation, biodegradation)

LS9_8 Biomimetics

LS9_9 Biohazards (including biological containment, biosafety,

biosecurity)

3. Organization of GEV

The GEV is organized as follows.

Coordinator: Anna Tramontano

Assistant:Elisa Melucci

Sub-GEVcomposition

Sub-GEV name and relevant

research areas (SSD) Coordinator Members

morpho-functionalScience:

BIO/8, BIO/9, BIO/16, BIO/17 Prof. SCHIEPPATI

Marino (BIO/09), Miniussi

(BIO/09) Schieppati (BIO/09),

Cappello (BIO/16), Martelli

(BIO/16), Adamo (BIO/17).

Biochemistry and Molecular

Biology: BIO/10, BIO/11, BIO/12 Prof. BOLOGNESI

Brancaccio (BIO/10), Bolognesi

(BIO/10), Rizzi (BIO/10),

Scorrano (BIO/10), Liuni

(BIO/11), Pastore L. (BIO/12)

Genetics and Pharmaceutical

Sciences: BIO/13, BIO/14,

BIO/18, BIO/19

Prof.ssa PAROLARO

Montecucco (BIO/13),

Montuschi (BIO/14), Parolaro

(BIO/14), Di Chiara (BIO/14),

8

Ziche (BIO/14), Perretti

(05/G1), Delledonne (BIO/18),

Visca (BIO/19)

Integrative Biology: BIO/1,

BIO/2, BIO/3, BIO/4, BIO/5,

BIO/6, BIO/7, BIO/15

Prof.ssa CARNEVALI

Balestrini (BIO/01), Peruzzi

(BIO/02), Pastore D.(BIO/04),

Silva (05/A2), IbanezEzequiel

(05/A2), Castellari (05/A2),

Terlizzi (BIO/05), Carnevali

(BIO/06), Yakimov (BIO/07),

Gordon (BIO/07), Cameron

(05/B1), Desneux (05/C1)

Table 4. Sub-GEVs, relevant Settori-Scientifico-Disciplinari (SSD), coordinators and members

A member allocated to a sub-GEV can be reallocated, depending on needs emerging during the

evaluation process.

3.1 Assignment of research products within the GEV

The assignment of research products to the Sub-GEV in charge of their evaluation will occur

according to the SSD specified by the author in the form associated with the product. The SSD

assigned by the author to the product may differ from the author’s SSD, since it refers to the

GEV and, within the GEV, to a specific Sub-GEV and/or SSD which, according to the author, are

more competent to evaluate the product. The GEV 05 will divide research products by type of

publication and by research area and will assign them to the appropriate Sub-GEV. The Sub-

GEV Coordinator will assign them to two competent Sub-GEV members. If a research product

is assigned to more than one GEV (for instance since co-authors indicate different SSD

belonging to different GEVs), the product will be evaluated according to the VQR Guidelines

for the Groups of Evaluation Experts (Section 3.2). If necessary, the Coordinators of the GEVs

involved will constitute specific Inter-Area Consensus Groups.

3.1 Operating rules of the GEV

The operating rules of the GEV are recalled as follows:

a GEV meeting is called with at least 15 days notice. The meeting is called by the

Coordinator, who also sets the agenda;

decisions within the GEV are made by simple majority rule among members who attend a

meeting. In order to vote, physical presence is not required if presence is assured via web or

phone connection. When an in-person meeting is not scheduled, votes are expressed

electronically;

9

the Assistant assigned by ANVUR to the GEV, Elisa Melucci, attends the GEV meetings,

with secretariat functions and without voting rights. At the end of each meeting, minutes and

a synthetic report outlining the main decisions will be drafted, circulated among GEV

members, approved by the Coordinator and then sent to ANVUR to be filed.

4. The evaluation of research products

The evaluation of products by the GEV follows the informed peer review methodology, which

consists in employing different, and whenever possible, mutually independent, evaluation

methods, to be harmonized within the GEV, which ultimately remains responsible for the final

evaluation.

The available evaluation tools are:

Peer review evaluation by (normally two)external reviewers independently selected

by two different GEV members.

Direct evaluation by the GEV, which can conduct an internal peer review according to

the same procedure described for external peer review (that is, two GEV members will

be involved).

Bibliometric analysis, to be conducted according to the procedure described below in

this document. Research products subject to bibliometric analysis are not assigned

automatically to the merit classes established by the Decreto-Ministeriale (DM) and by

the VQR Call. The allocation is instead based on the expert judgment of the GEV, which

will employ any available information beside bibliometric indicators, such as the

expertise of its members and the information described in the form associated with the

products.

10

5.Peer review evaluation

Each research product to be evaluated by peer review will be sent to two external reviewers,

independently chosen by the two GEV members to whom the product was assigned.

Alternatively, a product will be evaluated within the GEV according to the same procedure,

provided that the necessary expertise is available and that no conflict of interest is present.

5.1 The selection of external peer reviewers

The selection of external reviewers, among Italian and foreign scholars, given its essential role

for the public interest, follows the principle of honest institutional cooperation and is founded

on the criteria of correctness, objectivity and impartiality.

Great attention will be devoted to maintaining the anonymity of the reviewers, both at the

stage of preparation of the list of reviewers and at the operational stage of the evaluation. The

results of the evaluation of individual products and the identity of the reviewers in charge will

not be made public. A list with the reviewers’ names will be published by ANVUR within 30

days of the publication of the VQR Final Report.

Reviewers will be selected among the most authoritative and scientifically qualified scholars

and specialists in the disciplines relevant to the research products to be examined. They are

expected to have been active in research during the period covered by the VQR.

Starting from the MIUR reviewer archive REPRISE, the GEV will prepare an updated list of

external reviewers such to adequately satisfy the standards set by the GEV in terms of

scientific quality and experience with evaluation. The list will be extended with new

reviewers selected by the GEV. In particular, through the Sub-GEV Coordinators, the

Coordinator will invite GEV members to suggest a significant number of experts who satisfy

the required standards and are available for the evaluation. The GEV Coordinator will collect

suggestions together with information about the reviewers’ qualifications, as summarized in a

specific proposal form, to be prepared by the GEV Coordinator and approved by the GEV.

It will be possible to extend the reviewer list throughout the evaluation procedure, on the

basis of needs that may emerge after the products are transmitted by institutions.

In order to reduce potential conflicts of interest, the GEV will employ, whenever possible,

reviewers that are active in foreign universities and institutions.

Rather than internal reviewers chosen among GEV members, the GEV will preferably employ,

whenever possible, external reviewers.

11

5.2 Peer evaluation

The evaluation by external or internal reviewers is based on an evaluation form to be

prepared by the GEV, following ANVUR guidelines, together with instructions for reviewers.

The evaluation form will allow the reviewer to assign a score to the three evaluation criteria

established by the DM and the VQR Call, that is, originality, methodological rigor, and attested

or potential impact. The score granted is 1 to 10 and the form will also include an empty space

where a brief comment should be entered, to summarize the motivation for the answers

provided to the questions. The GEV will translate the indications contained in the evaluation

form into one of the five classes established by the VQR Call.

The assignment to the class of merit is based on three evaluation criteria:

a. originality, to be understood as the level at which the research product

introduces a new way of thinking, or new themes and/or sources, in relation to

the scientific object of the research, and is thus distinguished from previous

approaches to the same topic;

b. methodological rigor, to be understood as the level of clarity with which the

research product presents the research goals and the state of the art in

literature, adopts an appropriate methodology withrespect to the object of

research, and shows that the goal has been achieved;

c. attested or potential impact upon the international scientific community of

reference, to be understood as the level at which the research product has

exerted, or is likely to exert in the future, a theoretical and/or applied influence

on such a community also on the basis of its respect of international standards

of research quality.

The GEV transforms the information contained in the evaluation form in one of 5 classes of

merit defined in the Notice. In the case of non-convergent evaluations of peer reviewers, the

sub-GEV creates an internal Consensus Group with the task of proposing to the GEV the final

score of the product covered by the court does not comply with the methodology of the

external auditors of the consensus report. The Consensus Group may also make use of a third

expert opinion judgment when the peer evaluations are significantly divergent. In case of

conflict of opinion between the members of the Consensus Group, the Consensus Group will

be integrated by the Coordinator of the Sub-GEV or by the Coordinator of the GEV. In any case

the responsibility of the final assessment lies with the GEV.

12

6. Bibliometric Analysis

The research product subject to bibliometric evaluation are indexed products in databases

based on citations such as ISI Wos and Scopus, and in particular:

Scientific article, in Letter forms

Scientific article of critical literature review (Review)

10% of the articles ,already examined by bibliometric analysis, will be sent to peer review in

order to assess the degree of correlation between the two methods. The articles selected for

this double evaluation will be chosen through stratified random sample for SubGev.

6.1 Data Bases

The GEV will be using the databases of Thompson Reuters Web of Science and Scopus Elsevier

according to the author recommendations on the choice of database to be used, reported on

the product-linked form.

6.2 The time windows of citations

For computation of bibliometric indicators, the GEV will be using the updated citations on

the29th of February 2016.

6.3 Self-citations

Inclusion and Exclusion of self-citations in bibliometric evaluations is nowadays a debated

issue in the scientific community. For the VQR 2011-2014, GEV 05 decided, on the basis of

suggestions given from the Bibliometric evaluation Group during the first plenary meeting of

the GEV Coordinators, to include self-citations, but to pay specific attention to cases

wheretheyare more than 50% of the total citations of the article. The final decision on these

products will be based on the information contained in the product–linked form and

whenever deemed necessary, by submitting the product to an informed peer review, based on

the opinion of internal or external reviewers.

6.4 Bibliometric Indicators

The evaluation will use, for all articles published on indexed journals in WOS and Scopus

databases, an algorithm which considers the number of citations and the impact indicator

Journal Metric (JM) of the hosting journal according to the publication data. Following the

general opinion of the scientific community, and taking into account the differences between

different indicators in measuring the impact of a journal, the GEV 05, following the

suggestions of the Bibliometry Work group of ANVUR, decided to use more than one indicator

13

of Journal metric. For each databases, more than one indicator will be used to measure the

popularity of the source of publication (according to which citations are counted

independently from the origin of each citations) and an indicator which measures prestige

(whereby citations are weighted by the authoritativeness of their source).

The impact indicators proposed for the journal are listed below:

for WoS (https://www.webofknowledge.com): 5-year Impact Factor (5YIF) as an

indicator of Popularity, and Article Influence (AI) as an indicator of prestige1;

for Scopus (http://www.journalmetrics.com): Impact per Publication (IPP), as an

indicator of Popularity, and SCImago Journal Rank (SJR), as an indicator of prestige2.

In the product-linked form the author must identify the databases of choice and a single

impact indicator, between the two associated to the selected database, for evaluation

purposes.

6.5 The algorithm for classifying products

The algorithm used for the classification of the articles in the 5 classes of merit defined in the

callis based on a combined use of bibliometric indicators that relate to the impact of the

journal in which the article was published (JM) and the citations to measure the impact of the

single item (CIT).Depending on the year of publication the first or the second indicator can

have a larger or smaller relative weight. Each article is evaluated within a specific reference

category (more details below), and a year of publication. The evaluation procedure in the

reference category is previously calibrated in order to ensure that the probability ex ante of

each item of a particular category to fall within one of the classes is that defined by the call

when compared with the world production:

• Excellent [top 10% of the distribution of the international scientific production of the

area to which it belongs];

• High [10% - 30% of the distribution of the international scientific production of the

area to which it belongs];

• Discrete [30% - 50% of the distribution of the international scientific production of

the area to which it belongs];

• Acceptable [50% - 80% of the distribution of the international scientific production

of the area to which it belongs];

1The choice is 5 YIF instead of the more popular impact factor (IF) is a) the first has more stability with respect to vary

the year of publication, and b) the time window in which citations are considered (5 years)is the same used for the AI. 2 The time window in which the citations are considered is, in this case, 3 years for both indicators. Moreover, the

definition of IPP is the same as the 5YIF while that of the SJR, although non identical, is very similar to that of AI .

14

• Limited [80% - 100% of the distribution of the international scientific production of

the area to which it belongs].

The percentiles do not refer to the expected distribution of the products presented to the VQR.

The assessment of individual items is not comparative: each item will be placed in the

respective class regardless of the placement of the other products. The first step in the

evaluation of a given article is the identification of the reference category known as Subject

Category (SC) in WoS and All Science Journal Classification (ASJC) in Scopus. A journal can

belong to one or more SC, and the one to be used for the evaluation of the product will be

indicated by the author / institution who submitted the item. This indication is not, however,

binding and may be modified by the GEV if the content of the article appears more relevant to

another of the SCs the journal belongs to. A multidisciplinary category is present both in WoS

(Multidisciplinary Sciences) both in Scopus (Multidisciplinary) and includes journals, such as

Nature, Science, etc., characterized by a plurality of scientific arguments. Articles published in

a journal that appears only in this category will be assigned to a SC on the basis of (i) the

citations contained in the article and (ii) the references made to the article. In particular, it

will be assigned to the SC of the majority of journals cited / citing the item. In this way, the

publication will be compared with publications in the same subject area and / or discipline.

When an article published in an interdisciplinary journal will be assigned to an SC, it will

manta in the JM of the journal and the number of citations received, without changing the

distribution of the destination SC. The same procedure will be adopted for the journals

appearing only in other multidisciplinary subject categories of WoS and Scopus.

As mentioned above, the allocation in the article to one of the 5 classes specified in the call is

performed using a calibration of the world production in the SC identified in the specific year.

This procedure allows one to have, whatever the category analyzed and the year in question,

the percentage of items defined by the DM and the Call.

6.6 Calibration Procedure

The calibration of the algorithm bibliometric is a function of the particular SC in the particular

year analyzed. The algorithm also distinguishes journal articles and letters from reviews,

calculating empirical cumulative distributions separately because of the different number of

citations typically received from these types of publications.

The algorithm calculates the empirical cumulative distribution of the bibliometric indicator

JM for journals belonging to the SC and the year of publication of the product to be evaluated

and computes in which percentile the product falls. Next the empirical cumulative distribution

of the number of citations CIT of all products published in the journals belonging to the SC is

calculated and, for each product, the respective percentile is computed. After this procedure

each item will therefore have two associated percentile. The two percentiles represent a point

in the region Q = [0,1] × [0,1] of the coordinate plane, bounded by the JM percentile of the

15

journal (X-axis) and the percentile of quotes CIT (Y axis). Then Q is divided into five zones or

regions such that they comply with the call in terms of the percentage of items belonging to

each region.

This is obtained by simply drawing lines represented by:

nCIT A JM B

The angular coefficient of the straight lines delimiting the zones (A) is imposed equal for all

the straight lines in order to increase the homogeneity of the adopted criterion. Bn intercepts

are calculated by ANVUR, depending on the distribution of the particular SC, to ensure that

they reflect the percentages of the DM. An example of a subdivision of Q in the 5 zones is

represented in Figure 1. Regardless of the distribution of various articles from one category to

another and from year to year, the algorithm allows for an evaluation consistent with the

specific subset. The slope A of the straight lines is established by GEV. It has a very important

role because, depending on the value of A, the final classification will be based more on

percentile of citations (for slopes lower than 1in absolute value), or vice versa on the

percentile of the metric of the magazine (for slopes larger than 1in absolute value).For

example, with reference to Figure 1, a horizontal line corresponds to an evaluation based

solely on the percentile of the citations. Given to what is known in the field of bibliometry, and

given the various statements on the proper use of bibliometrics for evaluation3 , the use of

very steep slopes must be avoided as much as possible, given the absolute impossibility of

using the only JM of a journal as a surrogate (proxy) for the impact of a single article

published in it. In other words, values of A lower than 1 in absolute value should be used, as

much as possible, as to favor the information provided by the CIT, which is a measure of

impact at the level of the individual product subject to evaluation (article level metric).This

choice is however not absolute, but depends on the different citation practices of the various

disciplines / communities, as well as from the number and composition of the SC, which

makes it more or less reliable, in more recent publication years, the information provided by

the citation data.

3See an exampletheSan Francisco Declaration on Research Assessment (DORA) - http://www.ascb.org/dora/ - and the

IEEE Statement on Appropriate use of Bibliometric Indicators

- https://www.ieee.org/publications_standards/publications/rights/bibliometrics_statement.html.

16

Figure 1. Representation in percentiles of all articles published in a given SC in a particular year. Each publication is placed in the plot according to the percentile indicator of impact of the journal JM (row) and the percentile of the number of citations CIT (column). The plan is divided into five areas according to the percentages shown in the VQR call. The angular coefficient of the straight lines delimiting the zones is imposed equal for all the straight lines. Intercepts are calculated by ANVUR, depending on the distribution of the particular SC, to ensure that the percentage of the call are met.

Based on numerous simulations performed by the working group on bibliometric

evaluation ANVUR on data available at the date of this document, the GEV05 believes

that the gradients to be used to vary the years will be broadly as follows:

2011: −0,4

2012: −0,6

2013: −0,8

2014: −1,2

The slopes can vary from up to 30% for the years 2011, 2012, 2013 to avoid degenerate

cases4. In 2014, where the citation data is less stable, the slope will be in the range [-2,0-1,0]

As an example, in Figure 2 the calibration of a SC with four parallel lines is shown in Figure 2.

The slope of -0.6 was chosen in order to prioritize the weight of the citations in the final

evaluation. As can be seen from the figure, the points, which represent the articles of the SC,

are unevenly distributed. By appropriately selecting the values of the intercepts, one can

ensure that the percentages comply with the call, with and accuracy to the tenth of a percent.

In other words, when the algorithm is applied to the "world" production the percentages

defined in DM and in the call are respected. It follows that the specific article submitted to the

VQR is evaluated always with respect to the percentile of the "international scientific

production of its area."

4 Such as, for example, the possibility of classifying as excellent products that have not received any citations

Thresholds: nCIT A JM B

17

Figure 2. Example of application of the algorithm bibliometric to a SC. The division of the sub-space Q by parallel lines is done in order to respect the percentage defined in the call when the algorithm is applied to the world production of the specific SC.

Once the calibration procedure is defined, the assignment of a product subject to the VQR is as

follows. JM percentile for the journal in which the article was published and citation

percentiles are computed and the product is placed in the plot shown above. The class is

assigned depending upon the area where the product point falls.

There are borderline cases in which articles are published in journals of high prestige but

receive few citations (the area at the bottom right in Figure 2) or published in journals with

low JM, but with a high number of citations ( the upper left area in Figure 2). In such cases the

evaluation procedure will take place through informed peer review that will include a peer

review inside the GEV or outside, if the necessary expertise is not present in the GEV. To

identify items of this type, it is sufficient to draw two additional straight lines, with a positive

slope, which identify the areas at the top left and bottom right of Q (see Figure 3).

To identify the articles of this type, the GEV05, consistent with what other bibliometric GEVs,

will draw (see example in Figure 3) two straight lines with positive slope, so as to form two

triangles. The one in the upper left is defined by the left and top sides of Q and the straight line

connecting the point (0,0;0,05) with the intersection between the threshold line of the

"Excellent" area and the top side of Q. The one at the bottom is a right-angled isosceles

triangle which includes the 5% of products for 2011 and 2012 to 7% for 2013.

18

Finally, given the expected low numbers of citations for articles published in 2014, the GEV05 decided to submit to informed peer review all articles published in 2014 whose classification, on the basis of the proposed algorithm, does not result in a final evaluation of "Excellent".

. Figure 3. Example the definition of the uncertainty areas to be managed via informed peer review (IR).

19

6. Products for evaluations

Of the typology1(Monograph scientific and similar products), theGEV05will only consider type(a) "Research Monographs", meaning the books, type (b) "Collection of essays consistent own research", type(d) "Scientific Commentary " and type(s) "Manuals with critical content not merely educational."

Of thetypology2(Contribution in journals), the GEV will only consider type(a) "Scientific article," (b) "Scientific article of critical review of the literature or Review essay" and(c) "Letter".

Of thetypology3(Contribution in volumes), the GEV will only consider type(a) "Chapter or Essay" and type(b) "Scientific article in conference proceedings with peer review ".

7. Other products

The products that can not be evaluated are: contributions to meetings without referees and patents only filed in Italy or of level not at least B for European or international patents

For the evaluation of chapters in scientific books: these products can not reach the level Excellent but only acceptable/discrete

8. Conflicts of interest

GEV members will not evaluate or assign to external reviewers or other GEV members:

products they have authored or co-authored;

products which have been authored or co-authored by spouses and relatives up to the fourth

degree of kinship;

products submitted by universities of which they have been employees or official associates

(also to research centers) since 1/1/2011;

products submitted by research centers controlled by MIUR or other public and private

entities that are voluntarily subjected to the VQR of which they have been employees or

official associates (also research centers) since 1/1/2011.

For the above products there exists a conflict of interest in the following cases:

when the institution in question has a permanent internal division along a territorial or

disciplinary dimension (e.g., a local section of a research center, institute, department), a

conflict of interest exists only with respect to the products presented by the same internal

unit;

20

when the institution in question does not have a permanent internal division along a

territorial or disciplinary dimension (e.g., a local section of a research center, institute,

department), a conflict of interest exists with respect to the products presented by the

institution;

when the internal organization is based on several hierarchical levels (e.g., several institutes

within a single department) a conflict of interest emerges at the lowest level (e.g., GEV

members who are affiliated with different institutes belonging to the same department have

a conflict of interest only with respect to the products presented by authors belonging to the

same institute).

In case of conflicts of interest, the GEV Coordinator, or the sub-GEV Coordinator when

appropriate, will assign the product to be evaluated to another GEV member for whom no conflict

of interest is present.

In case of conflicts of interest involving the GEV Coordinator, the corresponding products will be

assigned by the VQR Coordinator or by a person designated by the VQR Coordinator.