Landscape and horizons in Peer Review - CNR of Good Practice 4. Peer Review Methodology Part 2: Guidelines for Specific Instruments 5. Individual Research and Career Development Opportunities

Farzam Ranjbaran

Corporate Science Operations

European Science Foundation

Landscape and horizons

in Peer Review

The European Peer Review Guide:

Main features, perspectives and outlooks CNR - Italy

14 September 2011

2

On the cover!

Bees to Peers: what’s in common?

• all look very busy

• some work very hard

• others watch

• some make noise

• both are under stress

• they can both cause sweetness

but both can also sting!

F. Ranjbaran: European Peer Review Guide – CNR Italy, 14 September 2011

3

Main Features Perspectives and Outlooks

...

Mission

Key Characteristics

Structure

Open and debateable topics


Mission of the Guide

• Draws on European and international good practices

• Seeks to promote coherence and effectiveness in the form of practical reference document at European level

• Is applicable in national settings, in Europe and beyond

• Engenders integrity and mutual trust in implementation of transnational programmes

4 F. Ranjbaran: European Peer Review Guide – CNR Italy, 14 September 2011

Key characteristics of the Guide

5

• Integrates diverse policies and practices into coherent procedures

• Descriptive rather than prescriptive

• Evidence-based

wide consultation

comprehensive survey on peer review practices

• Procedural but ambitious, addresses open and debatable topics (pluridisciplinary, breakthrough research, use of bibliometrics, and incentives)

• Rolling document, will evolve as needed


Structure of the Peer Review Guide


Part 1: Overview of

the Peer Review System

1. Introduction

2. Typology of Funding Instruments

3. Pillars of Good

Practice

4. Peer Review Methodology

Part 2:Guidelines for

Specific Instruments

5. Individual Research and Career Development Opportunities

6. Collaborative Research Programmes

7. Creation or enhancement of Scientific Networks

8. Creation of Centres of Excellence

9. New Research Infrastructures Programmes

Annex 1: Glossary Annex 2: ESF Survey Analysis Report on Peer Review PracticesAnnex 3: European Code of Conduct for Research IntegrityAnnex 4: Members of the ESF Peer Review Forum

2. Typology of Funding Instruments

7

Instruments • Individual research projects

• Collaborative research projects

• Career development opportunities

• Creation of centres or networks of

excellence

• Knowledge transfer and dissemination

grants

• Creation or enhancement of scientific

networks

• Creation or enhancement of research

infrastructure

• Major prizes or awards

Variants • Non-solicited (responsive mode)

versus solicited funding

opportunities

• Thematic versus non-thematic

focus

• Monodisciplinary versus

pluridisciplinary focus

• Breakthrough research


8

Core Principles

Governance Structure

Process Integrity

Methodology

Quality Assurance

3. Pillars of Good Practice in Peer Review

Core Principles adopted

• Excellence

• Impartiality

• Transparency

• Appropriateness for purpose

• Efficiency and speed

• Confidentiality

• Ethical and integrity considerations


9

Core Principles


Process Integrity

Methodology

Quality Assurance


Integrity: Say what we do and do what we say

• Conflicts of Interest

• Managing Confidentiality

• Applicant’s right to intervene


10

Core Principles


Process Integrity

Methodology

Quality Assurance


To ensure organisational and operational coherence and quality.

Key features of good Governance Structure: Effectiveness, Clarity and Simplicity in

• Identification of relevant actors

• Definition of Roles and Responsibilities

• Definition of decision making processes

• Continuous improvement (evaluation and modification)

• Availability and effective allocation of required resources

• Terms of Reference and Code of Conduct


11

Core Principles


Process Integrity

Methodology

Quality Assurance


Quality Assurance: Following elements are proposed for monitoring Quality:

• Dedicated committees or boards outside of the organisation (survey 47.7%);

• Staff members with an explicit mandate within the organisation (survey 46.7%);

• Dedicated office within the organisation (survey 6.7%);


12

Core Principles


Process Integrity

Methodology

Quality Assurance


Devoted Chapter 4 in Part 1

• General approach, terminologies and issues

• Decomposition of the process into sequential modules or steps


4. Peer Review Methodology

13

Call for Applications

Processing of Applications

Selection of Experts

Expert Assessment

Final Decision

Qu

alit

y A

ssu

ran

ce a

nd

Eva

luat

ion

Co

mm

un

icat

ion

an

d D

isse

min

atio

n

Preparatory Phase


14

4.1 Preparatory Phase

4.2 Selection and allocation of experts

4.1.1

Mandate and scope

4.1.2

Managerial and technical

implementation

4.1.3

Staff and resource allocation

4.1.4

Peer review process

4.1.5

Documentation

4.2.1

Dissemination of the programme

4.2.2

Opening of the call

4.2.3

Closing of the call

4.2 Launch of the programme


15

4.4.1

Identification of the types of experts needed

4.4.2

Number of experts required

4.4.3

Criteria for selection of experts

4.4.4

Allocation of experts to proposals

4.3.1

Eligibility screening

4.3.2

Acknowledgment

4.3.3

Resubmissions

4.3 Processing of applications

4.4 Selection and allocation of experts


16

4.5 The Reader System:

In order to overcome some of the inherent variability and inconsistency of the conventional approaches:

• The same readers review all the proposals in their remit

• They will then prioritise or rank all the proposals they have read.

• The results of the survey show that the reader system

procedure is only rarely applied


17

4.6 The use of incentives:

Selection of the best science to receive public funds through Peer Review and Evaluation (mainly done by scientists themselves) requires certain degree of self-organisation:

• The required self-organisation is under stress;

• Some organisations pay while others do not;

• Monetary incentives tend to increase the chances of acceptance but it is not clear if it increase s quality;

• The Guide recommends to use monetary incentives only when really necessary.

• The Guide recommends other types of incentives

• Incentives should have a motivational impact and not contribute to creating additional adverse side-effects.


18

4.7.1

Briefing

4.7.2

Evaluation criteria

4.7.3

Scoring

4.7.4

Right to reply

4.7 Expert Assessments

• Use of Bibliometrics

• Equal Playing Field; non-standard career paths

• Relevance and Impact

• Scientific Quality

• Applicant

• Research Environment


4.8.1

Constitution of the review

panel

4.8.2

Prioritisation or ranking meeting

4.8.3

Funding decisions

4.8.4

Informing the applicants and

other stakeholders

4.8.5

Possible redress or

appeals

4.8.6

Grant negotiations and wrap-up

19

4.8 Final Decision


Open and Debatable Subjects

• Peer Review tailored to variants:

– MICT and Breakthrough research

• Use of Incentives (based on survey)

• Use of Bibliometrics (recent publications)

• Single-blind versus double-blind

• Equal playing field, Non-Standard Career Paths


Thank you


Documents

Landscape and horizons in Peer Review - CNR of Good Practice 4. Peer Review Methodology Part 2: Guidelines for Specific Instruments 5. Individual Research and Career Development Opportunities