HORIZON 2020 Proposal Evaluation

Briefing for Experts Societal challenge 6

First stage of two-stage calls CULT-COOP 2017

Call Coordination Team CULT-COOP REA.B.3

Overview of the SC6 1st stage evaluations

Type

of

action

Budget

Recommended

EU contrib

(in WP)

Estimated

number of

projects

CULT-COOP-2017-1 Democratic discourses and the rule of law RIA

27,50

5 M€ 1

CULT-COOP-2017-2 Improving mutual understanding among Europeans by

working through troubled pasts

RIA 2,50 M€ 2

CULT-COOP-2017-3 Cultural literacy of young generations in Europe RIA 5 M€ 1

CULT-COOP-2017-4 Contemporary histories of Europe in artistic and

creative practices

RIA 2,50 M€ 1

CULT-COOP-2017-5 Religious diversity in Europe - past, present and future RIA 2,50 M€ 1

CULT-COOP-2017-7 Cultural heritage of European coastal and maritime

regions

RIA 2,50 M€ 1

CULT-COOP-2017-12 The significance of cultural and sacred values for the

migration challenge

RIA 2,50 M€ 1

CULT-COOP-2017-6

Participatory approaches and social innovation in

culture

RIA 2,50 M€ 1

CULT-COOP-2017-9 European cultural heritage, access and analysis for a

richer interpretation of the past

RIA 9,00 2 to 3 M€ 3

H2020- SC6- CULT-COOP: 2 STAGES - BUDGET

Schedule 2016 Evaluation schedule for the 1st stage of CULT-COOP 2017

Call closure Remote evaluation Time to inform

Feb/2nd March/ 1st - March / 21st May/ 5th

New this year: Full remote

evaluation; consensus is also

reached remotely.

Schedule 2016 What happens after the 1st stage is finished

Proposals passing :

Individual thresholds: (4 out of 5)

Overall threshold: for each budget split, set at a level that allows the total requested budget of proposals admitted to stage 2 to be as close as possible to 3 times the available budget.

Successful applicants will receive an invitation

to submit a full proposal for the 2nd stage

STAGE 2

Call closure

Remote evaluation Panel meeting Time to inform

Sept / 13th Oct / 9th - Oct / 20th

-May be extended-

Nov / 6th - Nov/ 10th Nov / 28th

Role of Independent Experts

HORIZON 2020

7

Role of independent experts

• As an independent expert, you evaluate proposals submitted in response to this call

• You are responsible for carrying out the evaluation of the proposals yourself

− You are not allowed to delegate the work to another person!

• You must close reports in the electronic system within a given deadline

− This is part of your contractual obligations!

− The allowance/expenses you claim may be reduced or rejected otherwise

• Significant funding decisions will be made on the basis of your assessment

• If you suspect any form of misconduct (e.g. plagiarism, double funding), please report this to the Agency staff

HORIZON 2020

8

Guiding principles

• Independence

− You are evaluating on your personal capacity − You represent neither your employer, nor your country!

• Impartiality

− You must treat all proposals equally and evaluate them impartially on their merits, irrespective of their origin or the identity of the applicants

• Objectivity

− You evaluate each proposal as submitted; meaning on its own merit, not its potential if certain changes were to be made

• Accuracy

− You make your judgment against the official evaluation criteria and the call or topic the proposal addresses, and nothing else

• Consistency

− You apply the same standard of judgment to all proposals

HORIZON 2020

9

Confidentiality

You must:

• Not discuss evaluation matters, such as the content of proposals, the evaluation results or the opinions of fellow experts, with anyone, including:

− Other experts or any other person not directly involved in the evaluation of the proposal.

− The sole exception: your fellow experts who are evaluating the same proposal in a consensus group.

• Not contact partners in the consortium, sub-contractors or any third parties

• Not disclose the names of your fellow experts

− The Commission publishes the names of the experts annually - as a group, no link can be made between an expert and a proposal

• Maintain the confidentiality of documents, paper or electronic, at

all times and wherever you do your evaluation work.

− Return, destroy or delete all confidential documents, paper or electronic, upon completing your work, as instructed

HORIZON 2020

10

Conflicts of interest (COI) (1)

You have a COI if you:

• were involved in the preparation of the proposal

• stand to benefit directly/indirectly if the proposal is successful

• have a close family/personal relationship with any person representing an applicant legal entity

• are a director/trustee/partner of an applicant or involved in the management of an applicant's organisation

• are employed or contracted by an applicant or a named subcontractor

• are a member of an Advisory Group or Programme Committee in an area related to the call in question

• are a National Contact Point or are directly working for the Enterprise Europe Network

HORIZON 2020

11

Conflicts of interest (COI) (2)

• In the following situations, the Agency will decide whether a COI exists

− You were employed by an applicant or sub-contractor in the last 3 years

− You were involved in a grant agreement/decision, the membership of management structures or a research collaboration with an applicant in the last 3 years

− You are in any other situation that casts doubt on your impartiality or that could reasonably appear to do so

Example: If you are involved in a competing proposal

COI conditions are spelled out in your contract, and in the Code of Conduct (Annex 1)

HORIZON 2020

12

Conflicts of interest (COI) (3)

• You must inform the Agency as soon as you become aware of a COI

− Before the signature of the contract

− Upon receipt of proposals, or

− During the course of your work

• If there is a COI for a certain proposal you cannot evaluate it

− Neither individually

− Nor in the consensus group

− Nor in the panel review

− The Agency will determine if there is a COI on a case-by-case basis and decide the course of action to follow

• If you knowingly hide a COI, you will be excluded from the evaluation and your work declared null and void

− The allowance/expenses you claimed may be reduced, rejected or recovered

− Your contract may be terminated

The evaluation procedure in practice

HORIZON 2020

14

Overview of the evaluation process

Receipt of proposals

Individual evaluation

Consensus report

Evaluation Summary

report

Evaluators

Individual Evaluation Reports (remote)

Consensus Report

Is submitted by Rapporteur and approved by the other

experts

Evaluation Summary Reports

are finalised

Above threshold proposals list is

prepared

Eligibility check

Allocation of proposals to evaluators

REA REA

Finalisation

Above threshold proposals are invited to the second stage

Fully remote

Vice-Chair persons

HORIZON 2020

15

2 STAGES evaluation: Process overview

STAGE 1

Short outline proposal

(10 pages)

Checked for eligibility (rules in annexes B & C of General

Annexes of Work Programme)

Assessed against the award criteria for ‘excellence’ and

part of ‘impact’.

Proposals passing :

Individual thresholds: (4 out of 5)

Overall threshold, applying to the sum of the two individual scores: set at level such that total requested budget of proposals admitted to STAGE 2 is as close as possible to 3x the available budget

But, not less than 2.5 x the available budget.

STAGE 2 Full proposal

Checked for eligibility (rules in annexes B & C of General

Annexes of WP)

Assessed against all the award criteria: ‘excellence’, ‘impact’ & ‘implementation’

HORIZON 2020

16

Admissibility and eligibility checks • Admissibility is checked by the Agency:

− Readable, accessible and printable

− Completeness of proposal presence of all requested forms

− Only for the second stage!!: Inclusion of a plan for exploitation and dissemination of results

• Eligibility checked by the Agency - however, if you spot an issue relating to eligibility, please inform the Agency

− Minimum number of partners as set out in the call conditions in the Work Programme

− Other criteria may apply on a call-by-call basis as set out in the call conditions

• “Out of scope” – you need to check if the content of a proposal corresponds, wholly or in part, to the description of the call or topic in the Work Programme

− A proposal will only be deemed ineligible in clear-cut cases

Page limits: Clearly set out in electronic system; excess page(s) marked with a watermark (Only

10 pages for the first stage)

STAGE 1 Participant Portal Differences of 1st stage proposals in

relation to a full proposal

PART A

Global budget for the proposal,

Not a detailed budget by partner.

PART B

Only the parts that are in brackets in the template must be filled:

Title of proposal

Excellence: all points They will be assessed against all the sub-criteria of criterion "Excellence"

Impact: only point 2.1 "Expected Impact" It will be assessed against the sub-criterion dealing with the expected impacts mentioned in WP

1st stage proposals can use only 10 pages to address the 2 mentioned

criteria. Experts are requested to disregard excess pages.

These excess pages will be watermarked.

Action consisting of activities aiming: • at establishing new knowledge and/or • to explore the feasibility of a new or improved

technology, product, process, service or solution. Actions may include basic and applied research but limited demonstration or pilot activities aiming to show the technical feasibility in a near operational environment.

Type of actions in CULT-COOP 2 stages

Research and innovation action (RIA)

Evaluation criteria (1st stage, RIA) • Two evaluation criteria:

Excellence Impact

• Clarity and pertinence of the objectives;

• Soundness of the concept and credibility of the proposed methodology;

• Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)

• Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge.

• Extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic.

HORIZON 2020

20

Proposal scoring

• You give a score of between 0 and 5 to each criterion based on your comments

− Half-marks can be used − The whole range of scores should be used − Scores must pass thresholds if a proposal is to be considered for

funding

• Dynamic threshold introduced in first stage evaluations

Criteria Threshold

Excellence 4

Impact 4

Implementation n/a

TOTAL Dynamic

total requested budget of proposals admitted to stage 2 is as close as

possible to three times the available budget

HORIZON 2020

21

Interpretation of the scores

The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.

Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

Good. The proposal addresses the criterion well, but a number of shortcomings are present.

Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.

Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.

0

1

2

3

4

5

Half

sco

res g

iven

HORIZON 2020

22

Additional question

In addition to the 2 award criteria, experts will also need to assess the proposal SCOPE

HORIZON 2020

23

Additional question

SCOPE

Before evaluating a proposal, experts need to assess whether the proposal is in scope or not, according to the scope defined in the Work Program for the topic

All experts need to agree

If the proposal is only partially in scope, they need to evaluate

If the proposals is completely out of scope, it will be declared ineligible

Role of vice chairs: FLAG ALL CASES TO PROJECT OFFICERS

In addition to the 2 award criteria, experts will also need to assess the proposal SCOPE

HORIZON 2020

24

Cross-cutting issues Cross-cutting issues are fully integrated in the work

programme (WP)

Social Sciences and Humanities (SSH) are integrated across all Horizon 2020 activities to successfully address European challenges

Gender dimension in the content of R&I which addresses the relevance of sex/gender analysis, is flagged in some topics

The new strategic approach to international cooperation which consists of a general opening of the WP and targeted activities across all relevant Horizon 2020 parts, concerns some topics

If cross-cutting issues are explicitly mentioned in the scope of the call/topic and not properly addressed (or their non-relevance justified), you must reflect this in a lower score for the relevant criterion.

Proposals addressing cross-cutting issues which are not explicitly mentioned in the scope of the call or topic can also be evaluated positively

− Role of vice chairs: check TOPICS & make sure experts have assessed the cross-cutting issues when/where relevant (right criterion)

HORIZON 2020

25

Cross-cutting issues

Topic identification

Topic title Gender SSH International

Cooperation

CULT-COOP-1 Democratic discourses and the rule of law x

CULT-COOP-2 Improving mutual understanding among

Europeans by working through troubled pasts

x x

CULT-COOP-3 Cultural literacy of young generations in Europe x x

CULT-COOP-4 Contemporary histories of Europe in artistic and

creative practices

x

CULT-COOP-5 Religious diversity in Europe - past, present and

future

x x

CULT-COOP-6 Participatory approaches and social innovation

in culture

x x x

CULT-COOP-7 Cultural heritage of European coastal and

maritime regions

x

CULT-COOP-9 European cultural heritage, access and analysis

for a richer interpretation of the past

x

CULT-COOP-12 The significance of cultural and sacred values for

the migration challenge

x x x

HORIZON 2020

26

First stage evaluation phase – Remote ONLY

Individual Evaluation

Report

Individual Evaluation

Report

Individual Evaluation

Report

Rapporteur

prepares Draft

Consensus Report

Approved Consensus

Report

Expert Expert Expert

Proposal Eligible proposal

Quality check by Vice-Chairs

Quality check by Vice-Chairs

HORIZON 2020

27

Evaluation Planning

• Call closure 02/02/2017

• Remote IER phase 01 – 09/03/2017

• Evaluators accept tasks by 02/03/2017

• First draft IER ready by 03/03/2017 → VC check quality of IERs

• Remote consensus 10 - 21/03/2017

• Draft CR by rapporteur 10 - 15/03/2017

• CRs finalised → VC check quality 21/03/2017

HORIZON 2020

28

Evaluation Process: Who is who

Evaluators

• Evaluate proposals

• Submit Individual Evaluation Reports (IERs) remotely

• Participate in remote consensus discussions to agree on scores and comments

Vice-chairs

• Do not evaluate proposals

• Quality check of Individual and Consensus Reports

• Support moderators (POs) in monitoring evaluation progress and deadlines

Observers

• Observe the evaluation process

• Provide advice / improvement suggestions to the REA

Contact persons:

• Primarily: your REA topic leader

• Call coordination: Rodrigo Martín and Pablo Ballesteros

HORIZON 2020

29

Vice Chairs (VC)

Vice Chairs are experts who assist REA officers in order to ensure that the evaluation exercise runs in a smooth way.

VCs do not evaluate proposals, and cannot give their opinion on any proposals.

Their contribution relates to support regarding the evaluation management

HORIZON 2020

30

Vice Chairs (VC) -2-

VC will contact experts at the start of the evaluation and will be in contact through the entire remote evaluation exercise. VCs ensure a quality and timely delivery of evaluators' reports – Individual Evaluation Reports (IERs) and Consensus Reports (CRs)

VCs provide guidance and feedback to evaluators, under the control and overall supervision of REA topic leaders. Please be receptive to their comments and guidance.

VCs are in constant touch with REA staff to raise any relevant issue or report on any necessary event.

HORIZON 2020

31

Individual evaluation

• Read the proposal and evaluate it against the evaluation criteria

− Without discussing it with anybody else

− As submitted - not on its potential if certain changes were to be made

− Do not penalise applicants that did not provide detailed breakdown costs – they are not required

• Disregard excess pages marked with a watermark

• Check to what degree the proposal is relevant to the call or topic

• Complete an Individual Evaluation Report (IER)

− Give comments and scores for all evaluation criteria (scores must match comments)

− Explain shortcomings, but do not make recommendations

• Sign and submit the form in the electronic system

Look at the substance: Some proposals might

be handicapped by language difficulties.

HORIZON 2020

32

If a proposal • Is only marginally relevant in terms of its scientific, technological

or innovation content relating to the call or topic addressed, you must reflect this in a lower score for the Excellence criterion

− No matter how excellent the science!

• Does not significantly contribute to the expected impacts as specified in the WP for that call or topic, you must reflect this in a lower score for the Impact criterion

• If cross-cutting issues are explicitly mentioned in the scope of the call or topic, and not properly addressed (or their non-relevance justified), you must reflect this in a lower score for the relevant criterion

− A successful proposal is expected to address them, or convincingly explain why not relevant in a particular case

− Proposals addressing cross-cutting issues which are not explicitly mentioned in the scope of the call or topic can also be evaluated positively

HORIZON 2020

33

Individual Evaluation Report (IER)

• Under each sub-criterion, all elements must be commented

• it is recommended to organise comments under headers 'strengths' or 'weaknesses'

• Evaluators will complete first IER by 03/03/2017

• Please submit all your IERs by 09/03/2017

HORIZON 2020

34

Remote drafting of CR by Rapporteur

Rapporteur's task

• The rapporteur is one of the evaluators.

• Prepare draft CRs:

• 'Initialise' the draft CR in SEP

• Identify points of convergence and propose a consensus wording

• Identify divergences as points for discussion or clarification with other experts (via SEP's task comments box), highlighting strengths and weaknesses of proposals to guide remote consensus and potential scores

• Put a comment in SEP's task comments box to invite other evaluators to check the draft CR before submitting it for vice-chair's reviewing

• All Draft CR to be completed by 15/03/2017

HORIZON 2020

35

Consensus - Remote only

The rapporteur is responsible for drafting the consensus report (CR)

− The draft CR is based on the comments and scores of all the evaluators as expressed in the Individual Evaluation Reports.

• The aim is to find agreement on comments and scores

Use the electronic system for evaluation (comment box) to discuss when needed.

− Agree comments before scores!

• “Outlying” opinions need to be explored

− They might be as valid as others – be open-minded

• The draft CR should be commented by all experts after it is reviewed by the vice-chair (Review CR task)

• To finalize the CR, all experts have to approve it in the system.

• The Vice Chair will perform a final quality check of the CR and may request editorial changes to the rapporteur.

HORIZON 2020

36

Consensus report (CR) (1)

• The rapporteur is responsible for drafting the CR

− Including consensus comments and scores

• The quality of the CR is paramount

• The aim of the CR is to give:

− A clear assessment of the proposal based on its merit, with justification

− Clear feedback on the proposal’s weaknesses and strengths

• Avoid:

− Comments not related to the criterion in question

− Comments that are too short or too long or use inappropriate language you should explain what you mean in an adequate length and clear manner

− Categorical statements that have not been properly verified e.g. “The

proposal doesn’t mention user requirements” – when there is a short reference…

Applicants can challenge those through evaluation review procedures

− Scores that don’t match the comments

− Making recommendations

− Marking down a proposal for the same critical aspect under two different criteria

HORIZON 2020

37

Consensus report (CR) (2)

Tips for rapporteurs:

• If you see great discrepancies between evaluators' scores:

first read the IERs, it is very possible that:

• Comments were 'softer' or 'harder' than the scores awarded and therefore the evaluators are not so far apart in their comments (remember first the comments and then the scores)

• A major failing/strength might have been identified by one evaluator which was not noticed by another and it is perfectly normal that the final scores for a CR are very different from an individual's IER

• Evaluators might really appreciate a proposal and score it high even if it only broadly addresses the call topic while another evaluator might also appreciate it but because it doesn't fully address the call topic has scored it low. It's important that the proposals are evaluated in the context of what was requested in the call text, even if an evaluator thinks it is an excellent proposal.

HORIZON 2020

38

Consensus report (CR) (3)

Tips for rapporteurs:

• Depending on the possibilities above and the clarifications you receive, try to draft a compromise report that balances both views and then see how the evaluators feel about it

• Usually several versions of a CR will be required before it is fully approved

• Work on the comments first without considering the scores

• Once you are satisfied with the CR, add a score for each criterion which reflects the comments (it should not necessarily be an average of the 3 individual scores)

→ If a consensus cannot be reached, then your vice-chair will find a

solution with REA moderator

HORIZON 2020

39

Observers

• Appointed by the Commission/Agency will monitor remote evaluation, to ensure a high quality evaluation

• Check the functioning and running of the overall process

• Advise, in a report, on the conduct and fairness of the evaluation sessions and, if necessary, suggests possible improvements

• Do not evaluate proposals and, therefore, do not express any opinion on their quality

HORIZON 2020

40

Logistics

• The electronic system (SEP) for the evaluation of proposals is available and accessible via your ECAS password

− Please make sure you know your ECAS login and password

You can access through the Participant Portal:

https://ec.europa.eu/research/participants/evaluation/

→ Check the briefing on SEP workflow for further details

HORIZON 2020

41

THANK YOU VERY MUCH FOR YOUR

ATTENTION