Embed Size (px)
Citation preview
HORIZON 2020
General Briefing for Experts
Societal Challenge 5
• Subject of the evaluation
• Evaluation Roles
• Evaluation Process
• Evaluation in Practice
• Tips for Reports
Content
• Achieve a greener, more resource efficient and climate-resilient economy in sync with
the natural environment
• Demonstrate a strong commitment to supporting SDGs and targets of COP21
• Enhance relevance and impact by delivering against EU policy priorities
Horizon 2020 - Societal Challenge 5 – WP 2018-2020
• Decarbonisation
• Climate adaptation, impacts and services
• Inter-relations between climate change, biodiversity and ecosystem services
• The Cryosphere
• Knowledge Gaps
Title: Building a low-carbon, climate resilient future:
climate action in support of the Paris
Agreement
H2020-LC-CLA-2018-2020
• Connecting economic and environmental gains – the circular economy
• Raw Materials
• Water for our environment, economy and society
• Innovating cities for sustainability and resilience
• Protecting and leveraging the value of our natural and cultural assets (including Earth Observation)
Title: Greening the economy in line with
the Sustainable Development Goals
(SDGs)
H2020-SC5-2018-2020
• Challenge-based approach, calls do not outline the expected
solutions to the problem or the approach to be taken
• More emphasis on innovation aspects, including technological
and non-technological and social innovation
• Greater emphasis on impact and the use of project results and
knowledge
• Different types of actions addressing the full research and
innovation cycle
Horizon 2020 main features
FAQ
H2020 Overview – Two Stage
PublicationOpening of
Submissions
Deadline
Stage 1
Individual Evaluation
Consensus FinalisationInformation to
applicants
Deadline
Stage 2
Individual
EvaluationConsensus
Panel Review and Ranking
Ethics Evaluation
FinalisationInformation to
Applicants
Grant Agreement Preparation
Start of projects
Role of the EC staff
Responsible Authorising Officer
Responsible for taking the final funding decision for the calls
Call Coordination Team
Responsible for planning and organising the evaluation procedure
Topic Coordinator (and moderators)
Responsible at topic or sub-topic level for the evaluation and support to experts, including facilitating the discussion and finalisation of reports while ensuring fairness, consistency, objectivity and accuracy
Quality Controller
Responsible for quality checking of the reports and provide feedback
• Evaluate proposals
• Prepare and submit Individual Evaluation Reports (IERs)
remotely
• Exchange views with other evaluators to reach a
consensus on a proposal (remotely or in Brussels)
• Approve Consensus Reports (CRs)
Your role
Rapporteur
Evaluator
• May or may not evaluate proposals
• Prepare draft Consensus Reports (CRs)
• Participate in consensus meetings remotely or in Brussels
to finalise the CRs
• Do not evaluate proposals
• Do not participate in consensus
• Perform quality control checks on CRs and provide
feedback to evaluators and rapporteurs
• Do not evaluate proposals
• Monitor the entire evaluation process checking
transparency and observance of relevant rules and
procedures
• Prepare a report with recommendations and suggestions
for improvement of the process
Super-Rapporteur
(Quality Controller)
Observer
Independence
• You are evaluating in a personal capacity
• You represent neither your employer, nor your country!
Impartiality
• You must treat all proposals equally and impartially on their merits, irrespective of
their origin or the identity of the applicants
Objectivity
• You evaluate each proposal as submitted; meaning on its own merit, not its
potential if certain changes were to be made
Accuracy
• You make your judgment against the official evaluation criteria and the call or topic
the proposal addresses, and nothing else
Consistency
• You apply the same standard of judgment to all proposals
Guiding Principles
Confidentiality rules are detailed in your contract – Annex 1
Before, during or after the evaluation you must:
• Not discuss evaluation matters with anyone including other EC staff (e.g. content of proposals or evaluation results)
• Not contact partners in the consortium, sub-contractors or any third parties
• Not disclose the names of your fellow experts
• Maintain the confidentiality of all evaluation documents
• Destroy or delete all documents, paper or electronic, upon completing your work
Confidentiality
Conflict of Interest rules are detailed in your contract – Annex 1
• You must inform the EC as soon as you become aware of a potential CoI
• Before the signature of the contract
• Upon receipt of the proposals, or
• During the course of your work
• Please discuss any potential CoI-related doubt with your topic coordinator before
rejecting a task in the evaluation system
• If you knowingly hide a CoI, you will be excluded from the evaluation and your work
declared null and void
• The allowance/expenses you claimed may be reduced, rejected or recovered
• Your contract may be terminated
Conflicts of Interest (CoI)
• Please respect the deadlines set by your topic coordinator
• This is part of your contractual obligations!
• The allowance/expenses you claim may be reduced or rejected otherwise
If you suspect any form of misconduct (e.g. plagiarism, double funding), please report this to your topic
coordinator
Deadlines
Experts
Individual
evaluation
IndividualEvaluationReports
Consensus
ConsensusReport
Receipt of
proposals
Eligibility check
Allocation of proposals to
experts
EC
Finalisation
Evaluation Summary Report
EC ranked list
Information sent to applicants
EC
Evaluation Steps
Main elements of a proposal
Part A
- General Information
- Administrative Data
- Budget
Part B
Section 1-3 (limit to 10 pages)
1. Excellence
2. Impact
Evaluation Tool (SEP)
• Evaluators work individually and evaluate the proposal(s)
against the evaluation criteria and provide explanatory
comments and scores for each criterion – Individual Evaluation
Report (IER)
• Without discussing the proposal with anybody
• On the proposal as submitted - not on its potential if certain changes
were to be made
• Excess pages or information included in sections originally
foreseen for other purposes should be disregarded
• In case of doubt please ask your topic coordinator
Individual Evaluation Phase
Individual
Evaluation
Individual
EvaluationReport
• Evaluators come together in a consensus group, to agree on a
common position (comments and scores) and finalise the
Consensus Report (CR)
• Discussion is on the basis of the individual evaluations and the draft
consensus report
• The expert appointed as rapporteur is responsible for drafting
the CR on the basis of the IERs
• In some cases, the rapporteur is not one of the evaluators
• The CR is finalised when all experts approve it in SEP
Consensus phase
Consensus
Consensus
Report
Draft
Consensus
Report
• The draft CR aims to facilitate the discussion and work during the consensus phase
• The quality of the CR is of utmost importance, the aim is to give:
• A clear assessment of the proposal based on its merit, with justification
• Clear feedback of adequate length and in an appropriate tone
• Explanations on shortcomings without making recommendations
• The CR is a compromise report that balances the views of the different
evaluators
Consensus report (CR)
There are three evaluation criteria for full proposals
• Excellence
• Impact
• Implementation
• For short proposals (10 pages), you only evaluate the criteria Excellence and
(part of) Impact
• Each criterion covers different sub-criteria adapted to the type of action (as specified in
the WP)
Evaluation Criteria
You give a score of between 0 and 5 to each criterion based on your comments
• The whole range of scores should be used
• Half-marks can be used
• Scores must pass thresholds if a proposal is to be considered for funding
Proposal Scoring
Thresholds
Excellence 4
Impact 4
Overall Dynamic (min 8)
The level of threshold will be
set so that success rate at
stage 2 will be as close as
possible to 1:3 (in terms of
budget)
The proposal fails to address the criterion or cannot be assessed due to missing or
incomplete information.
Poor. The criterion is inadequately addressed, or there are serious inherent
weaknesses.
Fair. The proposal broadly addresses the criterion, but there are significant
weaknesses.
Good. The proposal addresses the criterion well, but a number of shortcomings are
present.
Very Good. The proposal addresses the criterion very well, but a small number of
shortcomings are present.
Excellent. The proposal successfully addresses all relevant aspects of the criterion.
Any shortcomings are minor.
Scoring
0
1
2
3
4
5
• While you should evaluate (and score) each criterion as a whole, the sub-criteria
should be used as a guidance to ensure the completeness of your assessment.
• Proposals with significant weaknesses, that prevent the project from achieving its
objectives or with resources being seriously over-estimated, must receive below -
threshold scores
• Any proposal with scores above the thresholds will be invited to the two stage
phase
Proposal Scoring
• Is only marginally relevant in terms of its scientific, technological or innovation
content relating to the topic addressed, you must reflect this in a lower score for the
Excellence criterion
• No matter how excellent the science!
• Does not significantly contribute to the expected impacts as specified in the WP
for that call or topic, you must reflect this in a lower score for the Impact criterion
If a proposal:
1. Go to "SEP", the online system for proposal evaluation. You should receive a
notification by e-mail for each task assigned
2. Access each proposal by clicking on the acronym or number of the proposal in the
Active Tasks tab.
3. Screen the information on the proposal (Part A and Part B) to check for potential
Conflicts of Interest. You need to accept to work on the task to have access to
part B. If you have any potential CoI discuss it with your topic coordinator before
declining the tasks in the system
4. Study the briefing material and if being organised attend the topic web-briefing
IER – Getting started
5. Read each proposal in detail. Go through each evaluation criterion and write your
comments for every aspect mentioned. Reply to the additional questions.
6. Review your facts – when you cite facts, are they truly in the proposal and will you
be able to find them back during the Consensus?
7. Score each evaluation criterion! Use the full scale of scores! Check that your scores
match your comments.
8. Save the draft report in SEP by the deadline indicated by your topic coordinator. You
can ask him/her to provide you feedback
9. If needed, revise your assessment and submit the report
IER – Getting started
1. Go to "SEP", the online system for proposal evaluation and check the tasks assigned
to you in the Active Tasks tab
2. Access each proposal by clicking on their acronym or number
3. If you did not evaluate the proposals, screen them (Part A and Part B) to check for
potential Conflicts of Interest. You need to accept to work on the task to have
access to part B. If you have any potential CoI discuss it with your topic coordinator
before declining the tasks in the system
4. Read all the Individual Evaluation Reports and identify points of convergence and
divergence. You can use the functions "Merge" and "Initialise" of SEP to help you in
your work.
draft CR – Getting started
6. Propose a consensus wording for the convergent points highlighting strengths and
weaknesses; and identify the divergences as points for discussion with pros and
cons to guide the consensus
7. If needed, refer to the proposal for checking factual references
8. Save the report in SEP within the set deadline. You can ask your topic coordinator to
provide you feedback
9. During the consensus review and update your draft CR to reflect the outcome of the
discussion
10. If you did not evaluate the proposal do not express your own views
draft CR – Getting started
1. Read the draft Consensus Report
2. Use the comments box feature in SEP to discuss and agree on the comments and
scores with the other experts. Keep in mind that it is normal for individual views to
change after arguments are exchanged and that "outlying opinions" might be as valid
as others
3. The rapporteur will revise the draft to reflect the discussion
4. If needed, the topic coordinator might convene a phone meeting to finalise the
discussion
5. The consensus will be finalised when all experts approve the draft CR
Remote Consensus – Getting started
• Reports should provide clear feedback on the proposal using terms that are in the scoring table, as not leave
doubt for the reader
1. Provide an overall assessment for elements with an explanation. Think: excellent, very good, good, fair, poor, fail.
2. Explain positive points and strengths (if any)
3. Consider negative points (if any) and qualify each negative point to show its significance. Think: minor shortcoming,
shortcoming, significant weakness, serious inherent weakness.
• Good comments and reports are fair to applicants and close the issue. Poor comments and reports provide an
opening for a complaint
Tips for Reports
• A “minor shortcoming” is an issue that relates only to a marginal aspect of the proposal with respect to the criterion
and/or can easily be rectified
• A “shortcoming” is a problem that has been identified but does not render the proposal inappropriate for funding, i.e.
it will still produce useful results and have impact
Tips for Reports
• Comments should contain assessments, be substantial and of adequate length and
related to the criterion being evaluated.
• Use clear language, that can be easily understood
• Be precise and knowledgeable. Include words like "Because", "Specifically", "For
example"
• Be factual. “This proposal is...” not “I think that...”, "Perhaps", "Probably"
• Avoid categorical or short and superficial comments
• Be aware of the danger of 'dominating negatives' – don’t forget to mention the good
points as well!
• Ensure each argument is put under the right criterion and don’t penalise a proposal
twice for the same negative point.
Tips for reports
Have you fully explained this proposal’s strengths and weaknesses for all evaluation criteria?
Have you written at adequate length?
Do comments match scores? (high scores = positive comments, low scores = negative comments)
Would you give the same score based on the comments you read?
Have you addressed all aspects of each criterion?
Have you checked any matters-of-fact which you have quoted?
Final check-list
• For SEP issues - please contact the IT helpdesk:
• Phone: +32 2 29 92222
• Available on weekdays 9:00 -18:00 CET
• For reimbursement issues - please contact:
• For any other issues:
Who can support you?
• H2020 SC5 Work-Programme & General Annexes
• Topic briefing material (password sent by e-mail)
• Frequently Asked Questions
• Proposal Templates
• H2020 Online Manual – Experts role and tasks
Essential References for your work
Thank you
© European Union 2020
