117/3/2009

European Commission

Directorate General Information Society & Media

Briefing for Remote Reading

How to fill in the (IER)How to fill in the (IER)Individual Evaluation ReportIndividual Evaluation Report

217/3/2009

This tutorial gives guidelines on how to evaluate the proposals, i.e. it describes

• how to assess a proposal;• how to fill in the Individual Evaluation Form (IER);• scoring a proposal.

317/3/2009

When running the slide show, you can either go slide by slide, or navigate following the hyperlinks:

• access page• return to previous menu• Return to Contents page

Tutorial Manual

417/3/2009

1. Evaluator’s responsibilities

2. How to evaluate / how to fill in the IER

3. Scoring

4. The evaluation criteria

5. The evaluation forms (Call 4, Obj.1.1)

Contents

IP STREP

517/3/2009

• You have signed a non-disclosure agreement• Before, during & after the evaluation do not communicate any information

about the proposals

• You are independent• express your personal view (you do not represent your employer, country, …)

• You are the evaluator• do not delegate your responsibilities

• do not discuss the proposals with 3rd parties

• Do not contact the proposers for clarifications• proposals are evaluated on the information that is presented there, any lack of

information will downgrade the score

• Declare any potential conflict of interest • in case you suspect you have a conflict, contact us immediately

(see e.g. Call-4 Evaluation Handbook)

Evaluator’s Responsibilities

617/3/2009

• Give a fair and clear opinion on each proposal

• Give your personal views

• Evaluate proposals against the specific Objectives and Impact as defined in the Workprogramme

• Evaluate the proposal as written

• Consistently apply the same standard of judgment to each proposal

• Respect the code of conduct

How to evaluate - Introduction

717/3/2009

• Use the official evaluation criteria only• Proposals are evaluated on three criteria

• Scientific and technical quality • Implementation• Impact

• Assess the proposals in terms of all 3 evaluation criteria.• Each criterion is more fully defined by descriptive “bullet

points”, adapted to the instrument type. These are shown on the evaluation forms.

• Provide a relevant comment on each of the “bullet points” These sub-criteria are not scored individually or separately.

The Evaluation Criteria

817/3/2009

How to fill in IER - General

• Comments are confined only to the criterion concerned.• Comments describe only the final view of the proposal.• Start with drafting the comments, then adjust the score

accordingly.• Organise and structure your comment:

• Overall assessment • Followed by two or three illustrations (especially if the score is very

low or very high)• Followed by any comments which mitigate/reduce the overall

appreciation• For negative arguments, a distinction should be made between severe

shortcomings and recommendations for improvements.

917/3/2009

• Comments are of adequate length (not just one sentence !) • Do not be afraid to provide detailed comments. It is

valuable to have sufficient information to be able to understand how you reached your conclusions.

• A detailed comment is helpful for the preparation of the Consensus Report and the discussion that follows.

• Poor comments merely echo the score – Good comments explain it:• “This proposal does not adequately advance the state of the art.”• “This proposal fails to advance the state of the art in X or Y, it does

not take Z into account.”

IER- Level of detail

1017/3/2009

• Comments are clear and provide clear justifications.• Quote proposal text if useful.• Poor comments are vague - Good ones are precise and final :

• “We think the consortium management plan is probably inadequate given the duration of the project and the number of partners.”

• “The consortium management plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners.”

• Poor comments are ambiguous – Good comments are clear:• “The resources for the project are unrealistic.”• “The resources in Workpackages 4 and 6 are seriously

underestimated given the complexity of the activity involved.”

IER – Clear Messages

1117/3/2009

• Comments are substantial.• Comments are facts not opinions

• not “I think that....”• but “This proposal is.. .”

• Do not write generic comments such as “the proposal is not very innovative” without specifying where exactly it fails.

• Poor comments include words like:• “Perhaps, think, seems, assume, probably, …”

• Good comments include words like:• “Because, percent, specifically, for example, …”

IER – Factual Evidence

1217/3/2009

IER – Avoiding Conflicts

• Poor comments provide an opening for a complaint - Good comments close the question:• “There is no discussion of dissemination activities.”• “Dissemination activities are not adequately discussed.”

• “There are only two SMEs in the consortium.”• “The consortium lacks a sufficient SME participation.”

• “The proposal coordinator is not adequately experienced.”• “The proposal coordinator does not demonstrate in the proposal an

adequate level of experience of work in this field.”

1317/3/2009

IER – Varying the Vocabulary

Why say “Poor” when you can say:• Insufficient, minimal, fails to describe, unacceptable, inadequate,

very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile…

Why say “Excellent” when you can say:• Extremely relevant, credible, very clear, precisely specified,

realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …

1417/3/2009

IER – Final Checks

• Have you fully explained the proposal’s strengths and weaknesses on all criteria ?

• Do scores match comments (high scores = positive comments, low scores = negative comments) ?

• Have you highlighted any points needing special attention ?• Have you double-checked any matters-of-fact which you

have quoted ?• Have you written at adequate length ?• Overall comment = any comment not covered already in

individual criterion.

“If this was my proposal, would I find this report fair, accurate clear and complete?”

1517/3/2009

• Priority should be given to the comments as they will justify the outcome of the evaluation. Scores should be adjusted to reflect the opinion expressed in the comments. The scores are only a mathematical tool allowing the comparison of the relative quality of different proposals.

• A threshold applies to each individual evaluation criterion and there is an overall threshold. A proposal fails if it is below at least one of the thresholds (incl. overall threshold).

• The overall criterion is not scored separately but automatically calculated from the individual ones.

Scoring

1617/3/2009

The Scoring Scale

Use the full scale! Half marks may be given.0 The proposal fails to address the criterion under examination or cannot

be judged due to missing or incomplete information.1 Poor – The criterion is addressed in an inadequate manner, or there are

serious inherent weaknesses.2 Fair – While the proposal broadly addresses the criterion, there are

significant weaknesses. (can not be fixed @ negotiations)3 Good – The proposal addresses the criterion well, although

improvements would be necessary. 4 Very good – The proposal addresses the criterion very well, although

certain improvements are still possible.5 Excellent – The proposal successfully addresses all relevant aspects of

the criterion in question. Any shortcomings are minor.