Evaluator’s view Evaluator’s view

Borka Jerman-BlažičBorka Jerman-BlažičUniversity of Ljubljana andUniversity of Ljubljana and

Jožef Stefan InstituteJožef Stefan InstituteSLOVENIA SLOVENIA

Acting as an evaluatorActing as an evaluator

Evaluation process is very Evaluation process is very demanding procedure which demanding procedure which involves:involves:

commission officers and staff commission officers and staff andand

number of evaluatorsnumber of evaluators

Acting as an evaluatorActing as an evaluator

Evaluators are usually people Evaluators are usually people with expertise in the subject with expertise in the subject addressed by the call and are addressed by the call and are coming from different coming from different environments:environments:

AAcademia, industry, professional cademia, industry, professional societies, research institutes, societies, research institutes, national institutions dealing with national institutions dealing with research etc research etc

InvitationInvitation

The selection of evaluators is based The selection of evaluators is based on the data in CORDIS data baseon the data in CORDIS data base

If you want to become an evaluator If you want to become an evaluator you must register in the respective you must register in the respective data base of evaluators for particular data base of evaluators for particular programprogram

Sometimes, the selection process for Sometimes, the selection process for evaluators is supported through evaluators is supported through additional channels additional channels

InvitationInvitation

Selection: you are selected according to your Selection: you are selected according to your expertise and data recordsexpertise and data records

You are selected because you have entered You are selected because you have entered information about you in the CORDIS data base information about you in the CORDIS data base and because your expertise is in line with the and because your expertise is in line with the scientific/technical objectives of the relevant callscientific/technical objectives of the relevant call

So, be careful when entering data about your So, be careful when entering data about your expertise, level of education and referencesexpertise, level of education and references

Every call has different objectives and as a Every call has different objectives and as a consequence the selection of the evaluators consequence the selection of the evaluators follows their expertise and in each call is different follows their expertise and in each call is different

After the selection an invitation is launched usually After the selection an invitation is launched usually via e-mail via e-mail

The evaluator appointment The evaluator appointment start with:start with: An invitationAn invitation Your agreementYour agreement Your correct responseYour correct response Your on time provision of the Your on time provision of the

requested data by the Commission requested data by the Commission Final selection/invitation comes later Final selection/invitation comes later

when all proposals are known, the when all proposals are known, the call is closed and the number of call is closed and the number of needed evaluators is finally decidedneeded evaluators is finally decided

The number of evaluators depends The number of evaluators depends on the number of received proposalson the number of received proposals

The information requested by The information requested by the evaluatorthe evaluator AvailabilityAvailability Legal entity form (each time you are Legal entity form (each time you are

asked to act as an evaluator)asked to act as an evaluator) Bank information (if the procedures Bank information (if the procedures

do not change one signed document do not change one signed document by the bank and yourself is sufficient by the bank and yourself is sufficient for at least one year)for at least one year)

Copy of your ID documentCopy of your ID document

Information requestedInformation requested

You have to agree to the conditions You have to agree to the conditions of work as stated in the Commission of work as stated in the Commission documents sent to you documents sent to you

Confidentiality Confidentiality agreement/commitment of non-agreement/commitment of non-disclosure anddisclosure and

Conflict of interest declarationConflict of interest declaration Reimbursement of your personal Reimbursement of your personal

cost and feecost and fee

There are two options for an There are two options for an evaluatorevaluator You can act as independent expert orYou can act as independent expert or as employee of an organisation (if you are as employee of an organisation (if you are

employed)employed) The conditions regarding payment are The conditions regarding payment are

slightly different but the fee per day is slightly different but the fee per day is same same

There are two forms of reimbursement: the There are two forms of reimbursement: the fee per working day and reimbursement of fee per working day and reimbursement of travel and accommodation cost, both are travel and accommodation cost, both are specified in the accompanying documents specified in the accompanying documents

The selection processThe selection process

After the decision is taken for your After the decision is taken for your engagement as an evaluator the engagement as an evaluator the appointment procedure follows:appointment procedure follows:

you receive appointment letter where you receive appointment letter where all conditions of work are specified all conditions of work are specified as well your duties and commitments as well your duties and commitments (confidentiality/non disclosure and (confidentiality/non disclosure and conflict of interest declaration)conflict of interest declaration)

AppointmentAppointment

The appointment letter is a sort The appointment letter is a sort of contract with Annexesof contract with Annexes

You must sign the letter and the You must sign the letter and the enclosed documents and send enclosed documents and send them back to the Commission them back to the Commission officerofficer

Instructions for your work follow Instructions for your work follow as wellas well

On line and on site evaluationOn line and on site evaluation

Travel and accommodation is organized by Travel and accommodation is organized by yourselfyourself

The place of evaluation is known (for FP6 The place of evaluation is known (for FP6 this the building on Square Frere Orban)this the building on Square Frere Orban)

The date of evaluation is known as wellThe date of evaluation is known as well Some programs (e.g. Marie Curie) practice Some programs (e.g. Marie Curie) practice

individual evaluation on distance at your individual evaluation on distance at your sitesite

You receive evaluation guidelines for You receive evaluation guidelines for specific call that includes guidelines for specific call that includes guidelines for your work, evaluation forms and guidelines your work, evaluation forms and guidelines how to fill themhow to fill them

Commitments and dutiesCommitments and duties

If you are evaluating on distance then you If you are evaluating on distance then you are provided with the proposal on line via are provided with the proposal on line via secured access to the networksecured access to the network

You have restricted time to do the You have restricted time to do the evaluation (1-2 weeks) but you are more evaluation (1-2 weeks) but you are more free in planning your timefree in planning your time

You may influence on the selection of You may influence on the selection of proposals (by reading the abstracts) to be proposals (by reading the abstracts) to be evaluated but this is not always possible as evaluated but this is not always possible as other evaluators are also pointing to the other evaluators are also pointing to the same proposalsame proposalss

All your evaluation data are entered as well All your evaluation data are entered as well on line via secure routeon line via secure route

Consensus is achieved at site – in BrusselsConsensus is achieved at site – in Brussels

Evaluation in FP6Evaluation in FP6

All proposal are receipt, opened, All proposal are receipt, opened, acknowledged and their content entered acknowledged and their content entered into a data base by the Commission into a data base by the Commission officials to support the evaluation processofficials to support the evaluation process

Basic eligibility criteria for each proposal Basic eligibility criteria for each proposal are also checked by the Commission staff are also checked by the Commission staff before evaluation beginsbefore evaluation begins

Commission officials assign particular Commission officials assign particular number of proposals to the panel of number of proposals to the panel of experts and to particular evaluatorexperts and to particular evaluator

Before you start to evaluate a briefing Before you start to evaluate a briefing meeting is organized for all evaluators meeting is organized for all evaluators involved involved

Evaluation in FP6: your roleEvaluation in FP6: your role

Commission staff do not influence the Commission staff do not influence the opinion of an independent expert, if asked opinion of an independent expert, if asked they providethey provide to you to you only additional only additional information or assistanceinformation or assistance

Each independent expert is assigned cca 5 Each independent expert is assigned cca 5 to 6 proposals if they are IP or NoE, in to 6 proposals if they are IP or NoE, in case of STREPs that may be slightly morecase of STREPs that may be slightly more

Your working time is from 9 to 17 but Your working time is from 9 to 17 but usually the evaluators work much more as usually the evaluators work much more as the evaluation is very demanding processthe evaluation is very demanding process

The evaluation process in FP6The evaluation process in FP6

The working conditions are good and The working conditions are good and the staff is supporting and kindthe staff is supporting and kind

You are supposed NOT to talk to You are supposed NOT to talk to other experts about the proposals other experts about the proposals you are evaluatingyou are evaluating

The place to do so are the The place to do so are the consensus meetings where all consensus meetings where all appointed evaluators for particular appointed evaluators for particular proposal discuss for common proposal discuss for common evaluation reportevaluation report

The evaluation process in FP6The evaluation process in FP6

Your task is to read carefully the Your task is to read carefully the proposal document and to proposal document and to prepare an IAR – individual prepare an IAR – individual assessment reportassessment report

The report has several fields The report has several fields where you put your scores and where you put your scores and justification justification

Confidentiality/conflict of Confidentiality/conflict of interestinterest It is forbidden the evaluator to work It is forbidden the evaluator to work

closely with the proposer or to be closely with the proposer or to be involved in the preparation, the involved in the preparation, the evaluator should not attend the panel evaluator should not attend the panel meeting where such proposal is meeting where such proposal is discusseddiscussed

In case of indirect conflict of interest In case of indirect conflict of interest (proposal is coming from the (proposal is coming from the institution the evaluator is employed) institution the evaluator is employed) the evaluator should not be involved the evaluator should not be involved in evaluation of such proposal and in evaluation of such proposal and have to indicate such casehave to indicate such case

Confidentiality/conflict of Confidentiality/conflict of interest interest All data, documents, writings, comments All data, documents, writings, comments

etc must stay in the place of evaluation. etc must stay in the place of evaluation. Under no circumstances may an evaluator Under no circumstances may an evaluator

attempt to contact attempt to contact proposal submitter proposal submitter on on his own account, either during the his own account, either during the evaluation or afterwards.evaluation or afterwards.

It is also NOT recommended to visit WEB It is also NOT recommended to visit WEB references even if they are mentioned in references even if they are mentioned in the proposal especially if the information on the proposal especially if the information on the WEB are placed after the date of the WEB are placed after the date of proposal submissionproposal submission

Evaluation process: your roleEvaluation process: your role Assess and mark the proposal exactly as it Assess and mark the proposal exactly as it

is described and presented, do not make is described and presented, do not make any assumany assumpptions or interpretations about tions or interpretations about the project in addition to what the the project in addition to what the proposers have written in their proposalproposers have written in their proposal

Keep to the evaluation criteria as stated in Keep to the evaluation criteria as stated in the evaluation formsthe evaluation forms

Give marks and comments and write them Give marks and comments and write them in readable shape in readable shape

You should care the comments to be You should care the comments to be written in line withwritten in line with th the the marks e the marks

Maintain consistency in your scoring Maintain consistency in your scoring through the work through the work

Provide a brief but explicit justification of Provide a brief but explicit justification of your marks, be honest, but correct. The your marks, be honest, but correct. The language especially in case of low scores language especially in case of low scores must be correct as the comments will be sent must be correct as the comments will be sent to the proposers, you may use short extracts to the proposers, you may use short extracts from the proposal text to justify your opinion.from the proposal text to justify your opinion.

Where justified you may give Where justified you may give recommendations, but ensure that the recommendations, but ensure that the scores given reflect the proposal as scores given reflect the proposal as presented by the applicantpresented by the applicant

It is recommendable before the start of the It is recommendable before the start of the evaluation you to examine a number of evaluation you to examine a number of proposals before signing off their first proposals before signing off their first individual assessment forms. This will help to individual assessment forms. This will help to calibrate their scoringcalibrate their scoring

Evaluating: the good and the Evaluating: the good and the badbad Good proposals usually have clearly written text, Good proposals usually have clearly written text,

not excessive wording, but text that provide good not excessive wording, but text that provide good information for the evaluator to understand the information for the evaluator to understand the objectives and the presented workobjectives and the presented work

The objectives must be well set up and justified, The objectives must be well set up and justified, very clearlyvery clearly

The methods to achieve the objectives should be The methods to achieve the objectives should be sound, from technical and from management point sound, from technical and from management point of viewof view

There must be a red line along the proposal from There must be a red line along the proposal from the beginning to the end the beginning to the end

The proposal should look as compact package that The proposal should look as compact package that is oriented to achieve common joint goalsis oriented to achieve common joint goals

Evaluating: the good and the Evaluating: the good and the badbad The idea of the proposal (what exactly is planned to The idea of the proposal (what exactly is planned to

be done) described must innovative. The be done) described must innovative. The innovationmay have different aspectsinnovationmay have different aspects

Evaluators are not always very deeply involved in Evaluators are not always very deeply involved in particular specific field, so good proposal contain particular specific field, so good proposal contain short but convincing state of the art of the field or short but convincing state of the art of the field or subject background that justify the need for subject background that justify the need for specified and identified problem to be solvedspecified and identified problem to be solved

Good proposal contain text or description that Good proposal contain text or description that justify the project funding from public money: the justify the project funding from public money: the proposer must present enough evidence why the proposer must present enough evidence why the proposed way of research or proposed research proposed way of research or proposed research topic is carried out on European level and why topic is carried out on European level and why deserve public fundingdeserve public funding

Evaluating: the good and the Evaluating: the good and the badbad Each criteria from the IAR is important, so the proposal Each criteria from the IAR is important, so the proposal

must show more than average merit in all subjectsmust show more than average merit in all subjects This implies that the proposal must be well thaught and This implies that the proposal must be well thaught and

well prepared in all aspectswell prepared in all aspects Scientific and technical value are most important, Scientific and technical value are most important,

meaning that innovation must be present. Beside that, meaning that innovation must be present. Beside that, the constitution of the conzortium must be balanced, the the constitution of the conzortium must be balanced, the proposal must show good balance of all relevant proposal must show good balance of all relevant institutions, from industry, academia, SMEs, professional institutions, from industry, academia, SMEs, professional societies ectsocieties ect..

Distribution of the resources is also very important. The Distribution of the resources is also very important. The proposal must convince the evaluator that the allocation proposal must convince the evaluator that the allocation of the work and the resources are well prepared and that of the work and the resources are well prepared and that they give or show enough guarantee that the conzortium they give or show enough guarantee that the conzortium is capable to carry out the project up to the is capable to carry out the project up to the accomplishment of the goalsaccomplishment of the goals

Evaluating: the good and the Evaluating: the good and the bad bad All criteria from the IAR contribute to the overall ranking of All criteria from the IAR contribute to the overall ranking of

the proposal, in that context the proposal, in that context management part must be also prepared in line with the management part must be also prepared in line with the

type of the proposed worktype of the proposed work Information about involved institution and the major staff Information about involved institution and the major staff

must be provided as well. This information contribute to must be provided as well. This information contribute to the trust building among the evaluators that there is the trust building among the evaluators that there is enough capacity, knowledge and experience for enough capacity, knowledge and experience for successful accomplishment of the project objectivessuccessful accomplishment of the project objectives

The proposer must convince you during text reading that The proposer must convince you during text reading that his idea is innovative, sound and the methods proposed his idea is innovative, sound and the methods proposed for work are the most apfor work are the most appproprropriiate for the presented ate for the presented problem to be solved or idea implementedproblem to be solved or idea implemented

This implies enough information about the conzortium This implies enough information about the conzortium expertise, skills, knowledge, previous engagement, expertise, skills, knowledge, previous engagement, published references or similar data. The project usually published references or similar data. The project usually benefits from CVs of involved people and from good benefits from CVs of involved people and from good description of the institutions consisting the conzortiumdescription of the institutions consisting the conzortium

Evaluation in FP6: Consensus Evaluation in FP6: Consensus buildingbuilding One among the evaluators is appointed as One among the evaluators is appointed as

a raporteura raporteur NoE and IP have 5 evaluators + a NoE and IP have 5 evaluators + a

raporteurraporteur STREPs have 3 evaluatorsSTREPs have 3 evaluators The task of the raporteur is to accept and The task of the raporteur is to accept and

record the majority view of the other record the majority view of the other evaluators, together with the Commission evaluators, together with the Commission officer may bring additional evaluator(s) (up officer may bring additional evaluator(s) (up to 3) in case no consensus is achievedto 3) in case no consensus is achieved

The face to face consensus meeting is The face to face consensus meeting is usually convened by the officer usually convened by the officer

Evaluation FP6:consensus Evaluation FP6:consensus meetingmeeting The evaluators are usually presenting their The evaluators are usually presenting their

scores and the justifications then they scores and the justifications then they discuss in order to achieve common scores discuss in order to achieve common scores and commentsand comments

If this is achieved then the raporteur write If this is achieved then the raporteur write the report and all evaluators must agree the report and all evaluators must agree with it and sign it, the raporteur is with it and sign it, the raporteur is responsible for collecting the signaturesresponsible for collecting the signatures

Then the consensus report goes to the Then the consensus report goes to the panel panel

The panel consist of all experts from The panel consist of all experts from particular area of the call e.g. BBAll, particular area of the call e.g. BBAll, Security etcSecurity etc

Evaluation in FP6: Consensus Evaluation in FP6: Consensus meetingmeeting If consensus has not been reached, the report If consensus has not been reached, the report

sets out the majority view of the evaluators, sets out the majority view of the evaluators, but also records any dissenting views. In case but also records any dissenting views. In case any such disagreement has not been resolved any such disagreement has not been resolved a threshold score or average score given by a threshold score or average score given by the evaluators will be awarded for the the evaluators will be awarded for the proposal. proposal.

All consensus reports are entered in the data All consensus reports are entered in the data base and joint list of proposals ranked base and joint list of proposals ranked according the overall scores is generatedaccording the overall scores is generated

This list is discussed by the panelThis list is discussed by the panel The panel sometimes may agree to do some The panel sometimes may agree to do some

changes of the overall listing if this is explained changes of the overall listing if this is explained and justified e.gand justified e.g.. for some proposals covering for some proposals covering missing area in the call etc. missing area in the call etc.

Hearings and your role Hearings and your role

The best ranked NoEs and IPs by the panel are called The best ranked NoEs and IPs by the panel are called after few weeks to hearingsafter few weeks to hearings

Most of the experts panels are present at the hearings. Most of the experts panels are present at the hearings. Experts that have evaluated proposals presenting on Experts that have evaluated proposals presenting on the panel must be present at the hearingsthe panel must be present at the hearings

These meetings are convened by the Commission These meetings are convened by the Commission officialofficial

Evaluators after the presentation of the proposal that Evaluators after the presentation of the proposal that include mainly answers on the questions set up by the include mainly answers on the questions set up by the evaluators can set additional questionsevaluators can set additional questions

Questions are prepared iQuestions are prepared in n written form and passed to written form and passed to the convenorthe convenor

The presenters have no knowledge who was asking The presenters have no knowledge who was asking particular questionsparticular questions

Hearings are very helpful meetings as they clarify Hearings are very helpful meetings as they clarify many aspects that are not always evident from the many aspects that are not always evident from the written textwritten text

Decision makingDecision making

Outcomes of the hearings may Outcomes of the hearings may somehow change the ordering of the somehow change the ordering of the highly ranked proposal on the list highly ranked proposal on the list

After the hearings, the panel After the hearings, the panel convene and decide about final convene and decide about final rankingranking

With this your task as evaluator is With this your task as evaluator is almost finishedalmost finished

The further actions are connected The further actions are connected with the reimbursement formwith the reimbursement formss and and the final payment for your servicesthe final payment for your services

The panelThe panel

It is important the evaluators to have It is important the evaluators to have a look on the monitoring statistics a look on the monitoring statistics before they discuss the proposals at before they discuss the proposals at the panel meetingthe panel meeting

This will help to take in account the This will help to take in account the evaluators marking profile and that evaluators marking profile and that the other experts before revising the the other experts before revising the scores and the rankingscores and the ranking

The evaluators may agree on some The evaluators may agree on some recommendations regarding the recommendations regarding the negotiation processnegotiation process

CONCLUSIONCONCLUSION

If you are part of the new ERA If you are part of the new ERA or if you preparing yourself to or if you preparing yourself to take part then being an take part then being an evaluator is exciting experienceevaluator is exciting experience

You meet with your colleagues You meet with your colleagues and you contribute to the and you contribute to the success of the FP6success of the FP6

It is overall an exciting It is overall an exciting experienceexperience