WP5 MONITORING AND EVALUATING THE JPIjpi-ch.eu/wp-content/uploads/06JHEP-Mid-Term_WP5_MonitEval1.pdf · JPIs To Co Work Project aims to provide a platform to discuss good practice

JHEP Coordination Action in support of the implementation

of a Joint Programming Initiative (JPI)on Cultural Heritage and Global Change :

a new challenge for Europe "

MONITORING AND EVALUATING THE JPI

WP5

1Sylvie MAX-COLINARTVincent ISRAEL-JOST French Ministry of Culture and Communication

JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting

Brussels, 16 May 2013Brussels, 16 May 2013

OBJECTIVES OF WP5

Evaluation of the level of success of the JPI in tw o areas :

• Implementation of the Strategic Research Agenda and Action Programme(as defined by WP2 and WP3)

• Effectiveness of the pooling of national expertise and resources in order toestablish robust collaborations among participating states

2Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication



Description of work

WP5 is structured in 3 tasks:

• Task 5.1: Definition of the methodological framework

• Task 5.2: Monitoring and evaluatingDuring the lifetime of the Co-ordination action (CA), the methodological framework willbe implemented. The indicators will be measured according to the methodology andthe timeframe defined in the task 5.1.

• Task 5.3: Final recommendations for monitoring andevaluationThese final recommendations will revise the methodological framework (5.1)according to the actual evaluation performed in 5.2. and will allow for the monitoringand the evaluation of the JPI during its lifetime.




Oct 201

1

Oct 201

2

Oct 201

3

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36

WP 5Task 5.1 / 15 moisTaks 5.2 / 13moisTask 5.3 / 8 mois

WP2 SRAWP3 Implementation

Task 5.1 / 15 months / Jully 2012 to 31 st Sept 2013Taks 5.2 / 13 months / Sept 2013 to EndTask 5.3 / 8 months / Fev 2014 to End

March 2013

August 2013

March 2014

Sept 2014

Definition of the methodological framework (month 18)

Report on methodology, definition of indicators (month 22)

Evaluation of the implementation of the JPICH (month 30)

Report on the implementation of monitoring and evaluation (month 36)

Recommendation for future monitoring and evaluation activities

MS7

D5.1

MS8

D5.2

MS7

4

WP5 SCHEDULE

Sylvie MAX-COLINARTFrench Ministry of Culture and Communication JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting


Task 5.1 : Definition of the methodological framework (month 10 to 24)

This task will develop the methodological framework for the evaluation ofthe JPI which will be designed in order to be usable during the lifetime ofthe JPI.

• Key areas of the JPI will be identified.

• A set of indicators to assess the implementation of the JPI in the short,medium and long term will be developed.

• Indicators of success will be identified for each objective and key area ofthe JPI.




Each indicator will be accompanied by:

- a methodology for its measure (how to measure it ?) : use of internalinformation (questionnaire…) or external sources (publications)…

- an indicative target (if applicable)

- a timeframe (when to measure it ?)

The definition of the indicators will take into account their sustainability




A pool of experts appointed by the Member States and AssociatedCountries will participate to the validation of indicators together withthe Scientific Committee.




JPIs

8

WP5 – Context of the Evaluation framework

WP1 WP2 WP3 WP5 WP4 WP5 WP6

EVALUATION and MONITORING

Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication



Bibliographyand documents fromvarious JPI

JPIs To Co Work Project aims to provide a platform to discuss goodpractice and options for establishing best format o f the “frameworkConditions” for JPIs.

This project has received funding from the European Commission in the SeventhFramework Programme (FP7-ERANET-2011-RTD)

February 2013, Brussels meeting on evaluation : The JPIs To Co Worksuggests that all JPIs could have (part of) a commo n evaluationframework.

Some Questions raised:

- How much can different JPIs share in terms of their evaluation?

- Who would do the evaluation (externalise? Externalise only the commonpart?)




9

WP5 – Evaluation framework

Task 5.1 implied that we would gather a group of ex perts to assist usin discussing various points:

- logical framework (typology of indicators)

- identification of JPI CH main and detailed objecti ves

- definition and use of indicators

In October 2013, e-mails to the GB members to asked them to proposenames of experts.

The group was constituted early in January with the followingparticipants:




WP5 Pool of Experts

Belgium Mrs Sara Van Rompaey Executive Agency for Competitiveness andInnovation (EACI)

Belgium Mr Koen Van Balen Univ. Leuven / Chair of the JPI CH SC

Cyprus Mr Petros Pashiardis Open Univ. Cyprus

Denmark Mr Kim Christian Schroeder Roskilde Univ.

Ireland Mr Brendan Curran Health Res. Board

Italy Mrs Anna Misiani MIBAC

Netherlands Mrs Maartje de Boer Cultural Heritage Agency

Netherlands Mrs Eva Stegmeijer Cultural Heritage Agency

Poland Mrs Iwona Szmelter Polish Academy of Fine Arts

Spain Mrs Laura Hernandez FECYT

UK Mrs Laura Lugg AHRC

France Mr Roberto Casati Institut Jean Nicod/CNRS




12

WP5 – Task 5.1 Evaluation framework

15th March 2013: First meeting of the pool of experts i n Parisdedicated to the definition of the methodological f ramework(milestone MS7)

This methodological framework is not entirely settl ed yet — we ofcourse need the SRA to be validated to circulate it within the poolof experts — good progress have been made thank to this firstworkshop.

Many points have been discussed and could be summar ised here




WP5 – Task 5.1 Evaluation framework: Logical framework

Before proposing a list of indicators, we have disc ussed the “logicalframework”, that is, a way to rationally organize a nd classifyindicators.

This is usually performed by relating categories of indicators todifferent types of goals, according to a scheme as follows:




Challenges (Vision document/ SRA) Objectives / Research priorities (SRA)

Impact Outcome Output Actions Input

Implementation


Traditional typology of indicators : Input, Output, Outcome, Impact

This typology was enriched inJPND with also type A and typeB indicators:




Input

Output

Outcome

Impact

Type A (policy/funding)

Type B (scientific/societal)

Challenges (Vision document/ SRA) Objectives / Research priorities (SRA)

Impact Outcome Output Actions Input

Implementation

Proposal: Four categories, defined by the domains of application of the results


JPI Management& Actions

Science policy Scientific results Societal, economic,environmental

effects

The validation of the logical JPI CH framework is still pending




15

16

WP5 – Task 5.1 Evaluation framework: identification ofchallenges/objectives/ research priorities

This identification is key to the evaluation and mo nitoring process.

So far, we have relied on the Vision document that conveniently lists Challenges,Objectives and Research Questions.

However, the SRA draft shows significant differences with the Vision documentregarding priorities.

Ex.: Climate change is listed as first challenge in the Vision document and a mereresearch question in the SRA.

We certainly need to adapt the indicators to the updated priorities of JPICH.




WP5 – Task 5.1 Evaluation framework: Proposing indicators

A significant part of them (white background) is directly taken from JPND.

We have discussed and revised those more ‘polyvalent’ indicators(across JPIs) with experts.

The rest of indicators (colored boxes) depends directly on JPICH and itspriorities.Some have already been proposed but they are still under construction,following the SRA.




17

JPI Implementation Science policy Scientific results S ocietal effects

Participation of Member States inManagement Board meetings (a.1)

National research (funding)priorities adapted as a result ofJPICH and its research strategy(a.9)

Coordination of European prevention,emergency action and post disastermeasures against natural and man-made hazards (2.1)

The amount of JPICH commonresearch funding for cultural heritageas share of total EU research funding inthis domain (a.14)

Attitude towards JPICH goals andobjectives (a.2)

Alignment of national researchfunding programmes (a.10)

Develoment and application ofinnovative technologies for treatment,repair and maintenance of culturalheritage (3.1,3.2,3.4 & 3.5)

Total amount of European fundingavailable for cultural heritage (a.15)

Number of new joint transnational callsfor proposals (a.6)

Training and mobility ofresearchers across the partners'research insitutions (5)

Encouraging innovations,collaborations with European industry

Share of granted projects versusnumber of applications (a.7)

Sharing methods, protocols,infrastructures from local (regional) toglobal (European) levels

Number of new initiatives for non-project funded activities (a.8)

Raising public awareness of research

Number of collaborative researchprojects funded through JPICH jointcalls that address the various scientificpriorities (b.1)

Long term develoment of the CulturalResearch Area

Number of non-project funded activitiesthat address the various scientificpriorities (b.2)

Relation between cultural heritage andclimate change

Protection and security of culturalheritage

Transformational challenge




18

WP5 – Task 5.1 Evaluation framework: Future developments

- Validation of the logical framework, the identification of objectives andthe indicators by the pool of experts.

- In practice: work by e-mail with experts, second meeting, first weeks of July.

- D5.1 (Report on methodology) is scheduled for September.




19

Documents

WP5 MONITORING AND EVALUATING THE JPIjpi-ch.eu/wp-content/uploads/06JHEP-Mid-Term_WP5_MonitEval1.pdf · JPIs To Co Work Project aims to provide a platform to discuss good practice