JHEP Coordination Action in support of the implementation
of a Joint Programming Initiative (JPI)on Cultural Heritage and Global Change :
a new challenge for Europe "
MONITORING AND EVALUATING THE JPI
WP5
1Sylvie MAX-COLINARTVincent ISRAEL-JOST French Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
OBJECTIVES OF WP5
Evaluation of the level of success of the JPI in tw o areas :
• Implementation of the Strategic Research Agenda and Action Programme(as defined by WP2 and WP3)
• Effectiveness of the pooling of national expertise and resources in order toestablish robust collaborations among participating states
2Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
Description of work
WP5 is structured in 3 tasks:
• Task 5.1: Definition of the methodological framework
• Task 5.2: Monitoring and evaluatingDuring the lifetime of the Co-ordination action (CA), the methodological framework willbe implemented. The indicators will be measured according to the methodology andthe timeframe defined in the task 5.1.
• Task 5.3: Final recommendations for monitoring andevaluationThese final recommendations will revise the methodological framework (5.1)according to the actual evaluation performed in 5.2. and will allow for the monitoringand the evaluation of the JPI during its lifetime.
3Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
Oct 201
1
Oct 201
2
Oct 201
3
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36
WP 5Task 5.1 / 15 moisTaks 5.2 / 13moisTask 5.3 / 8 mois
WP2 SRAWP3 Implementation
Task 5.1 / 15 months / Jully 2012 to 31 st Sept 2013Taks 5.2 / 13 months / Sept 2013 to EndTask 5.3 / 8 months / Fev 2014 to End
March 2013
August 2013
March 2014
Sept 2014
Definition of the methodological framework (month 18)
Report on methodology, definition of indicators (month 22)
Evaluation of the implementation of the JPICH (month 30)
Report on the implementation of monitoring and evaluation (month 36)
Recommendation for future monitoring and evaluation activities
MS7
D5.1
MS8
D5.2
MS7
4
WP5 SCHEDULE
Sylvie MAX-COLINARTFrench Ministry of Culture and Communication JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
Task 5.1 : Definition of the methodological framework (month 10 to 24)
This task will develop the methodological framework for the evaluation ofthe JPI which will be designed in order to be usable during the lifetime ofthe JPI.
• Key areas of the JPI will be identified.
• A set of indicators to assess the implementation of the JPI in the short,medium and long term will be developed.
• Indicators of success will be identified for each objective and key area ofthe JPI.
5Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
Each indicator will be accompanied by:
- a methodology for its measure (how to measure it ?) : use of internalinformation (questionnaire…) or external sources (publications)…
- an indicative target (if applicable)
- a timeframe (when to measure it ?)
The definition of the indicators will take into account their sustainability
6Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
A pool of experts appointed by the Member States and AssociatedCountries will participate to the validation of indicators together withthe Scientific Committee.
7Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
JPIs
8
WP5 – Context of the Evaluation framework
WP1 WP2 WP3 WP5 WP4 WP5 WP6
EVALUATION and MONITORING
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
Bibliographyand documents fromvarious JPI
JPIs To Co Work Project aims to provide a platform to discuss goodpractice and options for establishing best format o f the “frameworkConditions” for JPIs.
This project has received funding from the European Commission in the SeventhFramework Programme (FP7-ERANET-2011-RTD)
February 2013, Brussels meeting on evaluation : The JPIs To Co Worksuggests that all JPIs could have (part of) a commo n evaluationframework.
Some Questions raised:
- How much can different JPIs share in terms of their evaluation?
- Who would do the evaluation (externalise? Externalise only the commonpart?)
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
9
WP5 – Evaluation framework
Task 5.1 implied that we would gather a group of ex perts to assist usin discussing various points:
- logical framework (typology of indicators)
- identification of JPI CH main and detailed objecti ves
- definition and use of indicators
In October 2013, e-mails to the GB members to asked them to proposenames of experts.
The group was constituted early in January with the followingparticipants:
10Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
WP5 Pool of Experts
Belgium Mrs Sara Van Rompaey Executive Agency for Competitiveness andInnovation (EACI)
Belgium Mr Koen Van Balen Univ. Leuven / Chair of the JPI CH SC
Cyprus Mr Petros Pashiardis Open Univ. Cyprus
Denmark Mr Kim Christian Schroeder Roskilde Univ.
Ireland Mr Brendan Curran Health Res. Board
Italy Mrs Anna Misiani MIBAC
Netherlands Mrs Maartje de Boer Cultural Heritage Agency
Netherlands Mrs Eva Stegmeijer Cultural Heritage Agency
Poland Mrs Iwona Szmelter Polish Academy of Fine Arts
Spain Mrs Laura Hernandez FECYT
UK Mrs Laura Lugg AHRC
France Mr Roberto Casati Institut Jean Nicod/CNRS
11Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
12
WP5 – Task 5.1 Evaluation framework
15th March 2013: First meeting of the pool of experts i n Parisdedicated to the definition of the methodological f ramework(milestone MS7)
This methodological framework is not entirely settl ed yet — we ofcourse need the SRA to be validated to circulate it within the poolof experts — good progress have been made thank to this firstworkshop.
Many points have been discussed and could be summar ised here
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
WP5 – Task 5.1 Evaluation framework: Logical framework
Before proposing a list of indicators, we have disc ussed the “logicalframework”, that is, a way to rationally organize a nd classifyindicators.
This is usually performed by relating categories of indicators todifferent types of goals, according to a scheme as follows:
13Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
Challenges (Vision document/ SRA) Objectives / Research priorities (SRA)
Impact Outcome Output Actions Input
Implementation
WP5 – Task 5.1 Evaluation framework: Logical framework
Traditional typology of indicators : Input, Output, Outcome, Impact
This typology was enriched inJPND with also type A and typeB indicators:
14Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
Input
Output
Outcome
Impact
Type A (policy/funding)
Type B (scientific/societal)
Challenges (Vision document/ SRA) Objectives / Research priorities (SRA)
Impact Outcome Output Actions Input
Implementation
Proposal: Four categories, defined by the domains of application of the results
WP5 – Task 5.1 Evaluation framework: Logical framework
JPI Management& Actions
Science policy Scientific results Societal, economic,environmental
effects
The validation of the logical JPI CH framework is still pending
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
15
16
WP5 – Task 5.1 Evaluation framework: identification ofchallenges/objectives/ research priorities
This identification is key to the evaluation and mo nitoring process.
So far, we have relied on the Vision document that conveniently lists Challenges,Objectives and Research Questions.
However, the SRA draft shows significant differences with the Vision documentregarding priorities.
Ex.: Climate change is listed as first challenge in the Vision document and a mereresearch question in the SRA.
We certainly need to adapt the indicators to the updated priorities of JPICH.
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
WP5 – Task 5.1 Evaluation framework: Proposing indicators
A significant part of them (white background) is directly taken from JPND.
We have discussed and revised those more ‘polyvalent’ indicators(across JPIs) with experts.
The rest of indicators (colored boxes) depends directly on JPICH and itspriorities.Some have already been proposed but they are still under construction,following the SRA.
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
17
JPI Implementation Science policy Scientific results S ocietal effects
Participation of Member States inManagement Board meetings (a.1)
National research (funding)priorities adapted as a result ofJPICH and its research strategy(a.9)
Coordination of European prevention,emergency action and post disastermeasures against natural and man-made hazards (2.1)
The amount of JPICH commonresearch funding for cultural heritageas share of total EU research funding inthis domain (a.14)
Attitude towards JPICH goals andobjectives (a.2)
Alignment of national researchfunding programmes (a.10)
Develoment and application ofinnovative technologies for treatment,repair and maintenance of culturalheritage (3.1,3.2,3.4 & 3.5)
Total amount of European fundingavailable for cultural heritage (a.15)
Number of new joint transnational callsfor proposals (a.6)
Training and mobility ofresearchers across the partners'research insitutions (5)
Encouraging innovations,collaborations with European industry
Share of granted projects versusnumber of applications (a.7)
Sharing methods, protocols,infrastructures from local (regional) toglobal (European) levels
Number of new initiatives for non-project funded activities (a.8)
Raising public awareness of research
Number of collaborative researchprojects funded through JPICH jointcalls that address the various scientificpriorities (b.1)
Long term develoment of the CulturalResearch Area
Number of non-project funded activitiesthat address the various scientificpriorities (b.2)
Relation between cultural heritage andclimate change
Protection and security of culturalheritage
Transformational challenge
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
18
WP5 – Task 5.1 Evaluation framework: Future developments
- Validation of the logical framework, the identification of objectives andthe indicators by the pool of experts.
- In practice: work by e-mail with experts, second meeting, first weeks of July.
- D5.1 (Report on methodology) is scheduled for September.
Sylvie MAX-COLINARTVincent ISRAEL-JOSTFrench Ministry of Culture and Communication
JHEP Mid-Term Assessment MeetingJHEP Mid-Term Assessment Meeting
Brussels, 16 May 2013Brussels, 16 May 2013
19