FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

1 Joint Action on Health Workforce Planning and

Forecasting

M.Vallimies-Patomaki, J.Lammintakanen

Brussels – April 12, 2013

Work Package 3

Finland

Evaluation of the Joint Action

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Agenda of the WP3 kick-off meeting

• Introduction of WP3 partners

• Scope and objectives of WP3

• Deliverables

• Preparation of the evaluation strategy

• Discussion topics

2

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Core team

Dr Marjukka Vallimies-Patomäki, WP leader, Ministry of Social Affairs and Health, Finland

Mr Jesmond Sharples, WP Co-leader, Ministry of Health, Malta

Prof. Juha Kinnunen, Senior Management Scientist, University of Eastern Finland, Finland

Prof. Johanna Lammintakanen, Management Scientist, University of Eastern Finland, Finland

3

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Partners• DG SANCO D2/EU, EAHC/EU

• Ministry of Social Affairs and Health/FI, Ministry of Health/MT, Ministry of Public Health/BE, Ministry of Health/PL, Ministry of Health/SK

• University of Eastern Finland/FI, Medical University of Varna/BG, NCPHA/BG, University of Bremen/DE, Semmelweis University/HU

• CfWI/UK

• EFN, EHMA

4

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Scope and objectives WP3 provides a systematic approach for the internal evaluation of the Joint Action

Aim is to verify if the JA is being implemented as planned and achieving the objectives:

- output evaluation is to verify compliance to achievement of the objectives

- outcome evaluation is to assess usefulness of outcomes, recommendations and sustainability beyond the Joint Action

- process evaluation is to ensure compliance to the rules and implementation

5

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Engagement of the member states and stakeholders

1.Mentoring group:

- WP leaders and representatives of stakeholders

- Discussing and giving feedback of the evaluation strategy

2.Expert reference groups:

- Representatives of member states and stakeholders

- Discussing and appraising the quality of the deliverables

3.Focus groups:

- Representatives of member states and stakeholders

- Appraising the impact and sustainability of the JA

6

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Deliverables

Evaluation strategy and evaluation tools, August 2013

Expert reference group report, November 2014

Intermediate evaluation report, January 2015

Expert reference group report, December 2015

Final evaluation report, March 2016

7

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Schedule for the preparation of the evaluation strategy• 1 st draft by April 30/Commission & WP1

•Basic contents and evaluation indicators•Comments by May 22

• Evaluation indicators and spesific objectives: consultation through teleconference/Commission, WP1 &WP leaders

• 2 nd draft by June 7 /Commission, WP1 & WP leaders•Comments by June 19(Dates changed by WP3)

• Stakeholder analysis, in collaboration with WP2 (> into an appendix)

• 3 rd version by June 30/Commission, WP1, WP leaders, partners, strategic stakeholders

•Comments by July 31

• Final version to WP1 by August to be submitted to EB

8

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Materials and methods

9

Approach

Material Method Evaluation Tools

Output evaluation

Deliverables / WP2,4,5,6 and 7

• Review process / Executive Board, MO• Expert reference group

Quality criteria for deliverables; semi-structured

Outcome evaluation

Changes in the HWF capacities / WP6Deliverables / WP2 and 7

• Content analysis / In collaboration with WP6• Delphi rounds/ Focus groups

Criteria for comparing the Cprofiles to updated Cprofiles

Interview themes

Process evaluation

Revisor´s reports / WP1Progress reports / MO and WPs

• Content analysis Mainly structured criteria for reports

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

1. Introduction

The aim of the evaluation (WP leader)• Some words on the whole EUHWForce project• Role, aims and tasks of evaluation in relation to JA

2. Evaluation Team

Members and division of work (WP leader)• Short introduction of members and areas of expertise• Responsibilities

3. Evaluation reports and timeline

Description of each evaluation deliverable• Interim evaluation report• Final evaluation report Data and methods used (J.Lammintakanen)• Documents (stage plans), formal reports• Mentoring group• Focus groups• Expert reference groupsTimeline (in relation to GA)Report objectives

4. Ev. criteria and tools

WP2, 4 - WP7, indicators (In collaboration with Commission, WP1, WP leaders, University of Bremen etc.)

5. Risk analysis

E.g. variety of laguages, cultural manners, unrealialistic milestones, conflicting interests

10

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Review of spesific objectivesSpesific

•Spesify the target group and the factors that need to be

changed

Measurable•Formulate objectives in a measurable format

Appropriate•Make sure objectives are acchievable, attainable, acceptable for the target group

Realistic• Ensure that you can realisticly acchieve the objectives

given the project resources

Timebound•Connect objectives to a timeline

11

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Key performance indicators• Ensure that WP leaders are involved in identifying KPIs for all

WPs of the JA

• KPI help the JA define and measure progress toward desired outcomes

• KPIs are quantifiable (measurable) measurements, agreed to beforehand, that reflect the critical success factors of the JA

School: graduation rates of the students

Social service organization: the number of clients assisted during the year

Source: www.management.about.com.

12

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Overview of the discussion topicsEvaluation tools:- Find examples and experiences

Spesific objectives and evaluation indicators : Review in collaboration with Commision, WP1 and WP leaders

Stakeholder analysis: In collaboration with WP2

Preparation of the evaluation strategy: In collaboration with Commission, WP1, WP leaders, University

of Bremen etc.

13

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Evaluation process: in co-operation with all WPs

14

Process evaluation (+ WP 1)Compliance to the rules

Output evaluation (WP2,4,5,6,7)Compliance to the content

Outcome evaluation (+WP2,6,7)Lessons learnt and recommendations

ACHIEVEMENT OF THE JA OBJECTIVES, IMPLEMENTATIONAND SUSTAINABILITY OF THE JA2016

Each WP’s objectives

Deliverable: Final report (> EB)

2013:SETTING OBJECTIVES FOR THE JA AND WPs”…to create a platform for collaboration and exchange between MSs to better prepere for the future of the health workforce”

Interim report > EB

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT

Thank you for your attention !15