Evaluation of EU Structural Interventions Ex-ante Evaluation Mid-term / Updated Mid-term Evaluation...

Evaluation of EU Structural

Interventions

Ex-ante Evaluation Mid-term / Updated Mid-term Evaluation

Ex-post Evaluation Project Assessments

Königsteiner Strasse 116D-65812 Bad SodenGermany

Phone: +49-(0)6196-654168Fax: +49-(0)6196-654178

Internet: www.prac.de

P L A N U N G & F O R S C H U N G POLI CY RESEARCH & CONSULTANCY

Purpose

Explanation on the Purpose

The quality and adequacy of projects and fanding activities go hand in hand with the quality and adequacy of the programme in which the project is part of. A regional or sectoral development programme (as e.g. the EU Structural Funds) aims at fostering inter-regional convergence and/or stabilising important regional growth poles for the national economy. The purpose is not to simply subsidise institutions, business and individuals, Funding is the means and not the end.

Programme evaluation has to consider the respective quality, adequacy and implementation of a programme. Therefore the criteria applied in programme evaluation need also be translated into the quality assessment of the projects materialising the programme. Hence, as a truism, a project can only be as good as the programme and the programme can only be as good as the projects.

1. Ex-ante Evaluation

Structural Funds interventions are implemented in accordance to Operational Programmes (or Single Programming Documents) which must be (1) based on a reliable qualitative and quantitative analysis of the socio-economic situation of the region including its SWOTs and previous experiences, (2) the determination of a target-oriented development strategy, (3) the choice of appropriate measures to materialise the strategy, (4) suitable projects and (5) sound provisions governing financing, responsibilities, evaluation etc..

The assessment of the consistency of the above listed contents of an OP (or SPD respectively) is the core rationale of the Ex-ante evaluation. Further tasks to be carried out in the context of the Ex-ante Evaluation are (7) the quantification of the objectives and (8) the estimation of the overall socio-economic impact.

Objectives of the Ex-ante Evaluation

• Assessment of the appropriateness of the plan in addressing the regional development problems;

• Assessment of the consistency of analysis and concept;

• Definition of suitable indicators and quantification of objectives;

Objectives ...

• Analysis of the adequacy of the implementation and monitoring arrangements; and

• Impact assessment

Assessment of Appropriateness

• Lessons learned from past experiences;

• Difficulties and shortcomings experienced and to be tackled;

• Assessment of the validity of data used in previous programmes;

• Consideration of thematical evaluations;

• Consideration of experiences in other programmes; and

• Verification of the socio-economic analysis and the consistency of SWOTs

Consistency of SWOTs

• Strengths are static features to be supported

• Weaknesses are static features to be reduced

• Opportunities are dynamic features to be reaped

• Threats are dynamic features to be forestalled

Consistency of Analysis and Strategic Concept

• Are the strategic objectives well specified ?• Do the strategic objectives sufficiently respond to the

needs identified in the socio-economic analysis?• Are the proposed measures justified by the strategy?• Is the strategy sufficiently coordinated with other

related supra-national or national policies/programmes (nat. regional policy, NAP, CIs, Objective 3 etc.) ?

Suitable Indicators (examples)

• Basic indicators:

• Programme indicators:• Measure related

indicators:

• Performance indicators:

• Gross regional value added

• Supported investment

• No. of supported SMEs

• Disbursement/allocated funds

Quantification of Objectives

• Related to INPUTS-OUTPUTS-RESULTS-IMPACTS

• Determination of a realistic quantification of indicators

• Comparison with results achieved in former programme periods

• Proxy-indicators to monitor intangible effects

Appropriateness of the Implementation System

• Management and implementation responsibilities;

• Transparency;

• Efficiency and effectiveness of inter-institutional co-ordination;

• Competitiveness in project choice;

• Accountability in line with national and community regulations (competition policy, additionality, horizontal priorities, innovation, NAP etc.); and

• Partnership in programming and monitoring.

Impact assessment

• Micro-economic net effects (notably employment, productivity, regional GDP);

• Macro-economic net effects for large programmes (macro-econometric estimations and simulations on short-term demand, long-term supply, labour market and government sector)

2. Mid-term Evaluation and its Update

Mid-term Evaluation

• Update of the analysis of relevance;• Update of the analysis of consistency of priorities;• Analysis of the validity of the objectives;• Comparison of plan and achievements (vertical

and horizontal);• Assessment of the implementation performance in

terms of financial and administrative management;• Contribution to the decision on the allocation of

the Performance Reserve;

Mid-term Evaluation ...

• The responsibility of the mid-term evaluation is with the member state;

• The evaluation is to be carried out by an independent assessor;

• An update of the mid-term evaluation is to be elaborated one year before termination of the CSF/SPD; and

• The evaluations are accompanied and approved by the monitoring committees.

Analysis at Programme Level

• Were recommendations from former evaluations sufficiently considered?

• Have the socio-economic problems (justifying the intervention) changed?

• Is the SWOT analysis adequate?• Is the Programme still consistent and coherent?• Has the programme been implemented

professionally?

Analysis at Priority/Measure Level

• Financial implementation analysis

• Effectiveness analysis (plan vs. achievement)

• Efficiency analysis (unit cost comparisons)

• Impact analysis (GDP, net employment)

Cf. EU Commission/DG Regio: Working Paper 8

Update Mid-term Evaluation

The role of the updated mid-term evaluation is to confirm the implementation of recommendations put forward in the mid-term evaluation and to cover fields of analysis which could not be evaluated earlier.

Cf. EU Commission/DG Regio: Working Paper 9

3. Ex-post Evaluation

The Ex-post evaluation is carried out under the responsibility of the EU Commission. It is elaborated as a global evaluation for each of the regional(Objective 1 and 2) and sectoral (Objective 3) programmes covering all targeted member states.

The main purpose is to assess the quality, effectiveness and impacts of the programmes. It should clearly reveal the strengths and weaknesses of the planning process, the strategic approach and implementation. Lessons learned from this exercise should contribute to better policy formulation in current and future programming periods.

Scope of the Ex-post Evaluation

• Analysis of the relevance of strategy and the expenditure at EU, national and regional levels;

• Effectiveness analysis;• Efficiency and impact assessment;• Assessment of the delivery system;• Analysis of the Community value added; and• Lessons learned for the future and policy

recommendations.

Relevance of Strategy and Expenditure

• Are the strategies adopted by the regions sufficiently relevant?

• Are the strategies appropriately implemented in practice?

• Has there been any deviation of the interpretation of the strategy (new direction)?

• Is the philosophy of the strategy still in line with that of the prior evaluations?

• Were the funds accordingly allocated (dispersion vs. Concentration?

• Were the changes of the strategy relevant or arbitrary?

• Was the aid intensity relevant?

• The degree of coherence in the approaches across all CSFs/SPDs of Objective 1 or Objective 2 respectively?

Effectiveness Analysis

• Comparison of plan and realisation in terms of policy priorities (e.g. private sector support, innovation, business related infrastructure, HRD etc.);

• Assessment of the formerly expected and realised performance of the policy instruments (e.g. financial engineering)

Efficiency and impacts

• Value for money assessment;

• Assessment of the net-effects (employment, productivity, regional GDP etc.); and

• Macro-econometric model estimations for major programmes (notably Objective 1)

Delivery System

• Assessment of the institutional management and implementation capacity;

• Functioning of partnership;• Sufficiency of transparency;• Functioning of the monitoring system;• Functioning of the project selection procedures;• Appropriateness of the practical provisions to

ensure the legal underpinnings; and• Functioning of the quality control mechanisms

Community Value Added

• Economic and social convergence;

• Internal and external Cohesion; and

• Institutional improvements (e.g. better coordination)

`Lessons Learned` and Policy Recommendations

• Conceptual level and supra-national regulations (EU regional policy formulation);

• Justification of the financial contributions and/or the level of additionality respectively; and

• Implementation level (management, M&E partnership etc.)

Programming

Ex-ante Evaluation

Implementation

Mid-term Evaluation

Adjustments

Mid-term Update

Programme results

Ex-post Evaluation

THE EVALUATION CYCLE

Project assessments

Project Assessment within Structural Funds Programmes

• Rules and funding regulations laid down in the Programme Complement

• Supra-national, national and regional funding regulations

• Assessment of the scope and aim of the application

• Control/audit• Funding approval … Proof of fund utilisation

Criteria listed the Programme Complement

The PC includes the specific funding regulations and criteria for all measures of the programme. Any institution/individual applying for funding needs to comply to the provisions made in the PC. The PC defines

•The valid funding provisions

•The thematical/sectoral scope of funding

•The pre-conditions of funding (e.g. specific target groups)

•Eligible expenditure

•Ceiling of the subsidy

•Period of validity

•Geographical coverage

Funding regulations

• Regulations of the EC (e.g. Council Regulation 1260/1999)

• European competition law (e.g. Article 92)

• National competition law

• Regional funding provisions

Note: all regulations need to be co-ordinated

Project assessment grid (I)

• Is the project thematically suitable for funding under the respective measure?

• Does the project contribute to the quantified objectives of the measure?

• Is the project idea sufficiently original?• Does the project suggest synergy effects for

the programme?• Is the required co-financing adequate?

Project assessment grid (II)

• Is the applicant (or the group of applicants) part of the target group?

• Is the applicant (or the group of applicants) trustworthy

• …

Controlling/auditing

• Internal controlling of the managing authority notifying fraud and irregularities

• Monitoring of the project progress by the Managing Authority

• Controlling and issuing of expenditure notes by the Paying Authority

• 5% random audits carried out by the Independent Auditing Unit

Author: Rolf Bergs, 4 November 2004