Upload
griffin-mason
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Evaluation within the context of the intervention
of the Structural Funds in Portugal
IFDRIFDR1 October 2008
General contents of the presentation
A. Fundamental issues for the evaluation of public interventions
B. Evolution of evaluation processes over the CSF
C. Evaluation in the NSRF
Evaluation
Fundamental issues for the evaluation of public interventions
Evaluation
Fundamental issues for the evaluation of public interventions
Relevance – Evaluation of the closeness of the strategy to reality
Efficiency – Evaluation of the way in which resources were transformed into output and results
Efficacy – Evaluation of the way in which the resources contributed towards realising the objectives
Utility – Evaluation of the benefits for recipients
Sustainability – Evaluation of the extent and durability of the alterations made
Aspects to take into consideration in the evaluation:
Fundamental issues for the evaluation of public interventions
Relevance
Evaluation of the closeness of the strategy to reality
Fundamental issues for the evaluation of public interventions
Efficiency
Evaluation of resources and its outputs
Fundamental issues for the evaluation of public interventions
Efficacy
Evaluation of the way in which resources were applied
Fundamental issues for the evaluation of public interventions
Utility
Evaluation of the benefits for recipients
Fundamental issues for the evaluation of public interventions
Sustainability
Evaluation of the extent and durability of the alterations made
Fundamental issues for the evaluation of public interventions
– Proportionality – Partnership – Independence
– Transparency
Principles of Evaluation
Principle of Proportionality Principles of Evaluation
Financial and administrative resources of evaluations
Proportional to the total amount of the expenses allocated to the OP
Principle of Partnership
Cooperation of the major agents throughout the planning process and implementation of evaluations
Principles of Evaluation
Principle of transparency Disclosure of the main results of the evaluations, as well as
their use as a resource for the qualification of public debate ACCOUNTABILITY
Evaluation exercises by entities (internal or external to the Public Administration) functionally independent of the
Management Authorities and of the entities with responsibilities in the NSRF and of the OP
Principle of independence
A. Fundamental issues for the evaluation of the public interventions
B. Development of Evaluation processes over the CSF
C. Evaluation in the NSRF
General contents of the presentation
Evaluation throughout the CSF
1. Evaluation in the CSF I (1989-1993)
2. Evaluation in the CSF II (1994-1999)
3. Evaluation in the CSF III (2000-2006)
CSF I Evaluation1989-1993
CONCEPTUAL MODEL
Institucional and Functional independence between Management & Evaluation
Different Structures methodologies
Ex-ante, ongoing e ex-post Evaluations In collaboration: Commission - MemberState Monitoring and evaluation functionally & institucionally independent from Management
Observatory of CSF I
Political Decision Makers
Independent Experts Ex-post Evaluation “IFDR” Coordination
Terms of
Reference
CSF II Evaluation
Main changes:
Increasing importance of the ex-ante appreciation and ex-post evaluation
Evaluations were carried out by each Operacional Intervention.
The Commission encouraged the ex-post evaluation of the CSF II.
1994-1999
CSF II Evaluation
Main conclusions:
High performance
Around 7.7% of the GFCF in the period 1994-2000 was directly induced by the CSF II
Around 77.000 jobs were created and maintained up to 1999 directly induced by the investments made in the CSF II
Ex-post evaluation 1994-1999
CSF II Evaluation
Changes into the Information System
Ex-post evaluation 1994-1999
creation of a single Information System Incorporating all of the Funds.
Improve the efficacy of the programmes
Improve monitoring functions
Separation between the activities of project control and monitoring
implementation of a system of management by objectives
Control Monitoring≠
Main Recommendations
CSF III EvaluationEvaluation Phases (2000-2006)Ex-ante Evaluation from the Member State responsibility carried out by independent evaluators
Mid Term Evaluation – December 2003 carried out by independent evaluators - responsibility of the MA allocation of the Performance Reserve
Mid Term Evaluation Update – December 2005
Ex-post Evaluation – 3 Years after programming period carried out by independent evaluators - responsibility of the EC
Diagnosis of the departure situation and analysis of the expected results.
Analysis of the first results of the interventions, taking into account the Ex-Ante Evaluation.
CSF III Evaluation
Ex-ante evaluation = basis for the preparation of the interventions
* being the responsibility of the Member StateChecksstrategy and objectivescoherence execution and monitoring rules planned
3 different moments of evaluation
Mid Term ReviewEx-ante Ex-Post
results of the evaluations
Important input
CSF III Evaluation
Enables:
the knowledge and analysis of the first results
the pertinence and the implementation of the objectives
the follow-up of financial allocations
the functioning of the monitoring and the execution of activities.
* being the responsibility of the Management Authorities in collaboration with the Commission
Mid Term Review
Each OP CSF
Ex-post Evaluation
CSF III Evaluation
Independent evaluators 3 years after end programming period Commission in collaboration with Member State
Intended to report on achievements and effects, the use of resources, the efficacy
and efficiency of the OP, The analysis of the causes of inefficacy, the impacts
Included results of the evaluation of macro-economic impacts
through the analysis of different reports cross-referencing them with the results of the
evaluation studies Performed by independent evaluators – until Dec 2003
Mid Term Review
CSF III Evaluation
Recommended
Changes in OP Mid Term Review
redefinition of strategies and priorities
redistribution of financial allocationsFundsOPPriorities
Performed by independent evaluations
Mid Term Review
CSF III Evaluation
Changes suggested
Intermediate reprogramming of 2004
Update of evaluation in 2005
until Dec 2003
Preparation of the subsequent interventions indicating priorities for the following
programming period
Mid Term Review
CSF III Evaluation
quality of the evaluation exercise
results of the Mid Term Review
4% of the total allocations planned
for each Member State
To reward the more efficient OP
attribution of the efficiency reserve
Reprogramming of 2004
Attribution of the Efficiency Reserve
CSF III Evaluation
Stage 1Stage 1
Selection of the OP which meet the criteria
Stage 2Stage 2
Determination of the amount of the efficiency reserve to be awarded
Efficiency Reserve
CSF III Evaluation
Criteria for all OP - Considering the fulfilment by the OP of good management practices
Man
agem
ent
crit
eria
Management System QualityQuality of the system of controlQuality of the selection criteriaQuality of the evaluation system
Fin
an
cia
l cri
teri
a Global financial execution Financial execution in LVT Leverage effect
Eff
ica
cy
c
rite
ria
Measures the effects of the Funds through output of result indicators
Attribution of the Efficiency Reserve
CSF III Evaluation
Common Management
Criteria
Very efficient: 5 of 6 criteria Efficient: 4 of 6 criteria Not Efficient: 3 criteria
Financial Criteria
Very efficient: at least 80% of output & result IndicatorsEfficient: at least 60% of output & result IndicatorsNot efficient: < 60%
Very efficient: reaches 90% of targets Efficient: 75 – 90% of targets Not Efficient: < 75%
Efficacy Criteria
Results of The Mid-Term Review
CSF III Evaluation
Main Conclusions – 2003
Relevance of the European Strategy For Employment
Reduction of unqualified employment
Reduction of early school leavers
Integration of education investment in companies
Modernization of Public Administration
Focus on Telecommunications and Energy (with direct impact of the Portuguese economy)
Results of The Mid-Term Review
CSF III Evaluation
Main Conclusions - 2005
Slow but progressive increase in the qualification of labour supply
Reduced expression of entrepreneurship in Portugal
High level of performance of OP in terms of approval, without perfect correspondence in terms of financial execution
Mid-Term ReviewCSF III Evaluation
Main Conclusions for 2007 – 2013
Integration of policies of innovation and
productivity
Development of territorial
competitiveness
Active employment policies bringing them closer to the regional
and local dimensions of structural
unemployment
Develop articulation of institutions and
organizations as to avoid losses in the
efficiency
Improve global coordination
Analysis of the evaluation exercises
In the whole period 2000 – 2006In the whole period 2000 – 2006
133 evaluation studies133 evaluation studies
39 = evaluations of an obligatory nature 39 = evaluations of an obligatory nature
One significant investment – but only 3% of the global One significant investment – but only 3% of the global financial allocations of technical assistance of the CSF III financial allocations of technical assistance of the CSF III
44% of the financial resources were intended for 44% of the financial resources were intended for evaluations of an obligatory natureevaluations of an obligatory nature
Evaluation studies of an obligatory nature
Analysis of the evaluation exercises
20
Mid-Term Review
N.º ofstudies
19 20
Total Cost l (1000 €)
4 086 2 293
o 15 Consultant Companies
o 15 Consultant Companies
Consultant Companies
o 1 Public Administration
o 1 Public Administration
Institutions
o 3 Research Universit. Centers
o 3 Research Universit. Centers
Follow up of Mid Term-Review
Analysis of the evaluation exercises
Evaluation studies of an obligatory nature
Strong points
Overall quality: credibility and utility Significant progress on analysis of implementation processes; global
rationality of the interventions; quantification of outputs; Institutional involvement and use of the evaluation as a support tool
for decision making
Weak points
Rigid calendars Exaggerated scope Weaknesses in quantification or estimate of impacts
Analysis of the evaluation exercises
Evaluation studies of an obligatory nature: Recommendations
Good practices of
Generalization of exercises of this nature in management practice
Flexibility and subsidiarity
Disclosure
A. Fundamental issues for the evaluation of the public interventions
B. The evolution of the evaluation processes throughout the CSF
– Evaluation in the CSF I (1989-1993) and CSF II(1994-1999)
– Evaluation in the CSF III (2000-2006)
C. Evaluation in the NSRF
General contents of the presentation
Evaluation in the NSRF
2007-2013
Evaluation in the NSRF
2007-2013
Strategic purpose:
Operational Programmes of three large Thematic Agendas
knowledge science, technology innovation promotion of high and sustained levels of economic and socio-cultural development and of territorial qualification
EDUCATION and QUALIFICATION
Through:
Equal opportunities.
Structural Funds and Cohesion Fund
Increased efficiency and quality of public institutions.
Promoted by
Short presentation of the NSRF
National strategic goals and priorities
NSRF OP
Renewed Lisbon Agenda
National Public
Policies
EU Strategic Guidelines
Growth and Employment
OUTPUTS
Short presentation of the NSRF
Thematic Operational Agendas
NSRF
Operational Programme
s
Human Potential
Factors ofCompetitivity
Short presentation of the NSRF
Enhancement of the Territory
Articulation of instruments of national public intervention ENDS
PNACE PNPOT
National StrategyFor Energy
National Employment Plan
National Programme for Inclusion
National Plan for Equality
Cultural Development Plan
National Strategic Plan for Tourism
Technological Plan
NSRF
Territorial Development Plans
Autonomous Regions Plans
Nat. Strat. Rural Develop. Plan / FEADER
National Nature and Biodiversity Conservation
Strategy
National Strategy for the Oceans
PEAASAR
Cities Policy
Nat. Forestry Plan
National Water Plan:Hydrographic Basins and
Efficient Use of Water
National Coastal Zone Integrated Strategy
National Waste Plan
PNAC
Short presentation of the NSRF
Characteristics
Before the start of the programming period
During the programming period
After the programming period
Ex anteEx ante
Ex postEx post
On goingOn goingTIM
ING
OB
JEC
TIV
E
To improve the quality, efficacy, efficiency and coherence of the intervention of the funds and of the strategy and execution of the OP.
Evaluation
Evaluation
Strategic
Operational
NA
TU
RE
RE
SP
ON
SI
BIL
ITY EC Ex-postEx-post
MS Ex-anteEx-ante
On-goingOn-going
Analysis of the evolution of an OP or group of OP in relation to the community and national priorities
To support the monitoringof an OP
Characteristics
Environmental Strategic
Evaluation of the effects of the Interventions on the
Environment
Institutional Framework
Evaluation – Responsibility
Institutional FrameworkOn-going Evaluation Responsibility
Strategic Monitoring
Financial and Operational Monitoring
Management, Evaluation,Internal Control
To ensure the efficient use of the Evaluations
To ensure that the evaluations are integrated and considered as a management tool during the implementation of the OP
Evaluation Plan
Covers the whole programming period
Nature of the evaluations
Description of the coordination and articulation mechanisms
An indicative list of Evaluation exercises
Foreseeable calendar
Evaluation PlanIncludes
Strategic evaluations
Operational evaluations
Level :
NSRF
OP/Fund
Parts:
I - Coordination
II – Evaluation activities and reports
From the Mid Term Review to the
on-going Evaluation
Regulation no. 1083 allows for a NEW CONCEPT of Evaluation:
More flexible
In line with the needs of the policy decision-making process and with the more efficient management of resources
More Flexible
By TopicBy OPBy PriorityBy Actions
Major projects
For all OP For one OP
At NSRF level
From the Mid Term Review to the
on-going Evaluation
Two closely related concepts, but with different objectives and functions
Monitoring analysis, monitoring and verification of results
Evaluation analysis and interpretation of the information obtained through the
monitoring, and other sources in order to find and explain the effects of the interventions.
From the Mid Term Review to the
on-going Evaluation
Monitoring Evaluation
Provides quality information and Provides quality information and respective analysesrespective analyses
Helps the decision-making process
From the Mid Term Review to the
on-going Evaluation
Regular monitoring Alerts for the need for Evaluation
Regular follow up of the Evaluation
Ensures implementation of reccommmendations
From the Mid Term Review to the
on-going Evaluation
Evaluation in the NSRF
Practical applications
Evaluation in the NSRF
Practical applications
Reflections of the Evaluation of the CSF III on the NSRF
CSF III NSRF 2007-2013
Insufficient concentration of financing options on the key areas corresponding to the major development problems of the country
Reduction in the no. of operational interventions More refined criteria in the prioritization of projects
Insufficient strategic alignment of operational instruments and of projects supported
On-going strategic monitoring mechanisms Consolidation of rationality centers Articulation between strategic objectives and financing models
Reflections of the Evaluation of the CSF III on the NSRF
CSF III NSRF 2007-2013
Difficulty in making the innovative potential of agents emerge
Dissemination of good practices, monitoring systems, Evaluation and benchmarking, inter-institutional coordination and innovative approaches
Insufficient attention to bolstering the institutional capacity of the Public Administration
Notion of Strategic State:Strategic planning Inter-sectorial coordination Monitoring and Evaluation
Insufficient focus on the quality of the effects, efficiency and sustainability of co-financed operations
Quality of the expense: Relevance of the investments and economic and financial sustainability Results to be achieved and the effects this will bring
SWOT Analysis of the Ex-ante Evaluation of the NSRF
What is it?
It defines a strategy through the relationships existing between the strong and weak points with the most important trends noted in the global external context, whether due to the economy, legal impositions, etc.
SWOT Analysis of the Ex-ante Evaluation of the NSRF
The term SWOT comes from the initials of the words:
Strengths
Weaknesses
Opportunities
Threats
What is it?
SWOT Analysis of the Ex-ante Evaluation of the NSRF
Weaknesses
Strengths
External context
Threats
Opportunities
Strategy
Internal context
SWOT Analysis of the Ex-ante Evaluation of the NSRFWhat is it?
Opportunities Threats
Limiting exogenous Limiting exogenous factorsfactors
(external analysis)
Strong PointsWeak Points
Existing applicationsExisting applicationsOn-going dynamicsOn-going dynamics(internal analysis)
For a good strategist there are no threats, only opportunities
SWOT Analysis of the Ex-ante Evaluation of the NSRF
How?
Strong Points
Weak Points
Opportunities
Threats
Build a table with the four elements: with the strong and weak points on one side and the opportunities and threats on the other
Identify the key elements which help to establish priorities and take strategic decisions
SWOT Analysis of the Ex-ante Evaluation of the NSRF
Opportunities Threats
Broadening of the process of integration of Iberian economies
Completion of the High Speed Lisbon – Madrid project
Financial difficulties in realising a fundamental project for the interna-tional connectivity of the Portuguese economy – high speed trains
Extending the market ado to companies which until now have been more focussed on the domestic market (potentially interesting for Portuguese SME in industry and services), within the space of proximity which is the Spanish economy
Growing affirmation of Spain as an European force, affecting the balanced development of Luso-Spanish trade and the maintenance of national decision-making centres in strategic sectors
Analysis of the opportunities and threats which confront the development process, as well as of the strengths and weaknesses of the Portuguese situation.
Strengths Weaknesses
Environment and natural heritage
Generalised coverage of the population in terms of water supply
Insufficient levels of service in basic areas with emphasis on the drainage and treatment of waste waters
Great diversity of natural heritage with high conservational value; 22% of the national territory is calssified with the status of nature protection and conservation
Insufficient protection and enhancement of natural heritage, frequently associated to the lack of knowledge for the management of protected species and habitats
SWOT Analysis of the Ex-ante Evaluation of the NSRF
SMART Criteria in the preparation of the Indicator File
The definition of the objectives may follow SMART criteria
Specific
Measurable
Attainable
Realistic
Timely
What is it?
Specific
They should clearly define what is intended
What?
Why?
How?
SMART Criteria in the preparation of the Indicator File
What is it?
Measurable
If it cannot be measured, it cannot be managed!!
The objectives should be quantifiable
SMART Criteria in the preparation of the Indicator File
What is it?
Attainable
The objectives should be achievable
SMART Criteria in the preparation of the Indicator File
What is it?
Realistic
Objectives should be defined taking in to account the effort required to achieve them
SMART Criteria in the preparation of the Indicator File
What is it?
Timely
Deadline - clarifies when it is intended for
SMART Criteria in the preparation of the Indicator File
What is it?
Indicator File
S
M
A
R
T
SMART Criteria in the preparation of the Indicator File
Identification
Planning and Execution
Characterization
Appreciation of the Quality of the EvaluationMid Term Evaluation CSF III
Excellent Good Sufficient Unacceptable
1. Information needs are satisfied
2.Pertinence of the scope of theEvaluation exercise3. Appropriate character of theMethodology
4.Reliability of the Data
5.Validity of the Analysis
6.Credibility of the Results
7.Impartiality of the Conclusions
8.Utility of the Recommendations
9.Clarity of the Report
Qualitative Appreciation Criteria
Thank you for your attention!
Thank you for your attention!