View
219
Download
0
Category
Tags:
Preview:
Citation preview
Monica Brezzi – Aline Pennisi - Simona De Luca
Data analysis and indicator
systems: our work with the Italian
National Statistical Institute
Objectives of
regional
development written in
Clarify, focus, measure
public policies
monitor andevaluate
Socio-economic contextdescribe
Indicators = a set of quantitative variables useful for planning,
project selection and monitoring and evaluation activities
INDICATORS AND PUBLIC POLICY
Community Support
Framework (CSF) 2000-2006
National Economic documents
a) Support implementation of policies and checking their progress
b) Producing information useful for general knowledge on the programme and for evaluation purposes
…..guaranteeing transparency on public policies
USE OF INDICATORS AND MONITORING DATA
POLICY MAKERS,
MANAGING AND
PAYMENT AUTHORITIES
EVALUATORS,
INSTITUTIONAL
PARTNERS,
CITIZENS
Aggregate data on programme prioritiesProject level data
a) Indicators for the general objectives: macroeconomic
model and breakthrough variables;
b) Indicators of the potential of the areas concerned with
development policies: context indicators and regional
public accounts;
c) Indicators of implemented projects : outcome, output and
result indicators.
WHICH ARE THE INDICATORS AND MONITORING
DATA DEFINED IN THE ITALIAN CSF2000-06?
FAILURE OF PREVIOUS EXPERIENCE ON MONITORING
WHY DID ITALY CHOOSE THESE INDICATORS? (1)
Member State’s and Regions’ Member State’s and Regions’
needs not fully expressed and needs not fully expressed and
attitude of compliance;attitude of compliance;
Evaluation demand weak;Evaluation demand weak;
Difficulty to build a coherent Difficulty to build a coherent
framework of monitoring framework of monitoring
indicators;indicators;
Difficulty to measure indicators Difficulty to measure indicators
chosen;chosen;
IMPOSSIBILITY OF “FILLING” THE INDICATORS CHOSEN;IMPOSSIBILITY OF “FILLING” THE INDICATORS CHOSEN; INCAPABILITY OF MONITORING THE PROGRESS;INCAPABILITY OF MONITORING THE PROGRESS; LITTLE KNOWLEDGE ON TERRITORIAL EXPENDITURE (BY LITTLE KNOWLEDGE ON TERRITORIAL EXPENDITURE (BY
THEME, PRIORITY, LEVEL OF GOVERNMENT)THEME, PRIORITY, LEVEL OF GOVERNMENT)
EX ANTE EVALUATION OF THE CSF 2000-2006
WHY DID ITALY CHOOSE THESE INDICATORS? (2)
““Interpret” the EC Interpret” the EC
framework to create a framework to create a
national framework; national framework;
Provide definitions and Provide definitions and
methodological support;methodological support;
Build a quantified framework;Build a quantified framework;
Choose both bottom-up and top-down Choose both bottom-up and top-down
procedures (according to the nature of procedures (according to the nature of
indicators);indicators);
Great care on available indicatorsGreat care on available indicators
SYSTEMS OF INDICATORS (AND LINKS AMONG THEM);SYSTEMS OF INDICATORS (AND LINKS AMONG THEM); LEARNING PROCESS ON HOW TO DEFINE/ MEASURE LEARNING PROCESS ON HOW TO DEFINE/ MEASURE
INDICATORS AND MONITOR RESULTS;INDICATORS AND MONITOR RESULTS; TO VERIFY ADDITIONALITYTO VERIFY ADDITIONALITY
BREAKTHROUGH VARIABLES AND CONTEXT INDICATORS
During PSM and ex-ante evaluation of CSF – written in CSF and
ROP;
Central & Local Evaluation Units not yet established or not fully
active;
REGIONAL PUBLIC ACCOUNTS
During PSM and ex-ante evaluation of CSF
Central Evaluation Unit and 21 Regional Units set up for this
purpose;
OUTCOME, RESULTS & IMPACTS INDICATORS
After CSF and OP;
Central & Local Evaluation Units established and in most cases
fully active.
WHEN DID ITALY IDENTIFY ITS INDICATORS?
Indicators with the necessary characteristics were not always readily available
Partnership with official statistics producers (co-decision)
• to search existing available information, collect and update information
• To define adequate indicators in specific sectors (or territorial level) not previously available
Involving stakeholders and interested users (bottom-up)
• to build information at the appropriate level for monitoring and evaluation
• to extract local information otherwise not available
• To make regional administrations aware of using secondary information for policy planning and evaluation
Formal agreement between DPS and ISTAT to produce
statistical information to support regional
development policies (around 7 mln euro)
HOW DID ITALY GET THESE INDICATORS?
Typical difficulties in the availability of territorial data:
• insufficient timeliness
• the administrative subdivisions (Regions, Provinces, Municipalities) are usually preferred to functional breakdowns (such as employment basins, protected areas, other important areas for development policies, etc.)
• relevant sectors may not be covered…
DPS – ISTAT Agreement to promote a greater production of statistics within the official national statistical system:
• Reliability, high quality, transparency and public dissemination
• Shared and standard methodologies, to ensure data comparability over time and space
• Easily accessible
DPS-ISTAT AGREEMENT TO PROMOTE A WIDER PRODUCTION OF STATISTICS WITH TERRITORIAL BREAKDOWNS
PRODUCTS OF THE DPS-ISTAT AGREEMENT
Type of projects financed under the DPS-Istat Agreement:
Increased timeliness
Extension of surveys to obtain regional /
sub-regional breakdowns
Territorial and multi-thematic
databases
Pilot studies
• Anticipation of regional economic accounts
• Household consumption survey: regional data on poverty since 2002;
• Survey on ICT use in enterprises; regional indicators since 2003
• Survey on innovation in enterprises: regional indicators in 2006
• Water supply and consumption census: regional indicators in 2006
• CONTEXT INDICATORS DATABASE
• Database on infrastructure in italian provinces
• Database Health for All at province level
• INCIPIT Database on natural, cultural and quality-product resources at various levels of aggregation
• Extracting data from Business Register
• Regional environmental accounts
• Regional indicators on industrial production
•…..
THE CONTEXT INDICATOR REGIONAL DATABASE
• well-being;• desired characteristics of regional economic systems
(productivity, environment, labour market);• quality of services;• supply of infrastructure;• demography and education
A quantified description of disparities, gaps, potential and territorial competitiveness of Italian regions in terms of
Database from official statistics containing 100 indicators with yearly values (starting from 1995) for each region
(NUTS2 level) and macro area. The database is of public domain on the website and updated versions are released
three times a year
Some examples of context indicators
Sector Indicatorlast
available year
Centre-North
South ItalyCentre-North
South
Priority I "Natural Resources"Water Water distribution irregularities (as perceived by households) 2004 9,0 23,8 13,8 65,12 172,81Electricity Frequency of long stoppages of electric power service 2004 2,0 3,4 2,5 81,18 137,38Pollution and waste Urban recycled waste over total urban waste 2004 29,8 8,1 22,7 131,24 35,64Priority II "Cultural Resources"Cultural resources Per capita average expenses for theatre and concerts 2004 10,9 4,2 8,5 127,96 49,41Priority III "Human Resources"Labour Unemployment rate of youth (age 15-24) 2005 15,3 38,6 24,0Education Drop-out rate (first year of high school) 2003 10,3 13,4 11,7 87,84 114,94
ResearchExpenditure of the public and private enterprises for R&D (percentage over GDP)
2003 0,6 0,2 0,5 119,96 40,38
Priority IV "Local development"
Local development Labour productivity in the touristic sector (added value in the turistic sector over labour force in the sector)
2003 26,0 24,6 25,7 101,23 95,71
Local development Net birth rate of enterprises 2005 1,7 2,3 1,9 88,90 124,84
Priority V "Urban development"
Urban development Share of public transportation use to work places over total transportation
2004 23,5 22,1 23,1 101,94 95,59
Priority VI "Material and information network"Transportation Air pollution (CO2) due to transportation 2002 2,1 1,9 2,0 103,89 93,07Information society Percentage of families with internet connection 2004 37,2 28,7 34,5 108,01 83,29
Italy=100
What do we use context indicators for?
1. To pin down regional weaknesses and strengths.
2. To help focusing the objectives and to provide some direction for policy.
3. To increase the accountability for policy beneficiaries.
How can these three tasks be addressed?
Using many indicators to describe the same phenomenon;
Partnership with the Regional Administrations in defining the indicators;
Measure the policy effort through setting targets for a small subset of indicators (by regional policy-makers);
What we call a “soft use”
Year 2004. Source: Elaboration from NSO data
How to address policy focusing?
Percentage of families under the poverty line (regions ranked by quartiles)
Percentage of families with difficulties to access to health services
year 2002. Source elaborations from NSO data
Per head expenditure for investments in the health sector
year 2002. Source RPA database
Potential for policy focusing
Kg per capita of urban waste produced
Potential for policy focusing
Source, elaborations from APAT
Multidimensional approach
400
420
440
460
480
500
520
540
560
580
600
1996 1997 1998 1999 2000 2001 2002 2003 2004
Italy
South
.
Centre-North
Total expenditure per waste produced (euro per tons)
150
170
190
210
230
250
270
1996 1997 1998 1999 2000 2001 2002 2003
Centre-North
South
Italy
Source, elaborations from RPA database
How does the South perform with respect to
Italy? What can policy do to fill the gap?
Families perceiving irregularities in water distribution (as % of total families): Italy and South; Southern regions with target values
Are there significant differences among regions?
What can regional policy makers do to reduce the gaps?
0
5
10
15
20
25
30
35
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Italy
South
0
5
10
15
20
25
30
35
40
45
Campania Puglia Basilicata Calabria Sicilia Sardegna
1999 2004 target al 2008
Ex-ante evaluation and definition of targets
0
5
10
15
20
25
30
35
40
45
Pie
mo
nte
Va
lle D
'Ao
sta
Lo
mb
ard
ia
Tre
ntin
o -
Alto
Ad
ige
Ve
ne
to
Fri
uli
- V
en
ezi
a G
iulia
Lig
uri
a
Em
ilia
- R
om
ag
na
Tosc
an
a
Um
bri
a
Ma
rch
e
La
zio
Ab
ruzz
o
Mo
lise
Ca
mp
an
ia
Pu
glia
Ba
silic
ata
Ca
lab
ria
Sic
ilia
Sa
rde
gn
a
Italia
2003 national objective
2001 national objective
Comparing the monitored impact with the national objective …
“New” indicators on information society
0
10
20
30
40
50
60
Campania Puglia Basilicata Calabria Sicilia Sardegna
Italia = 49%
UE-15 = 57% (*)
Since 2001 national level surveys started in EU Member states coordinated by Eurostat. Now, since 2003, regional data is available too.
Enterprises that have their own website/homepage - %(source: Enterprise survey on ICT usage)
Context indicators were chosen when defining the CSF 2000-06;
At that time Regional Evaluation Units not yet established or fully
active;
Authorities of the ROP selected the indicators and chose targets
mainly to comply with EU requirements;
USING CONTEXT INDICATORS
Indicators and targets have not been truly monitored by Regions;
Only recently Regions have started using them for evaluation;
The impact of context indicators on the public arena is still
inadequate (although the OECD World Forum on Key Indicators…)
Economic and social partners have not extensively used the results
of context indicators to pin down regional performances.
Media do not convey these data.
…As a result
Not enough partnership in selecting indicators (with
administrations in charge of policies);
Not enough partnership (and accountability building) in choosing
the targets;
“Cultural” difficulty to use quantitative information and to relate to
measurable policy objectives;
Database and indicators were not easily accessible /
downloadable;
Timeliness of the updating of the set of indicators not always in
line with the official sector-publication on the data;
Analysis or evaluation based also on context indicators were not
available at the beginning or in the past.
MAIN CAUSES FOR A LIMITED USE OF CONTEXT INDICATORS FOR EVALUATION
Indicators available on the web site of DPS and ISTAT and tables
can be downloaded in Excel format
Meta information available on the same websites
Newsletter produced after each database update
Sharing experience in the use of indicators for evaluation Context Indicators project presented in international meetings
Context indicators have been used in the update evaluations
both from the Ministry of Economy and Regional Administrations
Analysis based also on context indicators are regularly included
in the Annual Report of the DPS and in official documents of
Ministry of Economy
SOME INITIATIVES TO SUPPORT DISSEMINATION AND USE OF CONTEXT INDICATORS
STRENGTHS Resources (human and financial) to produce the indicators,
beyond the structural funds and OPs ; Growing interest of “experts” (researchers, socio-economic
institutions, evaluators etc.)
WHAT CAN STILL BE IMPROVED AS REGARDS CONTEXT INDICATORS?
WEAKNESSES Policy makers, regions, managing authorities, social and
economic partners are not sufficiently involved in selecting indicators and setting targets;
Some indicators are not satisfactory:
a) too far from the actual policy actions (e.g. productivity of tourism industry, variation in the value of exports) ,
b) proxies of what we would like to measure (e.g., household perception of the quality of services in public transport, water, etc.)
c) breaks in time-series / changes of definitions to be handled
Objectives of
regional
development written in
Clarify, focus, measure
public policies
monitor andevaluate
Socio-economic contextdescribe
Indicators = a set of quantitative variables useful for planning,
project selection and monitoring and evaluation activities
INDICATORS AND PUBLIC POLICY
Community Support
Framework (CSF) 2000-2006
National Economic documents
National Strategic Reference
Framework 2007-2013
National Economic documents
Lisbon and Goteborg strategy
Review of the current database – some “cleaning up” :
Intensify the link between the selected indicators and “new” policy priorities
Include regional quantification of Lisbon strategy structural indicators, where relevant
Specific NSRF 2007-2013 work group discussing the possibility of including “hard” indicators on the provision and quality of a small number of “essential” public services (on social inclusion, waste management, water supply and education)
New desirable features for the indicators and for the database: Easy interpretation: policy makers must immediately understand
how to read an improvement in the indicator’s trend Comparable definitions across regions but also with other
countries Achieve a continuous on-line updating of the database
Use a thematic rather than a priority axis classification
WHAT CAN BE DONE FOR THE NEXT PROGRAMMING PERIOD 2007-2013?
Selection of indicators and targets:
Try to have policy makers suggest the indicators and targets (and use the technicians/experts only to guide the choice of measurement tools and check consistency)
Select a limited number of indicators with targets at the NSRF level
At the OP level agree on this small list; have each OP fix its own targets and specify the course-of-action / strategy expected to achieve such targets
Continue to refer to a more extensive list of context indicators for more general purposes
WHAT CAN BE DONE FOR THE NEXT PROGRAMMING PERIOD 2007-2013?
INCLUDING REGIONAL LEVEL DATA FOR LISBON STRUCTURAL INDICATORS (March 2006)
8
17
9
12
7
14
3 34
8
4 4
0
2
4
6
8
10
12
14
16
18
General economy Economic reform Employment Innovation andresearch
Social cohesion Environment
total number of structural indicators available in the context indicator database
FOR FURTHER INFORMATION
Website of Department for Development Policies (DPS) http://www.dps.tesoro.it/eng/english_corner.asp DPS Annual Reports, 2004 Annual Report on Actions in Under-Utilised Areas, available at http://www.dps.tesoro.it/eng/english_corner.asp Context Indicators database on the National Statistical Office’s websitehttp://www.istat.it/dati/db_siti/contesto/
Newsletter on context indicatorshttp://www.dps.tesoro.it/numeri_del_sud.asp
F. Barca, M. Brezzi, F. Terribile, F. Utili (2004), Measuring for decision making: soft and hard use of indicators in regional development policies, Materiali uval,n.2 available at http://www.dps.tesoro.it/materialiuval/indice_eng.asp
Recommended