Monica Brezzi – Aline Pennisi - Simona De Luca

Data analysis and indicator

systems: our work with the Italian

National Statistical Institute

Objectives of

regional

development written in

Clarify, focus, measure

public policies

monitor andevaluate

Socio-economic contextdescribe

Indicators = a set of quantitative variables useful for planning,

project selection and monitoring and evaluation activities

INDICATORS AND PUBLIC POLICY

Community Support

Framework (CSF) 2000-2006

National Economic documents

a) Support implementation of policies and checking their progress

b) Producing information useful for general knowledge on the programme and for evaluation purposes

…..guaranteeing transparency on public policies

USE OF INDICATORS AND MONITORING DATA

POLICY MAKERS,

MANAGING AND

PAYMENT AUTHORITIES

EVALUATORS,

INSTITUTIONAL

PARTNERS,

CITIZENS

Aggregate data on programme prioritiesProject level data

a) Indicators for the general objectives: macroeconomic

model and breakthrough variables;

b) Indicators of the potential of the areas concerned with

development policies: context indicators and regional

public accounts;

c) Indicators of implemented projects : outcome, output and

result indicators.

WHICH ARE THE INDICATORS AND MONITORING

DATA DEFINED IN THE ITALIAN CSF2000-06?

FAILURE OF PREVIOUS EXPERIENCE ON MONITORING

WHY DID ITALY CHOOSE THESE INDICATORS? (1)

Member State’s and Regions’ Member State’s and Regions’

needs not fully expressed and needs not fully expressed and

attitude of compliance;attitude of compliance;

Evaluation demand weak;Evaluation demand weak;

Difficulty to build a coherent Difficulty to build a coherent

framework of monitoring framework of monitoring

indicators;indicators;

Difficulty to measure indicators Difficulty to measure indicators

chosen;chosen;

IMPOSSIBILITY OF “FILLING” THE INDICATORS CHOSEN;IMPOSSIBILITY OF “FILLING” THE INDICATORS CHOSEN; INCAPABILITY OF MONITORING THE PROGRESS;INCAPABILITY OF MONITORING THE PROGRESS; LITTLE KNOWLEDGE ON TERRITORIAL EXPENDITURE (BY LITTLE KNOWLEDGE ON TERRITORIAL EXPENDITURE (BY

THEME, PRIORITY, LEVEL OF GOVERNMENT)THEME, PRIORITY, LEVEL OF GOVERNMENT)

EX ANTE EVALUATION OF THE CSF 2000-2006

WHY DID ITALY CHOOSE THESE INDICATORS? (2)

““Interpret” the EC Interpret” the EC

framework to create a framework to create a

national framework; national framework;

Provide definitions and Provide definitions and

methodological support;methodological support;

Build a quantified framework;Build a quantified framework;

Choose both bottom-up and top-down Choose both bottom-up and top-down

procedures (according to the nature of procedures (according to the nature of

indicators);indicators);

Great care on available indicatorsGreat care on available indicators

SYSTEMS OF INDICATORS (AND LINKS AMONG THEM);SYSTEMS OF INDICATORS (AND LINKS AMONG THEM); LEARNING PROCESS ON HOW TO DEFINE/ MEASURE LEARNING PROCESS ON HOW TO DEFINE/ MEASURE

INDICATORS AND MONITOR RESULTS;INDICATORS AND MONITOR RESULTS; TO VERIFY ADDITIONALITYTO VERIFY ADDITIONALITY

BREAKTHROUGH VARIABLES AND CONTEXT INDICATORS

During PSM and ex-ante evaluation of CSF – written in CSF and

ROP;

Central & Local Evaluation Units not yet established or not fully

active;

REGIONAL PUBLIC ACCOUNTS

During PSM and ex-ante evaluation of CSF

Central Evaluation Unit and 21 Regional Units set up for this

purpose;

OUTCOME, RESULTS & IMPACTS INDICATORS

After CSF and OP;

Central & Local Evaluation Units established and in most cases

fully active.

WHEN DID ITALY IDENTIFY ITS INDICATORS?

Indicators with the necessary characteristics were not always readily available

Partnership with official statistics producers (co-decision)

• to search existing available information, collect and update information

• To define adequate indicators in specific sectors (or territorial level) not previously available

Involving stakeholders and interested users (bottom-up)

• to build information at the appropriate level for monitoring and evaluation

• to extract local information otherwise not available

• To make regional administrations aware of using secondary information for policy planning and evaluation

Formal agreement between DPS and ISTAT to produce

statistical information to support regional

development policies (around 7 mln euro)

HOW DID ITALY GET THESE INDICATORS?

Typical difficulties in the availability of territorial data:

• insufficient timeliness

• the administrative subdivisions (Regions, Provinces, Municipalities) are usually preferred to functional breakdowns (such as employment basins, protected areas, other important areas for development policies, etc.)

• relevant sectors may not be covered…

DPS – ISTAT Agreement to promote a greater production of statistics within the official national statistical system:

• Reliability, high quality, transparency and public dissemination

• Shared and standard methodologies, to ensure data comparability over time and space

• Easily accessible

DPS-ISTAT AGREEMENT TO PROMOTE A WIDER PRODUCTION OF STATISTICS WITH TERRITORIAL BREAKDOWNS

PRODUCTS OF THE DPS-ISTAT AGREEMENT

Type of projects financed under the DPS-Istat Agreement:

Increased timeliness

Extension of surveys to obtain regional /

sub-regional breakdowns

Territorial and multi-thematic

databases

Pilot studies

• Anticipation of regional economic accounts

• Household consumption survey: regional data on poverty since 2002;

• Survey on ICT use in enterprises; regional indicators since 2003

• Survey on innovation in enterprises: regional indicators in 2006

• Water supply and consumption census: regional indicators in 2006

• CONTEXT INDICATORS DATABASE

• Database on infrastructure in italian provinces

• Database Health for All at province level

• INCIPIT Database on natural, cultural and quality-product resources at various levels of aggregation

• Extracting data from Business Register

• Regional environmental accounts

• Regional indicators on industrial production

•…..

THE CONTEXT INDICATOR REGIONAL DATABASE

• well-being;• desired characteristics of regional economic systems

(productivity, environment, labour market);• quality of services;• supply of infrastructure;• demography and education

A quantified description of disparities, gaps, potential and territorial competitiveness of Italian regions in terms of

Database from official statistics containing 100 indicators with yearly values (starting from 1995) for each region

(NUTS2 level) and macro area. The database is of public domain on the website and updated versions are released

three times a year

Some examples of context indicators

Sector Indicatorlast

available year

Centre-North

South ItalyCentre-North

South

Priority I "Natural Resources"Water Water distribution irregularities (as perceived by households) 2004 9,0 23,8 13,8 65,12 172,81Electricity Frequency of long stoppages of electric power service 2004 2,0 3,4 2,5 81,18 137,38Pollution and waste Urban recycled waste over total urban waste 2004 29,8 8,1 22,7 131,24 35,64Priority II "Cultural Resources"Cultural resources Per capita average expenses for theatre and concerts 2004 10,9 4,2 8,5 127,96 49,41Priority III "Human Resources"Labour Unemployment rate of youth (age 15-24) 2005 15,3 38,6 24,0Education Drop-out rate (first year of high school) 2003 10,3 13,4 11,7 87,84 114,94

ResearchExpenditure of the public and private enterprises for R&D (percentage over GDP)

2003 0,6 0,2 0,5 119,96 40,38

Priority IV "Local development"

Local development Labour productivity in the touristic sector (added value in the turistic sector over labour force in the sector)

2003 26,0 24,6 25,7 101,23 95,71

Local development Net birth rate of enterprises 2005 1,7 2,3 1,9 88,90 124,84

Priority V "Urban development"

Urban development Share of public transportation use to work places over total transportation

2004 23,5 22,1 23,1 101,94 95,59

Priority VI "Material and information network"Transportation Air pollution (CO2) due to transportation 2002 2,1 1,9 2,0 103,89 93,07Information society Percentage of families with internet connection 2004 37,2 28,7 34,5 108,01 83,29

Italy=100

What do we use context indicators for?

1. To pin down regional weaknesses and strengths.

2. To help focusing the objectives and to provide some direction for policy.

3. To increase the accountability for policy beneficiaries.

How can these three tasks be addressed?

Using many indicators to describe the same phenomenon;

Partnership with the Regional Administrations in defining the indicators;

Measure the policy effort through setting targets for a small subset of indicators (by regional policy-makers);

What we call a “soft use”

Year 2004. Source: Elaboration from NSO data

How to address policy focusing?

Percentage of families under the poverty line (regions ranked by quartiles)

Percentage of families with difficulties to access to health services

year 2002. Source elaborations from NSO data

Per head expenditure for investments in the health sector

year 2002. Source RPA database

Potential for policy focusing

Kg per capita of urban waste produced

Potential for policy focusing

Source, elaborations from APAT

Multidimensional approach

400

420

440

460

480

500

520

540

560

580

600

1996 1997 1998 1999 2000 2001 2002 2003 2004

Italy

South

.

Centre-North

Total expenditure per waste produced (euro per tons)

150

170

190

210

230

250

270

1996 1997 1998 1999 2000 2001 2002 2003

Centre-North

South

Italy

Source, elaborations from RPA database

How does the South perform with respect to

Italy? What can policy do to fill the gap?

Families perceiving irregularities in water distribution (as % of total families): Italy and South; Southern regions with target values

Are there significant differences among regions?

What can regional policy makers do to reduce the gaps?

0

5

10

15

20

25

30

35

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Italy

South

0

5

10

15

20

25

30

35

40

45

Campania Puglia Basilicata Calabria Sicilia Sardegna

1999 2004 target al 2008

Ex-ante evaluation and definition of targets

0

5

10

15

20

25

30

35

40

45

Pie

mo

nte

Va

lle D

'Ao

sta

Lo

mb

ard

ia

Tre

ntin

o -

Alto

Ad

ige

Ve

ne

to

Fri

uli

- V

en

ezi

a G

iulia

Lig

uri

a

Em

ilia

- R

om

ag

na

Tosc

an

a

Um

bri

a

Ma

rch

e

La

zio

Ab

ruzz

o

Mo

lise

Ca

mp

an

ia

Pu

glia

Ba

silic

ata

Ca

lab

ria

Sic

ilia

Sa

rde

gn

a

Italia

2003 national objective

2001 national objective

Comparing the monitored impact with the national objective …

“New” indicators on information society

0

10

20

30

40

50

60

Campania Puglia Basilicata Calabria Sicilia Sardegna

Italia = 49%

UE-15 = 57% (*)

Since 2001 national level surveys started in EU Member states coordinated by Eurostat. Now, since 2003, regional data is available too.

Enterprises that have their own website/homepage - %(source: Enterprise survey on ICT usage)

Context indicators were chosen when defining the CSF 2000-06;

At that time Regional Evaluation Units not yet established or fully

active;

Authorities of the ROP selected the indicators and chose targets

mainly to comply with EU requirements;

USING CONTEXT INDICATORS

Indicators and targets have not been truly monitored by Regions;

Only recently Regions have started using them for evaluation;

The impact of context indicators on the public arena is still

inadequate (although the OECD World Forum on Key Indicators…)

Economic and social partners have not extensively used the results

of context indicators to pin down regional performances.

Media do not convey these data.

…As a result

Not enough partnership in selecting indicators (with

administrations in charge of policies);

Not enough partnership (and accountability building) in choosing

the targets;

“Cultural” difficulty to use quantitative information and to relate to

measurable policy objectives;

Database and indicators were not easily accessible /

downloadable;

Timeliness of the updating of the set of indicators not always in

line with the official sector-publication on the data;

Analysis or evaluation based also on context indicators were not

available at the beginning or in the past.

MAIN CAUSES FOR A LIMITED USE OF CONTEXT INDICATORS FOR EVALUATION

Indicators available on the web site of DPS and ISTAT and tables

can be downloaded in Excel format

Meta information available on the same websites

Newsletter produced after each database update

Sharing experience in the use of indicators for evaluation Context Indicators project presented in international meetings

Context indicators have been used in the update evaluations

both from the Ministry of Economy and Regional Administrations

Analysis based also on context indicators are regularly included

in the Annual Report of the DPS and in official documents of

Ministry of Economy

SOME INITIATIVES TO SUPPORT DISSEMINATION AND USE OF CONTEXT INDICATORS

STRENGTHS Resources (human and financial) to produce the indicators,

beyond the structural funds and OPs ; Growing interest of “experts” (researchers, socio-economic

institutions, evaluators etc.)

WHAT CAN STILL BE IMPROVED AS REGARDS CONTEXT INDICATORS?

WEAKNESSES Policy makers, regions, managing authorities, social and

economic partners are not sufficiently involved in selecting indicators and setting targets;

Some indicators are not satisfactory:

a) too far from the actual policy actions (e.g. productivity of tourism industry, variation in the value of exports) ,

b) proxies of what we would like to measure (e.g., household perception of the quality of services in public transport, water, etc.)

c) breaks in time-series / changes of definitions to be handled

Objectives of

regional

development written in

Clarify, focus, measure

public policies

monitor andevaluate

Socio-economic contextdescribe

Indicators = a set of quantitative variables useful for planning,

project selection and monitoring and evaluation activities

INDICATORS AND PUBLIC POLICY

Community Support

Framework (CSF) 2000-2006

National Economic documents

National Strategic Reference

Framework 2007-2013

National Economic documents

Lisbon and Goteborg strategy

Review of the current database – some “cleaning up” :

Intensify the link between the selected indicators and “new” policy priorities

Include regional quantification of Lisbon strategy structural indicators, where relevant

Specific NSRF 2007-2013 work group discussing the possibility of including “hard” indicators on the provision and quality of a small number of “essential” public services (on social inclusion, waste management, water supply and education)

New desirable features for the indicators and for the database: Easy interpretation: policy makers must immediately understand

how to read an improvement in the indicator’s trend Comparable definitions across regions but also with other

countries Achieve a continuous on-line updating of the database

Use a thematic rather than a priority axis classification

WHAT CAN BE DONE FOR THE NEXT PROGRAMMING PERIOD 2007-2013?

Selection of indicators and targets:

Try to have policy makers suggest the indicators and targets (and use the technicians/experts only to guide the choice of measurement tools and check consistency)

Select a limited number of indicators with targets at the NSRF level

At the OP level agree on this small list; have each OP fix its own targets and specify the course-of-action / strategy expected to achieve such targets

Continue to refer to a more extensive list of context indicators for more general purposes

WHAT CAN BE DONE FOR THE NEXT PROGRAMMING PERIOD 2007-2013?

INCLUDING REGIONAL LEVEL DATA FOR LISBON STRUCTURAL INDICATORS (March 2006)

8

17

9

12

7

14

3 34

8

4 4

0

2

4

6

8

10

12

14

16

18

General economy Economic reform Employment Innovation andresearch

Social cohesion Environment

total number of structural indicators available in the context indicator database

FOR FURTHER INFORMATION

Website of Department for Development Policies (DPS) http://www.dps.tesoro.it/eng/english_corner.asp DPS Annual Reports, 2004 Annual Report on Actions in Under-Utilised Areas, available at http://www.dps.tesoro.it/eng/english_corner.asp Context Indicators database on the National Statistical Office’s websitehttp://www.istat.it/dati/db_siti/contesto/

Newsletter on context indicatorshttp://www.dps.tesoro.it/numeri_del_sud.asp

F. Barca, M. Brezzi, F. Terribile, F. Utili (2004), Measuring for decision making: soft and hard use of indicators in regional development policies, Materiali uval,n.2 available at http://www.dps.tesoro.it/materialiuval/indice_eng.asp