14 Oct 2010, TELECENTRE-EUROPE Summit TELECENTRE-EUROPE Summit: Third Sector contribution to e-Inclusion in Europe: Impact Measurement and Analysis framework

14 Oct 2010, TELECENTRE-EUROPE Summit

TELECENTRE-EUROPE Summit:

Third Sector contribution to e-Inclusion in Europe: Impact Measurement and Analysis framework

Budapest, 14th October 2010

Gabriel Rissola, JRC-IPTS

Maria Garrido, Technology & Social Change Group | University of Washington

Joint Research Centre (JRC)Institute for Prospective Technological Studies The European Commission’s Research-Based Policy Support Organisation


Institute for Prospective Technological Studies

Part of Joint Research Centre of the EC

IPTS: Research Institute supporting EU policy-making on socio-economic, scientific and/or technological issues


MSP for eInclusion


Last TE Summit in Istanbul: IA policy relevance and constraints

Demonstrate Impact is useful for:

Get a recognition of telecentres role in EU policy making:- Evidence is essential to get policies adopted (including funding programs)- Evidence helps to set priorities, make the right choices

Smarter action by combining social innovation and economic innovation, stimulated by technological innovation

…but few IA good practices were identified in the field

In Vienna Study on Inclusive Innovation for Growth and Cohesion, 1000 cases were analysed but only 52 had measured outputs and outcomes (mostly from UK and IRL), for example:

– UK Onlines Centres trained 76,000 people; active job seekers increased from 66% to 100%

– EOL UK digital literacy training helped 76 people to get a job, saving £670,000 in state subsidies

– FIT Ireland trained 6500 young unemployed people in ICT; 3500 found a job, saving $13 million.

– Also examples of improved welfare benefits distribution, savings through ICT-enabled home care.


Last TE Summit in Istanbul:Measuring Impact (outputs, outcomes)

eIN

CLU

SIO

N

Acc

ele

rati

ng

Enab

ling

De

ma

nd

Su

pp

ly

MEASURE CATEGORY DOMAINS BY MAIN

FOCUS OF INITIATIVESTYPICAL OUTPUT

Su

pp

lypublic access points availability # of regular customers

basic digital literacy # individuals completing courses

tax relief & incentives for affordability % of eligible beneficiariesusing the incentives

broadband % Δ broadband coverage

eAccessibility # of accessible public services

Community/ territorial initiatives Holistic not possible to indicate 1

learning through ICT

skill-building for employability # individuals completing courses

ageing-well at home services

healthcare services Holistic not possible to indicate 1

access to welfare entitlement Holistic not possible to indicate 1

De

ma

nd

Contribution to Regional growth

# individuals completing courses

# of homes covered

TYPICAL POTENTIAL OUTCOME

beneficiaries becoming regular Internet users

Various measures of community social capital

Various health outcomes

trainees finding a job

fewer days spent in institutional care/hospital

Various learning outcomes

Beneficiaries increase in disposable income

Cristiano Codagnone et al, EC study March 2009 (Vienna Study on Inclusive Innovation for Growth and Cohesion)


Last TE Summit in Istanbul:Telecentre eInclusion critical role and challenge

• Increasing political awareness on the critical role of Telecentres and NGOs in promoting e-Inclusion goals through different social and digital programs

• Wide range of target groups addressed - from immigrant communities, women, impoverished youth to prisoners and isolated people in rural areas to people with physical disabilities and the elderly – which goes even beyond Riga’s targets

• However, 3rd Sector organisations working in the eInclusion field don’t balance their efforts in critical areas (like IA)


Last TE Summit in Istanbul:Telecentre eInclusion critical role and challenge

• NGOs often struggle to systematically:– collect information from their beneficiaries without

overburdening staff capacity– articulate how and the extent to which their programs

impact the lives of the people they serve – use this evidence as a tool to gain more visibility and

recognition to their work especially among policy makers and donors

• Measuring Outputs (quantity of trainees, quantity of Internet users) and End user satisfaction surveys are important, but insufficient; need to go a step forward and find a way to systematically measure Outcomes


Practitioner perspectiveTelecottage impact measurement system

• Adapted UK’s IMProving Accountability, Clarity and Transparency (IMPACT) to Hungary

• Prepare impact report of the real social effect of their work.

• Primary data (outputs), with Secondary (outcomes).

• Created intended effects and possible indicators, on whether life is better or not.

• Most important effects of telecottages: – strengthened community spirit,

– improved human conditions,

– increase public trust

• Challenge: – to adapt intended effects and possible indicators to other local

contexts

– to measure them in an aggregated way at regional/national/EU level


Funder perspectiveMS UP impact measurement system

• Driven by need for systematic and credible evidence of how UP helps unemployed & others excluded to get essential skills to find a job.

• Most important thing is that evaluation scales globally. Is it improving lives? Are they moving along a trajectory?

• UP goes beyond self-reported quarterly reports, doing evaluations and case studies.


Funder perspectiveMS UP impact measurement system

• Lessons learnt: – track results on ongoing basis and make adjustments as needed

– metric that decided to avoid is how many people they put into jobs;

– commissioned research can be expensive;

– strongest grantees & partners are data driven and report back consistently, which builds a solid picture on impact over time

– Need for common indicators, plural of anecdote is not evidence.

• Opportunity: telecentres are multipliers


Building an IA framework


1. Consensus building on the objectives of measuring impact– to provide funders or investors with data on impact; – to provide a tool for organisations to manage their own choices

internally– to better understand long-term processes of social change and impact.

2. Targeted methodological framework for IA based on the state-of-the-art (there is no “perfect” social metric framework)

3. Targeted set of indicators– Relevant enough as to measure different e-Inclusion dimensions– Simple and easy to implement by organizations of different sizes– Appropriate enough as to provide consolidated results related to the e-Inclusion

goals of the DAE– Compatible with existing indicators but going deeper into the specificity of the

measured field.

Building an IA frameworkPossible steps


4. Need to facilitate social organizations self-analysis and self-interpretation of their own local impact. Indicators should be:– Available online (save costs and facilitates its distribution)– Translated and contextualized in each country (localisation)– Complemented by adequate guidelines for a proper data collection, self-analysis

of data and production of reports for different organisational scopesGuidelines for implementation and interpretation would surely be necessary

5. Need a software for the implementation of an IA system based on those indicators

• To facilitates the process of user data collection and impact analysis for NGOs

• To create with these data a repository that can be aggregated and analyzed globally

• To protect anonymity of NGO users and of the organizations themselves

• Easily implementable by the interested organizations in each local context

• Centrally maintained for the provision of constantly-updated consolidated information

Building an IA frameworkPossible steps


Measuring indicators is a complex endeavor due to the complexity of isolating the effects of one single variable (e.g. effect of e-Skills training in people’s lives).

This process is further complicated by the diversity of NGO programs, target groups, local factors, individual factors, etc.

Different types of impact can be measured (i.e. Access, frequency and purpose of ICT use, skills development, size and diversity of social networks, user perception of usefulness of ICT for improving their lives)

However, how to connect them with desired Outcomes remains an issue (e.g. “Qty of DL trainees that become regular Internet users”, “DL trainees that improve their employment status”)

Building an IA frameworkIndicators to be measured


FEEDBACK FROM THE AUDIENCE

Constraints- Voluntary organisations- Data protection laws- Local advocacy skills

Org needs/Selling points- Carefully select and agree the objectives of IA- Start simple, small- Highlight useful bits of data- Simple and quick implementation- Pay attention to incentives

14 Oct 2010, TELECENTRE-EUROPE Summit 16

Thank you

[email protected] http://is.jrc.ec.europa.eu/pages/EAP/eInclusion.html http://is.jrc.ec.europa.eu/pages/EAP/eLearning.html

[email protected] http://tascha.washington.edu/


Last TE Summit in Istanbul One example (female migrant users)

Perceived usefulness of ICT Appreciated features of TCs

Satisfaction with Social services

END USER SATISFACTION IS IMPORTANT, BUT INSUFFICIENT

+ Education and Employment services


Building an IA frameworkObjectives of the Metrics

Metrics are useful to show what works and what deserves to be grown (from The Open Book of Social Innovation):

• There are many metrics for judging whether innovations are working – at various stages of development.

• Metrics can play a decisive role in determining whether innovations are scaled up, or deserve to be.

• A recent survey found 150 different metrics in use in the non-profit sector. However, relatively few of these are actually used to make decisions.

• This field has failed to make progress due to the confusion and conflicts of interest between three different tasks performed by metrics: – to provide funders or investors with data on impact; – to provide a tool for organisations to manage their own choices internally– to better understand long-term processes of social change and impact.

1. NEED TO BUILD A CONSENSUS ON THE OBJECTIVES OF MEASURING IMPACT


There is no “perfect” social metric framework | Pros and Cons

Example 1: Sustainable Livelihoods Framework• Increasingly popular among development agencies

• It recognizes that people have a wide range of resources at their disposal and their choice for deploying those resources are critical to assess impact

• Very closely align to Amartya Sen’s capabilities approachLIMITATION: DIFFICULT TO IMPLEMENT AND MEASURE HOW PEOPLE’S CHOICES

AND MOTIVATION AFFECTS THE OUTCOME OF A PROJECT

Example 2: Cost-benefit analysis• Used primarily by public authorities and government agencies

• Its goal is to assess a particular project taking into account costs and benefits

• Benefits and costs are quantified in money termsLIMITATION: ASCRIBING MONETARY VALUES TO CERTAIN SOCIAL IMPACTS THAT

ARE VERY DIFFICULT TO MONETIZE

Building an IA frameworkSocial metrics system

2. NEED TO BUILD A TARGETED METHODOLOGICAL FRAMEWORK FOR I.A.

BASED ON THE STATE-OF-THE-ART


• Relevant enough as to measure some dimensions of the impact of e-Inclusion initiatives driven by NGOs in the communities they serve;

• Simple and easy to implement by organizations of different sizes which are active in the field of e-Inclusion at regional/national level in Europe (extendible to smaller NGOs and other actors in a second stage)

• Appropriate enough as to provide consolidated results related to the e-Inclusion goals of the Digital Agenda of Europe.

• Compatible with existing indicators but going deeper into the specificity of the measured field.

Building an IA frameworkSelection of indicators

3. NEED A TARGETED SET OF INDICATORS


Aggregated level

• E-Inclusion activities

• Other social services

• NGO mission

• Reach of the work (local, regional, national)

• Target groups

• Partnerships

Building an IA framework Some examples of indicators

Organisational level

• User demographics

• NGO services and activities where users participate

• Digital skills level

• Frequency of ICT use

• Types of ICT use (email, chat, job searching, homework, etc)

• Employability situation

• Social inclusion situation

• Users perception of value of ICT to improve their lives


5. NEED A SOFTWARE FOR THE IMPLEMENTATION OF AN I.A. SYSTEM BASED ON THOSE INDICATORS

Building an IA frameworkSoftware for processing data

• To facilitates the process of user data collection and program impact analysis for NGOs

• To create with these data a repository that can be aggregated and analyzed by country and at European level

• In doing so, to protect anonymity of NGO users and of the organizations themselves

• Easily implementable by the interested organizations in each local context

• Centrally maintained for the provision of constantly-updated consolidated information at European level


Stakeholders interested in the metrics to be produced

• Organisation’s staff (managers, fundraisers, etc)

• Local policy officers in charge of social services and community development

• Donors/funders

• Policy makers operating at different levels, including the EC

Users who will collect the data

• eFacilitators in contact with the end users

• Volunteers

• Staff managing the telecentres

• Telecentre network coordinators

Building an IA frameworkStakeholders and Users


• Review of indicators currently used by EUROSTATS, OECD, ITU, and academic articles

• Needs assessment to determine the kind of information that social organizations are currently gathering about the beneficiaries of their programs and additional data they are interested in collecting.

• Piloting the Impact Measurement System in different contexts

• Wide implementation of the system across Europe promoted by a multistakeholder partnership (MSP) or an umbrella organization who represents several stakeholders (like TE)

Building an IA frameworkOther needs

Documents

14 Oct 2010, TELECENTRE-EUROPE Summit TELECENTRE-EUROPE Summit: Third Sector contribution to e-Inclusion in Europe: Impact Measurement and Analysis framework