Impact Evaluationwith Vicky Pelka

What awaits you…

• Enactus UK first ever Impact Evaluation session!!!

• 1 hour…

• Reflect

• Ask Questions

• electronic Hand-outs available on request!

Who am I?

• Vicky Pelka

• Ex- Enactus Sussex President

• Research and Evaluation Coordinator at a London-based charity

• I evaluate the impact of our projects

Learning Outcome Statements

1. I can define impact evaluation. 2. I understand why impact evaluation is important

and useful. 3. I know at what points in a project cycle impact

evaluation should take place. 4. I know what kind of things can be measured in

order to do impact evaluation. 5. I can name some research methods used in

impact evaluation. 6. I can name some ways to communicate the

impact of a project.

Definitions

• Need

• Empowerment

• Quality of Life (QOL)

• Standard of Living (SOL)

• Impact

• Evaluation

(PEOPLE in )Need

Lack of something requisite, desirable, or useful. Needs range from basic survival needs, to cultural, intellectual, and social needs (Maslow’s pyramid of Needs).

Empowerment

The process of increasing the capacity of individuals (or groups) to apply new skills and gain self-confidence, so they achieve a stage of independence.

Quality of Life (QOL)

Intangible components of someone’s life

e.g. satisfaction with life, confidence, self-esteem, social status, emotional / psychological well-being, etc.

Standard of Living (SOL)

Tangible components of someone's’ life

e.g. employment, income, housing, access to clean drinking water or sanitation, healthcare, food provision, etc.

Impact

The positive and negative, short-term and long-term changes produced by a project.

Impact can be direct or indirect, intended or unintended.

Evaluation

The objective assessment of an on-going or completed project, its design, implementation and results.

The aim is to determine the relevance and fulfilment of objectives, effectiveness, impact and sustainability.

WHY is it important?It’s in the Enactus Criterion

Considering the relevant social, economic and environmental factors,

which Enactus team most effectively empowered people in need

by applying business and economic concepts and an entrepreneurial approach,

to improve their quality of life and standard of living.

WHY is it important? It’s in the in the Enactus Criterion

Considering the relevant social, economic and environmental factors,

which Enactus team most effectively empowered people in need

by applying business and economic concepts and an entrepreneurial approach,

to improve their quality of life and standard of living.

Why is it important?

• To know for yourself

• To tell others (university, students, committed and potential funders)

• To improve

• To let beneficiaries have a say!

What Impact Evaluation are YOU already doing in your team

and when?

(Take 2 minutes to tell your neighbour)

Project plans you have seen so far

Need

• Find people in need

Solution

• Apply business and economic concepts

• Take an entrepreneurial approach

• Empower

Impact

• Improve quality of life and standard of living

Bringing in Impact Evaluation

Need

• Find people in need

Solution• Apply business and

economic concepts

• Take an entrepreneurial approach

• Empower

Impact

• Improve quality of life and standard of living

Conduct Needs Assessment

Collect Baseline Data

Record Outputs

Monitor Progress

Adjust/ Improve

if necessary

Measure Outcomes:

Measure QOL and

SOL changes

Review effectiveness

and sustainability

What Impact Data is worth collecting?

• Outputs – what you have done

e.g. delivered 5 business workshops, carried out 30 hours of teaching

• Outcomes - what this means…

e.g. successfully established one business, doubled beneficiaries’ income, enabled a family to send 5 children to high school

What Impact Data is worth collecting?

• Quantitative – relates to number, statistics, quantities, specifics

e.g. “We increased the number of orphans attending school from 10 to 20”

• Qualitative – relates to quality, but also subjective opinion, judgements or perceptions

e.g. “The beneficiaries reported to be more confident in their writing skills“

What Impact Data is worth collecting?

• Get into groups of 3

• Brainstorm data that you could collect to prove:

Group1: A Need, e.g. unemployment

Group 2: changes in SOL, e.g. number of households with access to clean water

Group 3: changes in QOL, e.g. self-confidence in public speaking

Group 4: Empowerment, e.g. ability to produce quality furniture independently

What Impact Data is worth collecting?

Need Average income, unemployment rate, current state of business, child mortality rate, local poverty line, confidence, independence,

SOL Income, literacy rate, number of books in a library, attendance at events, access to medicine, electricity, school attendance rate , employment, transport,

QOL Self-confidence in own skills, appearance, mental well being, social inclusion, self-fulfilment, independence,

Empowerment

Occupational success, ability to provide for family, independence,

Collecting Impact Data

Where do I find this data?

• Government statistics, Census Data

• International organisations e.g. UN / WB / OECD / Transparency International

• Other charity publication

Can’t find it anywhere?

Collect it yourself using Research Methods…

What are Methods of Impact Evaluation?

1. Questionnaires

2. Interviews

3. Focus Groups

4. Conversations and Observations

• Find a research method that suits you!

• Tailor methods to your project and evaluation

• All methods have benefits and weaknesses

Method 1: Questionnaire

Method for qualitative and quantitative data collection

Useful for e.g.

knowledge testing,

gathering baseline data for needs assessment, collecting testimonials

Method 1: QuestionnaireThings to consider:• Language• Length of the questionnaire• Using open or closed questions• Using textboxes or tick boxes • An absolute no-go: leading questions! • Scales of rating (e.g. 1-5; strongly agree to

disagree) to give you QUANT data• Logic of the question order• Online (surveymonkey) or paper?

Method 1: Questionnaire

Things to do:

• Trial the questionnaire

• Explain purpose of the questionnaire

• Ensure confidentiality

Weaknesses:

• Misunderstanding your question

• Possible misreporting

Method 2: Interview

Method for qualitative and quantitative data collection

Useful for e.g.

individual needs assessments,

Project progress review,

in-depth analysis of impact on an individual

Method 2: Interview

Things to consider:

• Language and use of translator (issues of culture, gender, etc.)

• Choice between structures and semi-structured

• Length of the interview

• Location

Method 2: Interview

Things to consider:

• Incentive for interviewee to come

• Open or closed questions

• An absolute no-go: leading questions!

• Logic of the question order

• Video recording?

Method 2: Interview

Things to do:• Practice the interview• Explain purpose of the interview• Ensure confidentiality

Weaknesses: • Possible misreporting• Bad chemistry with interviewee• Only get perspective of one individual

Method 3: Focus Group

Method for qualitative data collection mainly

“A focus group is a planned discussion among a small group (4-12 persons) of stakeholders facilitated by a skilled moderator.

It is designed to obtain information about people’s preferences, values, beliefs, perceptions pertaining to a defined topic and why these are held by observing the structured discussion of an interactive group.” Participatory Method Toolkit, Slocum, 2003

Method 3: Focus Group

• Useful for e.g. needs assessment, project design, market research, end of project impact evaluation

Things to consider:

• Group size (4 -12 is recommended)

• Makeup of the group (gender, caste, social class, etc.)

• Location (indoor, outdoor, private, public?)

• Providing incentives for people to attend (food, money?)

Method 3: Focus Group

Things to do:

• Use a local facilitator

• Think of ice breakers and warm ups

• Plan a series of questions

• Appoint someone in charge of taking notes and observing. Record discussion points, arguments, behaviour.

Method 3: Focus Group

Weaknesses:

• Only reflects opinions of the people present

• Domination of discussion by powerful group members

• Unpredictable group dynamics

• Facilitator is key!

Method 4: Conversation and Observation

Method for qualitative data collection

Conversational reporting such as: “I send my daughter to school now.” “I really like coming here.”

“Used in marketing and the social sciences, observational research is a social research technique that involves the direct observation of phenomena in their natural setting.” Wikipedia

Method 4: Conversation and Observation

Things to consider:

• In what context did someone say something?

• What do you want to observe – a process or result? Interactions between people; individual behaviour; physical facts?

Things to do:

• Write down quotes ASAP

• Make an observation checklist

• Separate observation and thought/interpretation!

Method 4: Conversation and Observation

Weaknesses:

• Possible misreporting

• Observers’ bias

• Presence of an observer can influence behaviour

How do you communicate your impact?

• Consider the different ways of publicising / communicating your findings

• Consider your audience!

• Consider mix of Quant and Qual, Outputs and Outcomes

Figures and Stats

Example:

• 100 young people took part in a business skills course in 2009-2010.

• 60% of homeless individuals now live in safe housing conditions.

Graphs and charts

The written word….Describing what you do

• Do not use jargon

• Take the reader on a journey

(Need, Solution, Impact, Meaning)

• Explain Evaluation Process, methodology

The written word….Back it up with a Testimonial

"I feel I’m a changed person. I had someproblems at school and I was badly bullied andthreatened. On the Outward Bound course I waschallenged to face my fears and get over thembut when I finished it I felt really proud ofmyself.“ Ryan, 13 years

Pictures –Choose them well!

From the Oxfam and ActionAid website

Video e.g. Windows Movie Maker

• Of the community in which you work

• Of project activities

• Testimonials from beneficiaries

• Animation to explain a complex project concept

Go home and put all information into one awesome Action Plan:

1. Know what you want to evaluate (what data you want to collect)

2. Think when you want to record this data (before, after, mid-project)

3. Decide what method of data collection you want to use?

4. Who will carry out the evaluation?5. What resources do you need for it?6. Do it! 7. Communicate it!

Learning Outcome Statements

1. I can define impact evaluation. 2. I understand why impact evaluation is important

and useful. 3. I know at what points in a project cycle impact

evaluation should take place. 4. I know what kind of things can be measured in

order to do impact evaluation. 5. I can name some research methods used in

impact evaluation. 6. I can name some ways to communicate the

impact of a project.

Get in touch!

• Vicky Pelka

• Skype: vickyp101

• Email: pelka.v@gmail.com

• London-based meetings

• Emergency calls: 07 580 430 866

• Sign up/ email me to receive the hand-out