Mountains or Molehills?

Reducing the perceived burden of monitoring and

evaluating outcomes

Rebecca

Colclough

Dr Mark Llewellyn

Workshop Aim

To provide participants with useful

background information and a practical step

-by-step guide to planning your evaluation

approach and developing a simple and

effective outcome monitoring questionnaire

for your project

STEP 1: Digging Around

• Use simple Google searches to do some research into your area of interest

• What evaluation tools and techniques are similar organisations using?

• Some common systems are available including Outcome Stars and Quality of Life Indicators: check if these could be appropriate

• Take stock each time you start a new project

• Talk to other people doing similar work

STEP 2: Set your Outcomes

“Outcomes are the difference your project

can make to your community or

beneficiaries”

Big Lottery

SMART Outcomes

• Specific• Measurable• Achievable• Realistic• Time-bound

– Significant– Meaningful– Attainable– Relevant– Resourced– Results-based– Timely

Outcomes need to be SMART before you can begin to work out how to measure whether you have achieved them or not!

Example 1

OUTCOME• Increased confidence

SMART OUTCOME• Older people who participate in the programme

will evidence an increase in their confidence to speak up for themselves, which will be shown immediately following their participation, and will still be evident 6 months later

Example 2

OUTCOME• Increased use of recreational facilities

SMART OUTCOME• The number of people using the recreational facilities

will be evidenced to have increased by 10% in the first year of the project and by 20% by the end of the project

Or• The number of people who report using more than one

of the recreational facilities at the Centre will have increased from 15% to 20% in the first year of the project and will reach 25% by the end of the project

STEP 3: Evaluation Questions

• Turn your SMART outcomes into evaluation questions

• Remember, evaluation is COMPARATIVE by naturepast with presentproject 1 with project 2method a) with method b)before and after

GOOD evaluation questions

• Easy to understand• Written in plain language (no jargon or

abbreviations)• Not too long• Only ask one thing per question • Don’t ask ‘leading’ questions • Offer mutually exclusive and exhaustive

categories• Can be closed or open-ended• Can use recognised ‘scales’

Ordinal Scales (LIKERT)

Excellent Very good Fair Poor

Strongly agree

Agree Neither agree nor disagree

Disagree Strongly disagree

Always Often Sometimes Rarely Never

None Very mild Mild Moderate Severe

Completely satisfied

Very satisfied

Somewhat satisfied

Somewhat dissatisfied

Very dissatisfied

Completely dissatisfied

Developing Evaluation Questions

SMART Outcome:

Older people who participate in the

programme will evidence an increase in

their confidence to speak up for

themselves, which will be shown

immediately following their participation, and

will still be evident 6 months later

…continued

Evaluation Metric:

What % of beneficiaries report an

improvement in their confidence to speak up

for themselves?

…continuedEvaluation Question:

To make this evaluation question comparative you must ask it at least twice: BEFORE and AFTER your project intervention (in this case the question must also be asked 6 months later as well)

STEP 4: Develop your Evaluation Plan

Develop an evaluation plan which details all

of your intended project outcomes and

evaluation questions, and sets out how you

will collect the data to monitor your

achievement of these, alongside wider

service monitoring and evaluation

Evaluation Plan: Contents (i)• Data reporting deadlines

• Service monitoring and evaluation questions (RBA ‘How much are we doing’ and ‘How well are we doing it’?)

• SMART outcomes (RBA ‘Is anyone better off’?)

• Outcome evaluation questions

Evaluation Plan: Contents (ii)• Use of monitoring and evaluation data

To meet funding requirements?To influence social policy?To gain further project funding? To scope a particular issue or problem?

• Evaluation participants (who you will collect the data from)?

Beneficiaries?Volunteers?Stakeholders?

Evaluation Plan: Contents (iii)

• The type of data you will collect

• The data collection methods you will use

Evaluation Plan: Contents (iv)

AnalysisHow you will analyse your results

ReportingWhat kind of reports you will need to produce from yourresults

DisseminationHow your findings will be shared to have thegreatest impact

Unfortunately there won’t be time today to go into these areas inmore detail

Quantitative / Qualitative

Quantitative data is essentially

NUMBERS

Qualitative data is essentially

WORDS

Quantitative

• Learning little from many: can make generalisations

• Standardised responses

• Non-deliberative• Numerical• Based on sample

sizes

• Learning a lot from a few: hard to make generalisations

• Open responses; conversational and flexible

• Deliberative• Non-numerical; ideas;

beliefs; meaning• Based on experience

Qualitative

What kinds of data can be collected?

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”

Albert Einstein

Data Collection Methods

Different types of data require different

data collection methods to be used

Quantitative• Survey• Routine monitoring

data collection (stats)

Qualitative• Interview• Focus group• Digital story

Ensuring Quality Data

• Making sure you collect the right data at the right time and in a standardised way is essential to making sure the data you collect, analyse and share is good quality

• Different ‘methods’ use different ‘tools’ to do this, such as a questionnaire, a structured interview schedule or a focus group discussion guide

What is an Outcome TOOL?

An outcome tool is something designed toensure you can assess or describe change in aconsistent and standardised way. Thiswill produce data which is of a better quality

Good quality data makes it possible to drawtogether information on a number of beneficiariesand give an overview of the change achieved for awhole project or group of projects

Developing an Outcome Tool for your Project

Measuring outcomes for beneficiaries

Simple and Effective

• Make the TOOL as useful as possible

• Simple design which is easy to follow

• Gather data which answers your evaluation questions

• Use recognised SCALES where appropriate to ensure quality

• Think about when and how the TOOL is used

Service Quality

Take the opportunity to

assess the QUALITY of

your service along with

beneficiary outcomes

Free Text Space

• It is always useful to include free text space for beneficiaries to say more about something

• Whilst not everyone will complete free text space, those who do will often have something constructive or positive to say

• You can get great quotes for future use!

Capture project ‘champions’

• Make sure there is space for beneficiaries to fill in their contact details if they wish

• Take the opportunity to ask if they would mind being interviewed or contributing further to your project evaluation

• User voice is exceptionally powerful so don’t risk losing it!

Simple Structure

• Order the questions well according to any natural timelines

• Move from familiar to unfamiliar

• Don’t ask sensitive questions near the beginning

• Limit the number of questions

• End with easy questions

• Make the completion instructions clear

Data Collection Points

“Timing has a lot to do with the outcome of a

rain dance”

Cowboy Proverb

Data Collection Points

• Baseline• Progress• End of Intervention• Impact

You must ask exactly the same questions eachtime when trying to evaluate outcomes. Servicemonitoring and quality questions can be one-off

Completion Method

• Face to face• By post• By the client alone or with support• By a proxy (for people with dementia)• By a volunteer

Standardising your data collection method

helps ensure the data gathered is high quality

Useful Resourceswww.ces-vol.org.uk

http://www.biglotteryfund.org.uk/er_eval_explaining_the_difference.pdf Link to a document called ‘Explaining the difference your project makes’

http://www.evaluationtrust.org/evaluation/evaluate

http://www.worldbank.org/ieg/ipdet/presentation/M_05-Pr.pdf Power point module on writing good evaluation questions

http://www.keene.edu/crc/forms/designingsurveysthatcount.pdfSlides on good practice in writing evaluation questions

http://www.homelessoutcomes.org.uk/resources/1/Guide%20to%20Outcomes%20Tools%20Second%20Edition.pdfReally useful publication describing outcome tools (written for the homelessness sectorbut more widely applicable)