IPA Funds Monitoring and Evaluation 12-16 December 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına

IPA Funds Monitoring and Evaluation 12-16 December 2011

1

Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım

Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey

This project is co-financed by the European Union and the Republic of Turkey

Evaluation: Evaluation: questions and questions and methodsmethods

Ankara, 1Ankara, 166 DecemberDecember 2011 2011Laura Trofin (PhD): TAT Non-Key Expert , Monitoring and [email protected]



We willWe will : :

1. Discuss about evaluation questions: how to design good evaluation questions

2. Go through some standard evaluation methodologies

3. Determine the methodologies we would select to carry out the evaluations determined yesterday



Evaluation questionsEvaluation questions

• Descriptive -(what happened?)

• Causal questions -(how and to what extent is that which occurred attributable to the programme?)

• Normative questions - are the results and impacts satisfactory in relation to targets, goals, etc?

• Predictive questions - will the measures create negative/positive effects?)

• Critical questions – e.g. how can equal opportunity policies be better accepted by SMEs?



What makes a good evaluationWhat makes a good evaluation question? question?

• Good evaluation questions include the following characteristics:

They are specific They are measurable They are answerable

They are realistic and reasonable They are appropriate to the local needs They can contribute to knowledge development



Where do evaluation questions Where do evaluation questions generally come from?generally come from?

• The first major source of evaluation questions includes the evaluation and programme stakeholders

• Most stakeholders will provide some questions if you ask them.

• You (evaluation expert) will need to work with these stakeholders to develop their questions into reasonable, testable evaluation questions.

• The second major source of evaluation questions comes from the programmes evaluator’s independent analysis of the programme/ToR.



Methodologies for evaluation…Methodologies for evaluation…

• According to “The Evaluation of Socio-Economic Development – The GUIDE”…

• There are 32 methodologies..!

• Strategic environmental assessment, stakeholder consultation, social surveys, regression analysis, priority evaluation, participatory methods, observational techniques, multi-criteria analysis, input-output analysis, cost-benefit analysis…



The main methodologies are…The main methodologies are…

1. Studying monitoring data2. Benchmarking3. Interviews4. Beneficiary/stakeholder/social surveys5. Case studies6. Focus groups7. Stakeholder consultation8. Gender Impact Assessment9. Cost-Benefit Analysis10. Econometrics



1. Studying monitoring data1. Studying monitoring data

• VITAL to every evaluation• You cannot properly evaluate without monitoring

data!• Financial data and physical data (performance

indicators)• Watch out for “variance” in analysing financial data• Do not read too much into performance indicator

data if programme at an early stage, particularly impact.



Challenges with studyingChallenges with studying monitoring datamonitoring data

• The data itself may be incorrect!• The data may be missing!• The performance indicators may be inappropriate!

(often an evaluator’s conclusion)• Monitoring templates are often too clumsy, too big,

unnecessary information• Extracting relevant data may be difficult



2. Benchmarking2. Benchmarking

1. Select the parts of the programme that will be subject to benchmarking

2. Identify a partner for the benchmarking3. Observe the process in the partner’s organisation4. Analyse the differences in performance.• Can be external (e.g. schools) or internal (e.g. faculties

within a university)• In the context of the Structural Funds, examples might

include comparing the operation of similar programmes in different member states or comparisons of EU funded programmes and domestic initiatives within the same country.



Challenges with benchmarkingChallenges with benchmarking

• For example, schools – recruitment practices, socio-economic background of pupils, better teachers go to better schools, etc.

• You have a to find a similar intervention in a very similar setting!



Beneficiary/Stakeholder/Social Beneficiary/Stakeholder/Social surveyssurveys

The evaluator can survey:•The final beneficiaries (e.g. participants on a training course)•Relevant stakeholders (e.g. managers of the training course)•The general population (e.g. members of a target group that did not take part in a the training course)



Evaluators use questionnaires Evaluators use questionnaires when…when…

• they wish to generate new, quantitative data on the programme

• they wish to reach large number of stakeholders• the population to be observed is large and

homogeneous (e.g. employers in the food processing sector, etc.);

• they have a precise and clear idea of what they want to observe.

• Any other???



Steps in carrying out a surveySteps in carrying out a survey

• Step 1. Designing the questionnaire • Step 2. Sampling• Step 3. Pre-test or pilot• Step 4. Administration of the questionnaire• Step 5. Codifying the data• Step 6. Analysing the data• Step 7. Reporting the data



Do’s and don’ts tips for good Do’s and don’ts tips for good surveys…surveys…

1. DO ask questions that can be quantified!!2. DO ask questions that are clear3. DO direct the person easily through the questionnaire (e.g. “If yes,

go to question 9, if no, go straight to question 10”)4. DON’T ask open-ended questions (e.g. “Tell us what you think

about…..”)5. DON’T ask a) too many questions on one page, b) questions that

are unnecessary, c) questions of an intrusive nature (e.g. “are you a Roma?”)

6. DON’T make the questionnaire too long7. DON’T ask questions of a “leading” nature (e.g. “Would you agree

that the programme was a success?”)



Challenges/problems with Challenges/problems with surveyssurveys

• Surveys• They are ONE way to gather data, but should never

be the exclusive method• Need to standardise answers leads to simplified

effects?• Telephone, face-to-face surveys very expensive• Postal surveys poor response rates• Need big enough samples!• If you have 10 categories of beneficiaries, do you

survey them all?!



Case StudiesCase Studies

How to carry out a case-study evaluation1. Select the cases2. Collect and process the information3. Report on the case4. Summarise (and extrapolate) the cases, where

possible



Case studies (contd.)Case studies (contd.)

• Of these the most fundamental is selecting the cases. Use three criteria1.Convenience/access2.Information you are trying to get from the cases 3.The extent to which you can extrapolate to provide wider insights beyond the particular case in question. •Issues to consider in selecting cases are:

1.What will count as a ‘case’? How many cases will be chosen?2.What units of analysis will be included within the case?3.How must the data be organised to allow meaningful comparisons to be made?4.What kind of generalisation is possible?



Challenges with case studiesChallenges with case studies

• Can you extrapolate?

• Expensive to conduct a lot of case studies

• Credibility problems with inferring causality



Focus groupsFocus groups

• A homogenous group of people who meet. The evaluator supplies the topics for and “chairs” the discussion.

• A variation is the workshop, a larger group with a more structured agenda

• When is it used?• analysing issues with very different opinions or complex

issues!• testing an innovative measure (ex ante evaluation)• identifying problems and needs and the improvements

required during the implementation of a programme.



Focus groups (continued)Focus groups (continued)

• Also good for validating quantitative data findings

• NB - Focus groups most suited to evaluations of programmes with a large “people” context (i.e. not good for transport or environmental infrastructure, good for HRD, or services-based programmes!)



Focus groups – how do Focus groups – how do you do it?you do it?

• Step 1 – Selection of participants• homogenous but diverse!• you may have to offer them something!

• Step 2 – Choice and training of facilitators• Don’t assume that you can do this yourself! Skills are

needed!• Don’t forget someone to take notes!

• Step 3 – Defining the interview topics• Don’t look to cover too much!• Start with the easy questions to promote discussion!



Focus groups – how do you Focus groups – how do you do it? (continued)do it? (continued)

• Step 4 – The discussion• “a combination of assertiveness and tact may be required,”

because…• Watch out for bullies!• Try to involve everyone!

• Step 5 – Analysis and reporting of results• compare the information given!• look for shared and divergent opinions!• code the data so as to organise the results in relation to

the objectives of the evaluation• include some direct quotes in your report!



Challenges with focusChallenges with focus groupsgroups

• “Hawthorne” effect

• Contrasting experiences of a minority of the participants

• Bullies!

• Straying outside the terms of the discussion



Stakeholder consultationStakeholder consultation

• Purpose – Ensuring that the preferences, interests and perspectives of different stakeholders are considered in the evaluation

• Why?• to allow stakeholders feel they have “ownership” of

the evaluation• to increase the likelihood that evaluation findings will

be used and acted upon• EU requirement!



Stakeholder consultationStakeholder consultation (continued)(continued)

• Used at 3 stages:1. Identifying the evaluation priorities

• identifying common ground and differences in what stakeholders’ want from the evaluation

• Arriving at a set of priorities, preferably through consensus.

2. Gathering data on programme performance3. Discussing possible conclusions



Stakeholder consultation – Stakeholder consultation – how do you do it?how do you do it?

• Step 1 – Identify the stakeholders• The Steering Committee (or commissioner) has to assist

with this!• Don’t assume that programme managers represent’ the

interests of the beneficiaries!• Step 2 – Structuring consultation with stakeholders

• How you achieve this will depend on:• how important is stakeholder consultation to the main

purpose of the evaluation?• the time and budget for the evaluation, • The number of stakeholders

• Key point - you can use other methodologies to consult with stakeholders! (e.g. focus groups, interviews, questionnaires, etc.)



How do you do it?How do you do it? (continued)(continued)

• Step 3 – Analysis of the data• This will not be a separate section of the report!

• Step 4 – Establishing mechanisms for ongoing consultation

• Give periodic feedback (through presentations to the Steering Committee, etc.)



Challenges with stakeholderChallenges with stakeholder consultationsconsultations

• The more stakeholders consulted, the more difficult the consultation process will be to manage.

• There is a limit to how many stakeholders can be consulted – which stakeholders to involve?

• Do the views of a participant equal the views of the programme manager?



Individual interviewsIndividual interviews – – some points…some points…

• Don’t assume people have all day! • Tell people how long you think it will last! • Explain their role in what you are trying to achieve!• Focus on a number of key issues (not more than 5 in 1

hour)• Let them read what you write!• Permission to use material from the interview must be

granted• Give them the overall results of the process at a later

date!



Strengths and weaknesses Strengths and weaknesses of interviews?of interviews?

• People can speak freely, no bullies!• Real insight into the issues• Allows for direct, anonymous quotes• Sensitive issues can be covered• BUT…if interviewing participants, can’t reach the

numbers that a survey can• Relies on the integrity of the interviewer• Interviewer shouldn’t be the interviewee!



Gender Impact AssessmentGender Impact Assessment

• estimates the different effects (positive, negative or neutral) of any policy or activity on gender

• allows changes to be proposed that will reduce any negative gender effects



GIA – how do you do it?GIA – how do you do it?

• Step 1 – Analysis of the present situation• Step 2 – Analysis of future trends• Step 3 – Determining priority• Step 4 – Assessing general impact of the Programme• Step 5 – Assessing specific direct impact of the

Programme• Step 6 – Assessing specific indirect impact of the

Programme• Step 7 – Defining further development• Step 8 – Defining a set of gender sensitive evaluation

indicators



Challenges with GIAChallenges with GIA

• If women benefit equally, but the overall number of people benefiting decreases, is that positive or negative??

• How do programmes that are targeted on men only or women only score under GIA?

• Is it okay to fund programmes that accept inequality?



Group exercise 4 Group exercise 4

For the types of evaluations developed under GE 3 define evaluation questions and choose the most appropriate methodological tools to carry out your evaluation

THINK ABOUT:• Purpose(s) and type of evaluation• Principles of good evaluation questions• Triangulation principle• Type and depth of information you want to collect• Strengths and challenges of each tool

Documents

IPA Funds Monitoring and Evaluation 12-16 December 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına