Some Evaluation Methods

• Formal surveys• Theory-based evaluation• Rapid appraisal methods• Cost-benefit and cost-effectiveness

analysis• Participatory methods• Public expenditure tracking surveys• Impact evaluation

Formal Surveys

What are they?

Formal surveys can be used to collect standardized information from a carefully selected sample of people or households. Surveys often collect comparable information for a relatively large number of people in particular target groups.

What can we use them for?

• Providing baseline data against which the performance of the strategy, program, or project can be compared.

• Comparing different groups at a given point in time.• Comparing changes over time in the same group.• Comparing actual conditions with the targets established

in a program or project design.• Describing conditions in a particular community or group.• Providing a key input to a formal evaluation of the impact

of a program or project.

ADVANTAGES:

• Findings from the sample of people interviewed can be applied to the wider target group or the population as a whole.

• Quantitative estimates can be made for the size and distribution of impacts.

DISADVANTAGES:

• Results are often not available for a long period of time.

• The processing and analysis of data can be a major bottleneck for the larger surveys even where computers are available.

• Some surveys are expensive and time-consuming.

• Many kinds of information are difficult to obtain through formal interviews.

COST:

• Ranges from roughly $30–60 per household to $170 per household for LSMS (Living Standards Measurement Survey).

SKILLS REQUIRED:• Sound technical and analytical skills for sample and

questionnaire design, data analysis, and processing.

معلوماتشخصية • : العمرية الفئة

.a من - سنة 18أقل من سنة 35إلى 18بمن- إلى 36ج - 50سنة من أكثر د سنة 50سنة

الجنس:-.a - أنثى ب ذكر

-: السكن مكانقرية- مدينة- أ مخيم- ب ج

الصناعات -1 من أفضل األجنية الصناعات جميعالمحلية

- بشدة موافق غير موافق- أ غير ب - رأي ال موافق-ج د

بشدة - موافق هـ

موازنة- 2 الفلسطينية السلطة تخصص أن يجب: الفلسطينية الصناعات لدعم خاصة

بشدة- موافق غير موافق- أ غير ب - رأي ال موافق-ج د

بشدة - موافق هـ

Theory-based evaluationWhat is it?

Theory-based evaluation has similarities to the LogFrame approach but allows a much more in-depth understanding of the workings of a program or activity—the “program theory” or “program logic.”

In particular, it need not assume simple linear cause-and effect relationships.

.

Example

For example, the success of a government program to improve literacy levels by increasing the number of teachers might depend on a large number of factors. These include, among others, availability of classrooms and textbooks, the likely reactions of parents, school principals and schoolchildren, the skills and morale of teachers, the districts in which the extra teachers are to be located, the reliability of government funding, and so on.

YY: The success of a government program to improve literacy Levels

I:I: increasing the

numberof teachers

المدرسين عدد زيادة

A:A: availability of Classrooms

and textbooksوالكتب الصفوف توفر

R:R: reactions of

parentsاآلباء فعل ردات

s:s: School

principals & schoolchildren

المدراس مدراءوالطالب

T:T: skills and morale of teachers

وأخالقيات مهاراتالمدرسين

D:D: Districtالمنطقة

Y = Y = c1c1 I + I + c2c2 D + D + c3c3 T + T + c4c4 S + S + c5c5 A + A + c6c6 R R

By mapping out the determining factors judged important for success, and how they might interact, it can then be decided which steps should be monitored as the program develops, to see how well they are in fact borne out. This allows the critical success factors to be identified. And where the data show these factors have not been achieved, a reasonable conclusion is that the program is less likely to be successful in achieving its objectives

What can we use it for?

• Mapping design of complex activities.

• Improving planning and management.

ADVANTAGES

• Provides early feedback about what is or is not working, and why.

• Allows early correction of problems as soon as they emerge.

• Assists identification of unintended side-effects of the program.

• Helps in prioritizing which issues to investigate in greater depth, perhaps using more focused data collection or more sophisticated M&E techniques.

• Provides basis to assess the likely impacts of programs.

DISADVANTAGES:

• Can easily become overly complex if the scale of activities is large or if an exhaustive list of factors and assumptions is assembled.

• Stakeholders might disagree about which determining factors they judge important, which can be time-consuming to address.

COST:• Medium—depends on the depth of analysis and

especially the depth of data collection• undertaken to investigate the workings of the

program.

SKILLS REQUIRED:• Minimum 3–5 days training for facilitators.

TIME REQUIRED:• Can vary greatly, depending on the depth of the

analysis, the duration of the program or activity, and the depth of the M&E work undertaken.

Rapid Appraisal Methods

• Key informant interview

• Focus group discussion

• Community group interview

• Direct observation

• Mini-survey

Key informant interview

A series of open-ended questions posed to individuals selected for their knowledge and experience in a topic of interest. Interviews are qualitative, in-depth, and semi-structured. They rely on interview guides that list topics or questions.

Focus group discussion

A facilitated discussion among 8–12 carefully selected participants with similar backgrounds. Participants might be beneficiaries or program staff, for example. The facilitator uses a discussion guide. Note-takers record comments and observations.

Focus group session concentrates on:

• Gathering opinions, beliefs, and attitudes about issues of interest to your organization • Testing your assumptions • Encouraging discussion about a particular topic • Building excitement from spontaneous combination of participants' comments • Providing an opportunity to learn more about a topic or issue.

Steps for Focus groups

1- Define the purpose, i.e. objectives of the focus group : This has to be clear and specific. The more defined the objective the easier the rest of the process

2- Establish a timeline • A focus group cannot be developed overnight.

The planning has to start several weeks ahead of the actual session; experts say 6 to 8 weeks realistically. Make sure you have enough take time to identify the participants, develop and test the questions, locate a site, invite and follow up with participants, and gather the materials for the sessions.

3.      Identify the participants• Determine how many participants you need and how

many to invite. • Develop a list of key السمات attributes to seek in

participants based on the purpose of the focus group.• Using the list of attributes, brainstorm about possible

participants. • Secure names and contact information, finalize the list,

and send invitations.

1. Focus groups should consist of six to twelve participants.  Fewer than six participants tends to limit the conversation, because there is not enough diversity to spark energy and creativity.  A group larger than twelve gets to be unwieldy, and voices get lost. However, you should invite more, allowing for no-shows. 

4. Generate the questions• Because a focus group will last for little more than one or two hours,

you will only have time for four to seven questions. You may to include one or two introductory or warm-up questions and then get to the more serious questions that get at the heart of the purpose.

• To be effective, focus group questions should be open-ended and move from the general to the specific. E.g., after asking the question, “What do you like about the user interface?” you might ask, “If you could build a new user interface from scratch, what would you put in to make a better one?” or “What would make the user interface more appealing to your peers?” or even more specific, “Do you have any suggestions about what the personae (faces)—what they should look like or what they should do?”

• Once you have a list of questions, look at your purpose statement again.

• Keep questions that are really important and that qualify for your purpose. Eliminate as many questions as possible.

• Rewrite the questions with good editing.• Order the questions that will be comfortable for the participants, i.e.

moving from general to specific.

5.      Develop a script 

• Generating questions is a prelude تمهيد to developing a more detailed script for your focus group.

• Plan on a one - to two -hour time frame. A minimum of one hour is recommended because the process requires some time for opening and closing remarks as well as at least one or two questions. Be cautious not to exceed two hours.

• There are three parts to a focus group script:

1.      The opening is the time for the facilitator to welcome the group, introduce the purpose and context of the focus group, explain what a focus group is and how it will flow, and make the introductions.

2.      The question section is where you ask the questions that you designed and tested in Step 4.

3.      The closing section wraps up the focus group. This includes thanking the participants, giving them an opportunity and avenue for further input, telling them how the data will be used, and explaining when the larger process will be completed. 

6.      Select a facilitator• A focus group facilitator should be able to deal

tactfully with outspoken group members, keep the discussion on track, and make sure every participant is heard.

• The facilitator should be knowledgeable about the project. He or she can be a staff member, volunteer, or member of a committee or task force.

• Be wary of anything about the facilitator (or facilitators) that might make participants uncomfortable. For example, you may not want the organization's executive director to facilitate a staff focus group about a new performance appraisal system.

7.      Choose the location• You Need a setting which can accommodate the

participants and where they would feel comfortable expressing their opinions.

• When choosing a location, ask these questions:  • What message does the setting send? • Does the setting encourage conversation?

• How will the setting affect the information gathered? Will the setting bias the information offered?

• • Can it comfortably accommodate nine to fifteen people (six to twelve participants plus facilitators), where all can view each other?

• Is it easily accessible? (Consider access for people with disabilities, safety, transportation, parking, etc.)

Once decided, reserve the location if necessary.

Conduct The Focus Group:

• It’s time to actually conduct the session!• The materials you might need for the session are:

Notepads and pencils Computer with presentationFlip chart Focus group script List of participants Markers Masking tape Name tags Refreshments Watch or clock

• The facilitator should arrive before the participants, set out the refreshments, and arrange the room so all participants can view one another -- U-shaped seating or all at one table is best.

• As participants arrive, the facilitator should set the tone for a comfortable, enjoyable discussion by welcoming them just as would any gracious host.

• Introduce yourself and the co-facilitator, if used.• Explain the means to record the session. Make sure you

record the session!• Carry out the focus group as per the plan and script.• The facilitator should have some room for spontaneity,

i.e., asking spontaneous questions that arise from the discussion, probing deeper into a topic.

Attention to the following items will help ensure success:

1. Set the tone; participants should have fun and feel good about the session.

2. Make sure every participant is heard; draw out quieter group members.

3. Get full answers (not just "we need more money" but "we need more money to hire a receptionist to answer phones").

4. Monitor time closely; don’t exceed time limits.

5. Keep the discussion on track; try to answer all or most of the questions.

6. Head off exchanges of opinion about individual items.

Community group interview—a series of questions and facilitated discussion in a meeting open to all community members. The interviewer follows a carefully prepared questionnaire.

Direct observation—use of a detailed observation form to record what is seen and heard at a program site. The information may be about ongoing activities, processes, discussions, social interactions, and observable results.

Mini-survey—a structured questionnaire with a limited number of close ended questions that is administered to

50–75 people. Selection of respondents may be random or ‘purposive’ (interviewing stakeholders at locations such

as a clinic for a health care survey).