Upload
sailoa
View
223
Download
0
Embed Size (px)
Citation preview
©FASID
The Users Guide to L-E-A-D
Log-frame Evaluation Application Design Year 2004 version
- A New Approach for Policy and Program Evaluation -
FASID Foundation for Advanced Studies on International Development
March 2005
Log-frame Evaluation Application Design
L E A D
©FASID
Preface
A decade ago, the Foundation for Advanced Studies on International Development (FASID) developed PCM (Project Cycle Management) that has become the de-facto standard management tool of international development projects.
Nowadays, evaluation is needed not only for a project, but also for a program or a
policy under which multiple projects are conducted. In 2002, FASID established an inter-institutional study group, named “Study Group for Evaluation on Public Policy and Programs”, to develop a new evaluation method for the next generation to be applicable for policy and program levels. With intensive efforts of experts participating in the study group, a new method “LEAD” (Log-frame Evaluation Application Design) was developed. In April 2003, “Production Unit for LEAD Users Guide” (PULUG) was established by young members of the study group. Since then, the unit pursued the refinement of the new method to be more practical and useful, in order to enable the method to be more easy to use and thereby leading to significant improvements for the users. This Users Guide was written to extend the practical knowledge of “LEAD” to people who are interested in using this new evaluation method. The readers are assumed to be planners, practitioners and managing officers in international development. In addition, this booklet will provide a new insight for others working for various social programs, such as public officers in national or local government, and it may be useful even for officers in private companies who wish to comprehensively improve the evaluation of their own projects.
This Users Guide is intended to use plain and clear expressions in explanation for every reader to understand, whilst it may still be difficult or ambiguous for some readers. Therefore, any comments, complaints and suggestions are welcomed as “the evaluation for improvement”.
We hope that LEAD leads you to the improvement of your projects and programs.
International Development Research Institute, FASID
©FASID
Contents 1.INTRODUCTION ..............................................................................................................................1
0-0. FROM “EVALUATION” TO “IMPROVEMENT” ......................................................................................2 0-1. ABOUT LEAD ...................................................................................................................................3 0-2. CONTENTS OF LEAD .......................................................................................................................4 0-3. STRUCTURE OF THE USERS GUIDE .................................................................................................6 0-4. CASE.................................................................................................................................................6
2.LEAD IN PRACTICE........................................................................................................................8
1. INITIATION PHASE.................................................................................................................................9 1-1. Evaluation Scope Planning ........................................................................................................9
a) Outline ......................................................................................................................................................... 9 b) Output Sample: Scope of Evaluation ......................................................................................................... 10 c) Explanatory Notes:..................................................................................................................................... 11
1-1-1.Evaluation for Whom? ............................................................................................................. 11 1-1-2. Preparing “Scope of Evaluation” ........................................................................................... 11 1-1-3. Setting Evaluation Criteria ..................................................................................................... 11 1-1-4. Cases Which can not be Applied to Theory-Based Evaluation like LEAD...................... 12 1-1-5. Good Relationship with Stakeholders .................................................................................. 12 1-1-6. LEAD as a Tool of Capacity Building.................................................................................... 13
2-1. Hierarchy Tree Development .................................................................................................14 a) Outline ....................................................................................................................................................... 14 b) Output Sample: Hierarchy Tree ................................................................................................................. 15 c) Explanatory Notes:..................................................................................................................................... 16
2-1-1. Hierarchy Tree ......................................................................................................................... 16 2-1-2. Building a Hierarchy Tree ...................................................................................................... 16 2-1-3. Tips for Building a Hierarchy Tree ........................................................................................ 16 2-1-4. Coverage of a Hierarchy Tree ............................................................................................... 17 2-1-5. The Project without Activities and the Activity without the Project Name ....................... 17
2. PREPARATION PHASE ........................................................................................................................18 2-2. Logic Tree Development ........................................................................................................18
a) Outline ....................................................................................................................................................... 18 b) Output Sample: Logic Tree ........................................................................................................................ 19 c) Explanatory Notes...................................................................................................................................... 20
2-2-1. Logic Tree ................................................................................................................................ 20
©FASID
2-2-2. Setting Overall Goal ............................................................................................................... 20 2-2-3. “Means-Ends” Relationship ................................................................................................... 21 2-2-4. Tips for Setting Intermediate Objectives .............................................................................. 22 2-2-5. Theory-based Evaluation by Building a Logic Tree............................................................ 22
2-3. Projects Mapping ..…………………………………..…………………………………………...........23
a) Outline ....................................................................................................................................................... 24 2-4. Log-frame Formation ..............................................................................................................24
a) Outline ....................................................................................................................................................... 24 b) Output Sample : Log-Frame ...................................................................................................................... 25 c) Explanatory Notes:..................................................................................................................................... 26
2-4-1. Log-frame................................................................................................................................. 26 2-4-2.Special features of Log-frame for LEAD ............................................................................... 27 2-4-3. Log-frame as a Tool for Management .................................................................................. 27 2-4-4. Converting a Logic Tree into a Log-frame ........................................................................... 28 2-4-5. Setting Indicators and Means of Verification ....................................................................... 29 2-4-6. Setting Important Assumptions ............................................................................................. 31 2-4-7. Setting Inputs........................................................................................................................... 32
3. EVALUATION PHASE ...........................................................................................................................34 3-1. Evaluation Study Design .........................................................................................................34
a) Outline ....................................................................................................................................................... 34 b)Output Sample: Evaluation Grid................................................................................................................. 35 c) Explanatory Notes...................................................................................................................................... 37
3-1-1. Evaluation Purpose and Evaluation Criteria ....................................................................... 37 3-1-2. Program Level Evaluation...................................................................................................... 37 3-1-3. Designing Evaluation Study................................................................................................... 40 3-1-4. Setting the Methods of Data Collection ............................................................................... 41 3-1-5. Data Collection for Factor Analysis ...................................................................................... 42
3-2. Evaluation Study Implementation ...........................................................................................44 a) Outline ....................................................................................................................................................... 44 b) Output Sample: Evaluation Grid with Results ........................................................................................... 45 c) Explanatory Notes...................................................................................................................................... 47
3-2-1. Implementation of the Evaluation Study .............................................................................. 47 3-2-2. Question of Attribution ............................................................................................................ 47 3-2-3. Sorting the Results of the Evaluation Study ........................................................................ 49
3-3. Conclusions, Recommendations and Lessons Learned .........................................................50 a) Outline ....................................................................................................................................................... 50 b) Output Sample : Conclusions, Recommendations and Lessons Learned ................................................... 51 c) Explanatory Notes...................................................................................................................................... 52
©FASID
3-3-1. Drawing Conclusions.............................................................................................................. 52 3-3-2. Drawing Recommendations .................................................................................................. 52 3-3-3. Drawing Lessons Learned ..................................................................................................... 53 3-3-4. Preparing Presentation Materials ......................................................................................... 53
4. IMPROVEMENT PHASE .......................................................................................................................54 4-1. Feedback .................................................................................................................................54
a) Outline ....................................................................................................................................................... 54 b) Output Sample: Commitment to Improvement .......................................................................................... 55 c) Explanatory Notes...................................................................................................................................... 56
4-1-1. Commitment of Improvement ................................................................................................ 56 4-1-2. A Participatory Approach in Making the Commitment of Improvement ........................... 57
4-2. Improvement Planning .............................................................................................................58 a) Outline ....................................................................................................................................................... 58 b) Output Sample: Improved Log-frame ........................................................................................................ 59 c) Explanatory Notes...................................................................................................................................... 61
4-2-1. Cases Where an Improvement of the Log-frame is Not Necessary ................................ 61 4-2-2. Workshop for Improving the Log-frame ............................................................................... 61 4-2-3. Ex-ante Evaluation of the Improved Log-frame .................................................................. 62
4-3. Improvement Implementation .................................................................................................63 a) Outline ....................................................................................................................................................... 63 b) Output Sample: Improvement Cycle.......................................................................................................... 64 c) Explanatory Notes...................................................................................................................................... 65
4-3-1. Progress Monitoring ............................................................................................................... 65 4-3-2.Monitoring System ................................................................................................................... 66 4-3-3. A Periodic Evaluation for Further Improvement .................................................................. 67
5. EPILOGUE: EVERLASTING CYCLE OF IMPROVEMENT ........................................................................68
©FASID 1
1.Introduction
©FASID 2
0-0. From “Evaluation” to “Improvement”
By definition, evaluation is to ascertain the achievement of a project or a program (i.e. a group of projects or activities) from a particular valuating point of view. Project cycle has long been considered in the general concepts of project management as follows.
The project cycle begins with making a plan, and is followed by the implementation process to carry out the contents of the plan. In following, evaluation is conducted by comparing the results of the project with the original plan. The evaluation is connected to the improvement of plan making in the next cycle. The project cycle works best when a well-documented original plan is prepared at the planning stage. This condition is, however, difficult to be realized in reality.
Moreover, “policy evaluation” or “program evaluation”, which evaluates a group of projects with similar overall goals, is one of the most popular but challenging issues in evaluation studies. However, there is not any standardized method to undertake it.
In both cases, the critical point is how to make the proxy plan to be compared with the results. Thus, it calls for a new approach that combines the various projects (or activities) and integrates them into a body of a program that can be evaluated in a logical manner. In this case, the cycle is not commenced from the planning stage, but rather it starts from the evaluation process to improve the plan. This is called “the cycle of improvement.”
Planning
(Plan)
Implementation
(Do)
Evaluation
(See)
①
②③
Planning
(Plan)
Implementation
(Do)
Evaluation
(See)
④ ②
③
Integration (Combine)
①
Cycle of Improvement
Program BProgram CProgram A
©FASID 3
0-1. About LEAD
LEAD stands for Log-frame Evaluation Application Design. This method is based on the PCM (Project Cycle Management) evaluation method that is the de-facto standardized method for evaluating international development projects of Japan and other donors. LEAD employs its functions to be applicable for the program-level evaluation. This method is applicable for the following needs.
- To evaluate various types of projects implemented independently, but
having similar overall goals as a comprehensive program. - To evaluate multiple projects with similar overall goals conducted by
various agencies as a comprehensive program. - To link the results of evaluation with improvement of management.
©FASID 4
0-2. Contents of LEAD
LEAD consists of several steps in four phases as follows.
Phase Step Issues to be addressed
1. Initiation Phase
1-1. Evaluation Scope Planning
For what, and to what extent will it be evaluated?
2. Preparation Phase
2-1. Hierarchy Tree Development 2-2. Logic Tree Development 2-3. Projects Mapping 2-4. Log-frame Formulation
What is the structure and content of the target program?
3. Evaluation Phase
3-1. Evaluation Study Design 3-2. Evaluation Study Implementation 3-3. Conclusions,
Recommendations and Lessons Learned
What are the results of the evaluation?
4. Improvement Phase
4-1. Feedback 4-2. Improvement Planning 4-3. Improvement Implementation
What should be improved? How is it to be improved?
Each step produces output as follows.
©FASID 5
Road Map:LEAD Method, Process and Outputs
2-1 Hierarchy Tree
Scope of Evaluation
Evaluation Result
LEAD Hierarchy Tree
LEAD Logic Tree
Projects Map
LEAD Log-frame
Evaluation Grid
Conclusion, Recommendation and Lessons
Learned
Action Plan for
Implementation
Continued Improvement
Based on the Cycle of
Improvement
1-1 Scope of Evaluation
4-3 Implementation of
Improvement Plan with Log-frame
4-2 Planning of Improvement
with Log-frame
3-3 Conclusion, Recommendation
and Lessons Learned
3-2 Implementation of Evaluation
Research
3-1 Evaluation Design
2-4 Log-frame
2-3 Projects Mapping
2-2 Logic Tree
Outputs LEAD Steps
Improved Log-frame
4-1 Feedback Commitment of Improvement
©FASID 6
0-3. Structure of the Users Guide
The LEAD Users Guide explains each step in three parts, those of “Outline”, “Output Sample” and “Explanatory Notes”.
a) “Outline” clarifies the main parts: objectives, procedures and output, on one page. b) “Output Sample” shows an example of output from an imaginary case. c) “Explanatory Notes” covers important points in a more detailed explanation.
0-4. Case
This guide employs an imaginary case as follows, and to be used for “Output Samples”.
Case: Evaluation of Assistance of Country J for Improvement of
Second-Grade Education in the Republic of Amelia.
The Republic of Amelia is located on the west coast of the African continent. Country J has
been assisting the development of Amelia since its independence in the 1960s, when Amelia
had been considered as one of the least developed countries. Recently, Amelia has been witnessing the significant
growth of agriculture as the core sector of the national economy, with the exports of its agricultural products having
been increasing., This has now drawn Amelia away from the group of poorest countries.
Along with national economic growth, however, the economic disparity between the rich and the poor has been
aggravated, and the trend seems set to continue without any intervention. As one of the main factors for the
disparity, the educational disparity is considered the most serious and fundamental factor, according to the results
of some social studies that have been conducted. Although more than 90% of Amelians have graduated from the
elementary schools on average, the secondary school education is not well extended yet in Amelia since the
graduation rate against the total population is still less than 40% on average.
In order to improve this situation, Country J has been implementing three projects and a scholarship program in
the capital city of Amelia for a decade.
List of Project Name
(Department of MOFA of Country J)
・Rehabilitation Project of Secondary School Facilities.
(Dept. of Grant Aid)
・Training Project of Secondary School Teachers
(Dept of Technical Assistance)
・Development Project of Secondary Education Curriculum
©FASID 7
(Dept of Technical Assistance)
・Scholarship Program for Secondary Education
(Dept of International Communication)
These projects are conducted independently by various departments of the Ministry of Foreign Affairs of Country
J, with no comprehensive evaluation yet being carried out. In order to improve effectiveness and efficiency of
country J’s assistance for Amelia, MOFA calls for a comprehensive evaluation from a broader point of view.
In addition, Country M, one of the other main donors in this field, has been conducting a “Project for Capacity
Building of Education Administrators”. In the comprehensive evaluation, this project needs to be considered as one
of the influential factors.
©FASID 8
2.LEAD in Practice
©FASID 9
1. Initiation Phase
1-1. Evaluation Scope Planning
a) Outline
Objective: To comprehend the needs of users (or clients) of evaluation in order to clarify the purpose and the scope of evaluation
Procedures:
1. Clarify the clients for whom evaluation is conducted, to whom the results will be fed back. (Who will be the user of evaluation?)
2. Set the purpose of the evaluation based on the needs of the evaluation.
3. Determine the scope (area) of the evaluation in order to deal with the needs.
4. Consider the main stakeholders, such as evaluators, clients, supporters, etc.
5. Consider the documents needed for the evaluation research.
6. Put the above into a “Scope of Evaluation” that will be shared with stakeholders.
Output:
Scope of Evaluation
©FASID 10
b) Output Sample: Scope of Evaluation Scope of Evaluation
Evaluation Purpose (What is the purpose of
evaluation?)
To evaluate assistance of Country J to Amelia’s secondary
education from a broader point of view, in order to redesign the
coordination of projects into an effective and efficient program.
Projects and Programs Concerned (What will be evaluated?)
・ Rehabilitation Project of Secondary School Facilities.
・ Training Project of Secondary School Teachers.
・ Development Project of Secondary Education Curriculum
・ Scholarship Program for Secondary Education
Evaluation Criteria (What viewpoints or criteria will be
focused in evaluation?)
・ Effectiveness (including impact)
・ Efficiency Etc.
Needs of Clients and Users (Who will be the direct
beneficiaries of evaluation?)
・ Management officers of Country J’s aid
(Needs: to improve the assistance program of secondary
education in Amelia.
Stakeholders/ Supporters (Information sources?)
・ Management officers of Country J’s aid ・ Project staffs of concerned projects ・ Relevant agencies of secondary education in Amelia ・ Ministry of Foreign Affairs of Amelia, etc.
Evaluation Team (Who conducts the evaluation?)
・ Ms. A (Leader) ・ Mr. B (Education expert) ・ Mr. C (Coordinator)
Schedule of Evaluation (How long is evaluation conducted
for?)
Period : June to December, year 20xx.
Progress Report: Monthly (to clients)
Final Report : December, year 20xx
Reporting and Feedback (How will the results be fed back?)
In addition to monthly reporting, the final report will be prepared
and submitted to stakeholders, including the clients.
Post-evaluation commitment (How will the evaluation be
utilized?)
By reviewing the evaluation results, Country J will draft a
redesigned assistance program for secondary education in
Amelia, which will be implemented after reaching a consensus
with relevant organizations in Amelia.
Budget No more than 30 thousand USD.
©FASID 11
c) Explanatory Notes: 1-1-1.Evaluation for Whom? The implementation of evaluation also requires the concept of result-based management.
Evaluation needs to achieve expected results, which should be clarified prior to the commencement of
the evaluation, by asking “what is the purpose of the evaluation?”, or “whose needs does this evaluation
expect to satisfy?”. Without clarification of such basic issues, the evaluation tends to be “evaluation
just for evaluation sake”, and which is pointless. Evaluation is conducted because of a context in which
there are stakeholders who need the evaluation to be undertaken for specific reasons. The evaluator
should comprehend the needs of the clients or the users of the evaluation results, in order to conduct
effective and efficient evaluation.
1-1-2. Preparing “Scope of Evaluation” Before beginning an evaluation, a Scope of Evaluation needs to be prepared. This is not
only for the evaluator who confirms the scope of works, but also for stakeholders (including
clients or users) in order to understand the evaluation’s expected outcomes and limitations. By
clarifying this in the printed document, the understanding of the evaluation can be accurate. When it
is necessary to change the contents of the evaluation because of, for instance, a lack of information
sources, the change can be properly done through the Scope of Evaluation, which works as a compass.
Sharing the Scope of Evaluation among stakeholders enables more support and collaboration for the
evaluation to be obtained.
It is important to note that the shown Scope of Evaluation is just one example template. The
evaluators should configure their own Scope of Evaluation by adding or deleting components according
to the needs of their specific evaluation. Any other topics that need to be evaluated should be clarified
in detail and confirmed with the stakeholders.
1-1-3. Setting Evaluation Criteria The evaluation criteria are particular points of view with which an evaluation is conducted. DAC
(Development Assistance Committee) under the OECD recommends five criteria for the evaluation of
international cooperation projects/ programs. Many international cooperation agencies employ all or
some of the five criteria for their projects and programs.
Efficiency.................................. Project has produced outputs with least inputs?
Effectiveness ............................ Project effectively achieved the project purpose?
Impact ...................................... Project accrued positive or negative impacts?
Relevance ................................. Project is still relevant to the needs of stakeholders?
Sustainability........................... Project’s outcomes can be sustained?
©FASID 12
There may be other criteria such as “urgency”, “equity as a public project”, “coherence with other
projects”, “validity and coverage of beneficiaries”, etc. The evaluation criteria can be different to meet
the purpose or the needs of evaluation.
Evaluation criteria is selected (even just tentatively) and clarified in the Scope of Evaluation, which
should be presented to the stakeholders, including the clients, before the start of the evaluation.
In reality, the greater the number of evaluation criteria, the more time and cost is expended.
Including all of the evaluation criteria is wasteful, not only in the matter of efficiency, but also in the
quality, since analysis with too many tasks tends to be flawed. On the other hand, selecting evaluation
criteria only by the consideration of time and cost is not appropriate in order to satisfy the needs of
evaluation. In practice, it is a matter of balancing the evaluation needs with the available resources.
1-1-4. Cases which can not be Applied to Theory-based Evaluation like LEAD There are some needs that can not be satisfied by LEAD. For example, the need to evaluate a model
project for poverty alleviation implemented in several places all over the world, or the need to evaluate
the several impacts which road construction would accrue, are not very applicable to LEAD. LEAD is
relying on the program theory (i.e. Logic Tree) that utilizes the logical relationship between “Means”
and “Ends”. If the “Means-Ends” relationship can not be developed with activities of target projects (or
programs), then evaluation by LEAD can not be proceeded. A model project of poverty alleviation that
is implemented at dispersed places can not be “combined” into a single logical diagram (a so called
“Logic Tree”), so it is impossible to apply a theory-based evaluation method like LEAD. With a
multiple impacts evaluation on road construction, applying LEAD is not recommended since it is not
necessary to form a complex logical model for evaluation.
Although it is true that advantages of LEAD are not fully mobilized in these cases, this Users
Guide covers various topics that may be very useful to all types of evaluation.
1-1-5. Good Relationship with Stakeholders In evaluation, establishment of a good relationship with stakeholders is a very important issue. It is
the project (or program) implementer who will receive and turn recommendations from evaluation into
actions. The recommendations will not be implemented when the project implementers have the
feeling of resistance towards the evaluator, much like the relationship between the criminal and the
judge or between the artist and the critique. Even in collecting the information, if the evaluator’s
attitude is over critical or judgmental, the evaluator will not be able to obtain the real information
from stakeholders, especially the implementers.
The collaboration of stakeholders, including the implementer, is indispensable for conducting LEAD,
which needs the formation of a logical diagram of target projects (or programs) at the preparation
©FASID 13
phase with these stakeholders. Thus, LEAD is a cooperative evaluation
method rather than a judgmental or criticizing one. In other words, LEAD is
“formative rather than summative.” The evaluator must fully understand
this characteristic of LEAD, and they must make the best effort possible for
forming a good relationship with all the stakeholders.
It is, however, a matter of balance. An evaluator that accepts all of the
demands and information from the stakeholders will lose the credibility in the end. The evaluator
must be independent and preserve the mind of objectivity and fairness.
1-1-6. LEAD as a Tool of Capacity Building In case that the evaluation is a program level evaluation concerned with several projects by multiple
agencies, it is necessary to clarify the coordinating agency who realises the recommendations from the
evaluation. If the coordinating agency is the local government, the evaluation team should contain
officers from the local government from the commencement of the evaluation. The participation of the
officers should be integrated into the evaluation, i.e. it is not just by the name of the team or the
attendance of periodical meetings. The evaluation should be designed in accordance with the
understanding of the participating officers. It is no one but the participating officers who actually
improve the projects with their coordinating capabilities. Coordination of multiple projects needs the
competent capacity of understanding and analysing of the projects. Thus, the participation in LEAD is
very effective capacity building for the coordinating officers.
Planning and implementing of LEAD are led actually by the evaluator (or the evaluation team), but
involvement of officers working for the implementing agency or a coordinating agency will enable the
recommendations of the LEAD evaluation to be more feasible and effective. By participating in LEAD
from the preparation phase, participating officers will learn the strengths and weaknesses of the
target projects (programs) at full length. Thus, the recommendations and lessons drawn from such a
participatory evaluation will be more concrete for realisation.
©FASID 14
2. Preparation Phase 2-1. Hierarchy Tree Development
a) Outline
Objective: To describe the whole picture of target projects (including activities) by stratifying them into a hierarchical tree.
Procedures:
1. List the name of all projects (or programs) to be evaluated.
2. List all of the main activities of the projects.
3. Arrange (or stratify) the projects into a hierarchical tree according to the way of the institutional arrangement or as expressed in reports (e.g. by implementing agencies or administrative departments).
4. Confirm the Hierarchy Tree with all stakeholders to form a consensus of understanding.
Output:
Hierarchy Tree
©FASID 15
b) Output Sample: Hierarchy Tree
J's assistance
Dept. of Grand Aid
Rehabilitation Project of Secondary School
Facilities
Dept. of Teachnical Assistance
Dept. of International Communication
Training Project of Secondary
School Teachers
Scholarship Program for Secondary Education
Development Project of Secondary
Education Curriculum
Construction of a building
Development of
Curriculum & Syllabus
Improvement of textbook program
Provision of scholarship
Development &
maintenance of database
Provision of Toilet
Procurement of teaching materials
Trainig on maintenance
and management
of facilities
・Omitted・
Organization of a
committee of textbook
development
Development of Teaching Materials
Provision of a water tank
Female education support
Public relations for female educaiton
Provision of free
textbooks
Training for Teachers
Implementation of a training
program
Training for teachers
Education for female education
Training female teachers
Developmeof manualon financimanagemen
・Omitted・
Development of a
guidebook on teaching materials
Development of a
guidebook on teaching methods
Planning of training
program for teachers
Curriculum Development
・Omitted・
Hierarchy Tree
Issue a news letter on female
education
Research on the
promotion of femnale
educaiton ・Omitted・
・Omitted・
Responsible Department
Projects
Activities
Agency
©FASID 16
c) Explanatory Notes: 2-1-1. Hierarchy Tree It is difficult to understand the whole picture of the projects to be evaluated when the numbers and
areas of the projects are many. In some cases, the project name exists but the activities are not clear
or not actually conducted. In other cases, the evaluation must cover some schemes like a scholarship
program that does not include proactive activities or regulations like legal punishment to prevent
crimes. The LEAD method needs to construct the logical “means-ends” relationship from these sorts
of things before proceeding to the phase of evaluation study. It is, however, very difficult to form the
logical relationship directly from these “raw” materials, and it is somewhat dangerous in the sense
that some important components may be lost when forming the rigorous logic at the beginning. Thus,
the preliminary stage is just to construct a diagram by following the existing institutional approach,
or the existing hierarchical order. The diagram is called “Hierarchy Tree ”. Building a Hierarchy Tree
is considered the preparation for the next stage of building a Logic Tree. For stakeholders, a
Hierarchy Tree would be more familiar than a Logic Tree and a Log-frame, therefore the Hierarchy
Tree would help their future understanding of the Logic Tree and Log-frame that are developed later.
2-1-2. Building a Hierarchy Tree It is recommended to build a Hierarchy Tree cooperatively with the main stakeholders based on
relevant documents and opinions of other stakeholders. It is begun with establishment of a database
into a table with a spread sheet software such as “Excel”. Database contains “Name of agency”,
“Name of department”, “Name of project”, “Main contents”, “Budget”, and so on. The data is put in
order of each column in a table, because it is easy to change it into a hierarchy tree. While a
table-style is handful as a database, the tree-style like “a hierarchy tree” is more powerful in
presentation. To draw the tree, it is recommended to use the drawing functions in a spreadsheet
software program, or in a drawing software program such as “Inspiration”. A more effective method
of building a Hierarchy Tree is undertaking a participatory hands-on workshop using paper cards
on which the name of the projects and activities are described. The Hierarchy Tree is then
constructed by arranging these cards on a large sheet of paper on a wall. Such a participatory
workshop is helpful in building a consensus among the stakeholders.
2-1-3. Tips for Building a Hierarchy Tree As indicated in the name, a Hierarchy Tree stratifies the projects to be
evaluated according to the way of the institutional arrangement. One of the
most common forms of arrangement is “by department” or “by type of
scheme” (e.g. technical cooperation or grant aid). The hierarchy (or
stratification) can be made into multiple layers. For example, the first layer
is “by country”, the second layer is “by organization”, the third layer is “by
An increase in
agricultural production in the Country
of C
Department A Department B
Program A Program B Program C Program D
©FASID 17
scheme”, the fourth layer is “by project name”, and the lowest layer is “by activities”. It is, however,
not recommended to build a complex diagram that has too many layers. The purpose is to build the
most recognizable diagram that can be understood easily by the main stakeholders. Thus,
participation of the main stakeholders in building a Hierarchy Tree is important.
2-1-4. Coverage of a Hierarchy Tree When there is any project or activity other than the target projects that influence the common
objectives, it is recommended to include them into a Hierarchy Tree. This is because such information
will be a useful reference when building a Logic Tree in the next stage, and when considering the
important assumptions in the Log-frame. In our case, the project conducted by Country M is
considered as one of the important external components, influencing the achievement of the outcome
of the target projects. Therefore, the Hierarchy Tree includes the Country M’s project within it.
2-1-5. The project without Activities and the Activity without the Project Name In developing a Hierarchy Tree, you may find a project that exists on paper but no actions are
taken at all, or you may find some activities that are not formed as a project. Moreover, there is an
institutional regulation, such as law with punishment, or a scheme like a scholarship program that
does not need activities, but that does influence the objectives. In making a Hierarchy Tree, such
things must be put in just as they are. For example, activities without any project name are put in
without a name. In this stage of building a Hierarchy Tree, it is not necessary to make a logical
arrangement. Instead, one should try to express the structure of subjects in a naturally and
institutionally accepted way. It is the preliminary stage, and so the Hierarchy Tree can be an
unorganized one as long as it reflects the reality.
Describe it as it is!
©FASID 18
2. Preparation Phase
2-2. Logic Tree Development
a) Outline
Objective: To clarify the logical structure of the target projects as a whole program by arranging their activities along the relationship of “Means-Ends”.
Procedures: 1. List all activities from the Hierarchy Tree and place them in
the bottom line.
2. Set an Overall Goal (a policy or program level goal) by referring to documents and opinions of stakeholders.
3. Consider Intermediate Objectives which bridge between activities and the Overall Goal in a logical sequence.
4. Put activities with the same (or very similar) Intermediate Objectives into groups.
5. Iterate the above processes to complete the Logic Tree in which “Activities – Intermediate Objectives” and “Intermediate Objectives – Overall Goal” can be expressed by the logical relationship of “Means-Ends”.
6. Confirm the Logic Tree with the consensus of stakeholders.
Output:
Logic Tree
©FASID 19
b) Output Sample: “Logic Tree”
Enrollment of middle school
education is improved in
Amelia
Running regular classes
Construction of building
Training on maintenance
J's Assistance
Improve a middle school
curriculum
Teacher's teaching methods improve
Students' morale toward
schooling increases
Equal educational opportunity for both boys and girls
Provision of a water tank
Provision of bathroom
Modification of curriculum and syllabus
Making a guidebook of teaching material
development
Organizaiton of Committee
for the improvement of textbooks
Training for teachers
Acceptance of trainees
Training for top managers
Provision of scholarship
Support for female
education
Procurement of teaching material
OverallGoal
Intermediate Objectives
Activities
Provision of free
textbooks
Training for female
education
Provision of free
textbooks・Omitted・・Omitted・
Logic Tree
Planning of a training program for teachers
Making a guidebook on teaching methods
Development of teaching materials
・Omitted・Omitted ・Omitted・
©FASID 20
c) Explanatory Notes 2-2-1. Logic Tree The Logic Tree expresses the logical structure of projects and activities to be incorporated into a
proxy program by reconstructing the logics of the components. Establishing a Logic Tree is necessary
to conduct a comprehensive evaluation. With the Logic Tree the following aspects can be evaluated.
- Holistic effects of various projects as a comprehensive program
- Relationships among projects and activities
- Single effects of projects, including the indirect effects
It is the evaluator who is responsible for developing a Logic Tree but the building process must be
participatory with stakeholders, as with the development of a Hierarchy Tree.
2-2-2. Setting Overall Goal Overall Goal is the common goal for which all related projects and activities are conducted. It is set
at the top of a Logic Tree, so that there is only one Overall Goal. If there is no Overall Goal identified in
advance, the evaluator sets a new Overall Goal. In setting this Goal , care should be taken when
deciding how to express it. It is NOT the summary of what the projects (or program) will do, but it
expresses a positive change or impact on the beneficiaries that the projects (or program) are intended
to accrue.
Bad and good examples of an Overall Goal in evaluating traffic-related projects are:
(Bad Example) × Traffic infrastructure is upgraded.
Overall Goal
(Good Example) ○ Traffic congestion is reduced.
In this case, the ’bad’ example is just describing what the projects do, and does not say what the
purpose of the traffic infrastructure development is for. Thus, it should be written in such a way as
that stated in the ‘good’ example, as this describes the outcome (or impact) of the development projects.
It can be said that the traffic infrastructure is not only for vehicle traffic, but also for pedestrians as
well. In this case, the expression of the Overall Goal could be “The city becomes a place where the
citizens feel able to be transported more easily”. This expression is a wider concept compared with the
original one. If an expression is too vague, some definitions may be added to the Overall Goal.
In practice, some clues for setting the Overall Goal can be found within the main goals of each
project, which is set above the direct project purpose. Referring to the needs and the purpose of
©FASID 21
evaluation, or meeting with stakeholders such as the program officers and representative beneficiaries,
will also give good clues. It is important to determine the Overall Goal by reaching a consensus among
stakeholders as explained in the previous steps.
2-2-3. “Means-Ends” Relationship The Logic Tree consists of components such as the Overall Goal, Intermediate Objectives and
Activities, among which the relationships are constructed in the logics of “Means-Ends”.
As stated below, an Activity “to teach mathematics to a student” is one of the Means for the End
(objective) that “the student can calculate”. For achieving this objective, there may be other means,
such as “to provide a textbook of mathematics”. Moreover, the statement that “the student can
calculate” can be one of the means for a higher objective such as “the student can promote to the next
grade”. Like this, a Logic Tree can be expanded flexibly.
The above Logic Tree may not be accepted by some readers. For example, it can be criticized that “to
provide a textbook” is not a direct means and that, instead, this should be “the student to study by
himself”. In practice, all Logic Trees can be true for the developers who create the tree with their own
logic. In other words, there is no “perfect” Logic Tree that everyone in the world would agree to as
being the ultimate model. A Logic Tree should of course be developed in an objective and rigorous way.
It is, however, impossible to form such a perfect model, because human beings cannot escape from
subjectivity. The state of objectivity results from a reorientation of the subjective opinions of human
–beings, in reality. In LEAD, the Logic Tree is a basis of following the evaluation, so it is necessary to
make a consensus on the Logic Tree with the stakeholders, including the groups for whom the
recommendations will be made. Thus, the evaluator should be good at not only creating logic, but also
dealing with the likely diverse opinions of stakeholders in order to reach a consensus.
The student can promote to the next
The student can calculate
To teach mathematics
End
↑
Means
End
↑
Means
To provide a textbook
©FASID 22
2-2-4. Tips for Setting Intermediate Objectives The Logic Tree consists of three layers such as the Overall Goal, Intermediate Objectives and
Activities. In practice, however, it is difficult to set Intermediate Objectives to be filled between the
Overall Goal and Activities at first. The following procedures are recommended.
For projects that have their own project purposes, just assume the project purposes as candidates for
the Intermediate Objectives, rather than decomposing them into activities. After setting the tentative
Intermediate Objectives directly from the project purpose, extract unnecessary activities which can not
be connected to Intermediate Objectives, and consider whether their purposes can become
Intermediate Objectives or not. If there are just activities or schemes without clear-cut purposes,
consider what they are intended to achieve, then set the Intermediate Objectives. When all activities
are set under Intermediate Objectives, check if there are any Intermediate Objectives that are the
same, or very similar. If you find any, then group them under a common Intermediate Objective.
Where there are a lot of projects and activities, input the information into a spreadsheet program
such as Excel. Put in the project names and main activities, and make a rough categorization of the
activities into several groups. By using the sorting function of the software, put the activities into
groups. Then consider the common objective of each group as Intermediate Objectives.
2-2-5. Theory-based Evaluation by Building a Logic Tree The establishing process of a Logic Tree is considered as evaluation, since this process can
distinguish unnecessary activities and illogical projects with unclear purposes. In evaluation, it is very
important to know which activities (or projects) are not necessary to achieve the Overall Goal. You
may be faced with an extreme situation in which you can not construct a Logic Tree since the purposes
of the activities are too diverse to be considered as a program. Continuing to the next step by setting an
irrational Logic Tree is a waste of time and resources. In this case, the evaluator can recommend fast
tracking the reforming of the program, jumping to the improvement phase to start reforming the
program as a new plan, rather than doing a wasteful evaluation study with such an illogical Logic
Tree.
Improvement
Phase
Preparation Phase
Research and Analysis Phase
Scope of Evaluation
Improvement Planning
Feedback
Implementation of Improvement Plan
Logic Tree
Hierarchy Tree
Log-frame
Evaluation Study Implementation
Evaluation Study Design
Conclusion, Recommendations and Lessons
No logic at all….
Need for re-planning.
Initiation Phase
©FASID 23
2. Preparation Phase 2-3. Projects Mapping
a) Outline
Output: Projects Map
Objective: To make geographical map that shows areas where are covered by each project.
Procedures: 1. Collect information in terms of areas where activities are
implemented or positive impact is expected to be reached by each project.
2. Draw above areas on the map by different colour. 3. Confirm the projects Map with the consensus of stakeholders.
©FASID 24
2-4. Log-frame Formation
a) Outline
Objective:
To form the Log-frame which describes the main components of the program based on the Logic Tree.
Procedures: 1. Transfer the components in the Logic Tree into corresponding
cells of “Narrative Summary” of the Log-frame.
2. Set external factors influencing the program into “Important Assumptions” of the Log-frame.
3. Set quantifiable indicators for Overall Goal and Intermediate Objectives.
4. Set “the Means of Indicators”
5. Set “Inputs” for enabling Activities
6. Confirm the Log-frame with stakeholders
Output:
Log-frame (Logical Framework)
©FASID 25
b) Output Sample: Log-frame
Log-frame of Country J’s assistance for secondary education for Amelia
Target Area: The Capital of Amelia Narrative Summary (of
Program/Policy) Objectively Verifiable
Indicators Means of Verification Major Important
Assumptions Super Goal: ・ Poverty in the capital of
Amelia is reduced.
・ The population below the
poverty line is less than 10% in the capital of Amelia in the year XXXX.
・ National statistics from the
Ministry of Social and Health.
・ UNDP Statistics Year Book.
Overall Goal: ・ The secondary school
enrollment in the capital of Amelia is improved.
・ The rate of students receiving
secondary education becomes more than 80% in the city in the year XXXX.
・ The ratio of graduation/entrance of secondary students becomes more than 1 to 10 in the year XXXX.
・ National statistics on
education derived from the Ministry of Education.
・ Government policies
on secondary education are kept unchanged.
・ Other planned poverty reduction programs are implemented.
Intermediate Objectives: 1. The number of school
facilities with sufficient educational function is increased.
2. School curriculum is improved.
3. Teachers’ skills and
knowledge are improved. 4. Students’ morale toward
schooling increases. 5. Accessibility of girls’
schooling is improved.
1-1. More than 90% of schools in the
city meet the national school facility standard level A.
2-1 An improved school curriculum
is endorsed by the educational committee and the third party in the year XXXX.
2-2 More than 80% of schools in the city introduce an improved curriculum in the year XXXX.
3-1 More than 70% of all teachers
have a license with which they can teach the curriculum in the year XXXX.
4-1 More than 90% of all students in
the district enjoy schooling in the year XXXX.
5-1 Less than 10% of all women feel
that a woman has a limited opportunity in the year XXXX.
1-1 Results of audit of the
national standards of school facility.
2-1 Minutes of the Boards of Education.
2-2 Statistics on education (the
Ministry of Education). 3-1 Statistics on education (the
Ministry of Education). 4-1 Results of Questionnaire
and Survey. 5-1 Results of Questionnaire
and Survey.
・ Trained teachers
continue working for the school.
・ School buildings constructed by the project are adequately maintained.
・ Social condition
surrounding students do not worsen.
・ M’s projects to improve secondary school education are implemented.
Activities: 1. Provide school facility 1-1. Set up national standards
for school buildings. Construct school facility
1-2. Equip the facility with bathroom
1-3. Equip the facility with a water tank
1-4. Training on how to manage and maintain the facility
1-5. Procure teaching materials
1-6. Audit the national facility standards
(The rest is omitted)
Inputs: Project Name ・ Secondary school reconstruction project ・ Secondary school teacher training project ・ Secondary school curriculum development project ・ Secondary school scholarship program Total Amounts: ooooooooo yen Major Inputs: ○○、��、×× Stakeholders: Amelia: the Ministry of Education, Ministry of Social Insurance, Ministry
ofJustice Local Public Agency: Department of Secondary Education
Pre-conditions:
©FASID 26
c) Explanatory Notes: 2-4-1. Log-frame Log-frame (or Logical framework) was originated in a management tool of USAID in order to express
the summary of a development project. Log-frame consists of the main components of a project such as
project purpose, outputs, activities inputs, quantifiable indicators, external risks, etc. In PCM,
Log-frame is introduced as PDM (Project Design Matrix), which is created through a participatory
workshop with stakeholders. PDM is utilized in monitoring and evaluation in a project cycle.
Log-frame in LEAD is also a matrix table containing an overall goal, expected objectives, activities,
inputs, indicators, risks, etc., of the program to be evaluated. It is formed to confirm the structure of
the program prior to designing an evaluation study.
The form of a Log-frame is depicted as follows.
Program Title:
Target Area: Date:
Narrative Summary of the Program
Objectively Verifiable Indicators
Means of Verification
Important Assumptions
Overall Goal: Objective that the program should achieve within the program period
Indicators for measuring the level of achievement of Overall Goal
Data sources from which indicators are derived
Conditions for sustainability of the program effects.
Intermediate Objectives: Objectives which should be achieved in order to achieve the Overall Goal
Indicators for measuring the level of achievement of Intermediate Objectives.
Data sources from which indicators are derived
Conditions for Overall Goal besides that of Intermediate Objectives
Conditions for Intermediate Objectives besides that of Activities.
Activities: Summary of Actions to achieve the Intermediate Objectives
Inputs: Personnel, materials, equipment, facilities and funds required by the project
Pre-Condition Conditions that must be fulfilled before the program starts
©FASID 27
2-4-2.Special features of Log-frame for LEAD The Log-frame of LEAD omits the top row of the matrix, compared with
the general Log-frame used in project management. LEAD is developed for
a program or policy level evaluation method, so that the core objective of the
program is the Overall Goal level which is higher than the project purpose
level in the general Log-frame. In cases where there is a higher objective
than the Overall Goal, “Super Goal” should be inserted on the top row of the
matrix. For an example from the Amelia case, the Overall Goal of the
program (a group of projects) is “to increase the number of secondary
education graduates”. In the Amelia case, this Overall Goal is one of several
program goals of the poverty alleviation policy, and the Super Goal is “to
reduce the poverty population in Amelia”, which is located at the top of the
matrix.
On the other hand, there can be a case where the Logic Tree has several layers from the Overall goal
to Activities. In this case, the new row (e.g. “Outputs”) can be added between Intermediate Objectives
and Activities. The Log-frame can be modified flexibly for the needs of the evaluation. It is, however,
recommended to avoid making too complex a Log-frame, that would reduce the understanding of the
stakeholders and that would be difficult to be used in practice.
2-4-3. Log-frame as a Tool for Management The Log-frame contains important information of the program about its objectives and indicators,
and which becomes essential for evaluation on the achievement of the program. The distinct parts in
the Narrative Summary of the Log-frame, which segregate the program into “Overall Goal”,
“Intermediate Objectives”, “Activities”, “Inputs” and “Important Assumptions”, enable the evaluator to
pursue detailed factor analysis on the results of the achievement.
While the Log-frame is used for evaluation, it is the Log-frame’s strength that means it can be used
for program management in planning and implementation, too. PCM (Project Cycle Management)
applies PDM (Project Design Matrix = Log-frame) in all the phases of project management, such as
planning, implementation, and evaluation. By forming a Log-frame in evaluation, it can be used as a
basis for the following phases of planning and implementation. Forming the Log-frame may be
cumbersome if it is used only for evaluation. However, when considering that the evaluation within
LEAD is a step for the next phase of improvement, the Log-frame formation is effective and efficient in
the long-run. If the evaluation needs to be conducted just only for evaluation purposes, it is not
necessary to use LEAD. LEAD is an evaluation method that is integrated intrinsically with the
subsequent improvement of the program by using the Log-frame. Implementation with the Log-frame
is discussed in section 4-3.
Narrative Summary of Program/Policy
Objectively Verifiable Indicators
Super Goal: Poverty reduction in Amelia is promoted
Overall Goals:
Intermediate
Objectives:
Activities: Inputs:
©FASID 28
2-4-4. Converting a Logic Tree into a Log-frame If a Logic Tree has been formed logically, it is technically easy to convert the Logic Tree into a
Log-frame, since the Log-frame’s left column “Narrative Summary” consists of “Overall Goal”,
“Intermediate Objectives” and "Activities” that are connected in a logical relationship of “Means-Ends”,
just like the Logic Tree.
・ Overall Goal (i.e. Program Goal)
This is the goal (purpose) of the program (or a group of projects) to be evaluated, and it is
recommended to set only one Overall Goal. It should describe, in a sentence, the concrete
benefits or impacts given by the program to the beneficiaries (i.e. the target group). It is not a
description of what the program will be doing, instead it should clarify what the program
intends to achieve.
(Bad Example)×
(Good Example)○
Overall Goal:Rice production in village “A“ is increased.
What to achieve
What to do
Overall Goal:Technical assistance for Rice production in village “A“ is conducted
Overall Goal
Objective A Objective B
Activities
Logic Tree
Activities Activities Activities
Narrative
Summary (of Program/Poli
cy)
Objectively Verifiable Indicators
Means of Verificatio
n
Major Important
Assumptions
Overall Goals (Program Goals)
Intermediate Objectives
(Sub-program Goals))
Activities:
Inputs:
Pre-conditions
©FASID 29
・ Intermediate Objectives(i.e. Sub-program Goal)
Intermediate Objectives are intermediate goals to achieve the Overall Goal. For one Overall
Goal, several Intermediate Objectives are set. They are described in a single sentence, as with
the Overall Goal. They are results from Activities. In other words, Activities should not be
confused with Intermediate Objectives.
(Bad Example)×
(Good Example) ○
・ Activities
Activities are descriptions of actions to achieve Intermediate Objectives. For one Intermediate
Objective several Activities are set, which are described with a “to do” expression. It is
recommended to put identical outline numbers (or reference codes) for activities.
2-4-5. Setting Indicators and Means of Verification Objectively Verifiable Indicator (OVI) is a quantifiable degree to be achieved for each objective
expressed in the Log-frame, such as the Overall Goal and Intermediate Objectives. For each objective,
one or more OVI is set. The OVI is described in a sentence, which includes the following information.
Information Example
・ When............................ “By September of year 2005”
・ Where........................... “In area A of country A”
・ Who/What.................... “Average income of farmers”
・ How much.................... “By 30% from the last year”
・ What to do ................... “To be increased.”
Intermediate Objective:Farmers apply the new techniques.
Result
2005 年9月までに、A国 A
地区の農民の平均所得が前
年比で 30%増加する。
Tips: In practice, it is recommended to set one Overall Goal, four-to-seven Intermediate Objectives, and
around four Activities for each objective, to enable a better understanding. Putting codes to
Intermediate Objectives and Activities will help further.
Intermediate Objective:Training for farmers are conducted.
Action
By September 2005,
average income of farmers
is increased by 30% from
the last year.
©FASID 30
By setting the OVI, vague expressions in the Narrative Summary in the Log-frame become clearer to
enable an objective evaluation to be conducted. In the course of setting the OVI, contents of the
Narrative Summary can be elaborated in its definition and modified if necessary.
In practice, the following information will be useful in setting the OVI.
・ Planned indicators in existing policy papers or program documents.
・ Standards regulated in public (National or international standards set by the governments, UN,
ISO, etc.).
・ Indicators used in similar projects.
・ Opinions of program officers, policy makers and other stakeholders.
・ Estimation from logical relationships expressed in the Log-frame.
Choosing the OVI requires utilising the following selection criteria.
・ “Objectivity” .......... It should be based on fact, and not influenced by personal beliefs or feelings.
・ “Direct”.................. The type of data should be exactly what should be measured. For example,
the type of data for an objective ”Income from agriculture is increased” is “income from
agriculture”. It is neither “agricultural production” nor “general income of households”.
・ “Adequate Level” .. The level of data should be exactly what should be measured. For example,
the OVI of an objective “Agricultural technique is upgraded” is not “the number of training
courses conducted”, since it is one of the activities for the objective. On the other hand, “the
amount of agricultural product” is not the proper OVI, since it is the result of upgrading the
agricultural techniques. Technically, there can never be any case where the same OVI is used in
different levels of the Narrative Summary in the Log-frame.
・ “Coverage”............. The OVI should cover all of what should be measured. Sampling should be
conducted without bias.
The cells in the right hand side of the OVI are for "Means of Verification", which indicate the source
of information. Means of Verification states relevant documents, methods, sourcing organizations,
locations and so on, in order to make the setting of indicators and their monitoring more efficient and
effective. When considering the Means of Verification, "availability", "reliability" and
"cost-performance" of information should be considered.
When there is no available source of information, it is necessary to collect information by conducting
researches by the evaluation team. It is, however, recommended to use the existing data as much as
possible in practice, since information collecting itself consumes a lot of time and monetary resource.
The selection of Means of Verification should proceed carefully, along with the selection of OVIs.
©FASID 31
Improved farmer’s skill
Important
Assumption
2-4-6. Setting Important Assumptions "Important Assumptions" are the conditions which are necessary for the program achievement, but
are not controllable by the program, and it is uncertain whether they will be realized or not.
Important Assumptions are found from logics within the Narrative Summary of the Log-frame. As
described below, Important Assumptions for "Activities" to "Intermediate Objectives" can be found by
asking whether there is any condition to be realized besides that of the Activities for accomplishing the
Intermediate Objectives.
For instance, the increase of agricultural production (as the Overall Goal) cannot be achieved only
by the improvement of farmer’s skills (as an Intermediate Objective), as there also needs to be rain. In
this case, "Amount of rainfall is enough" can be considered as one of the Important Assumptions. For
another example, the upgrading of the teaching quality in a school cannot be achieved if the trained
teachers leave the school, and therefore despite the fact that the training has been well conducted for
the teachers. In this case, "The trained teachers continue to work in the school" can be thought of as
one of the Important Assumptions. There are other issues such as economy, politics, environment,
society, culture and so on, for considering Important Assumptions.
Outcomes
Activities Important Assumptions
©FASID 32
When considering Important Assumptions, cross-cutting issues, which the OECD recommends, are
of some help.
Cross-cutting Issues Examples
・ Political ................................................. "The government keeps the policy of supporting
secondary education."
・ Technical ............................................... "The electricity necessary for agriculture is provided."
・ Environment ......................................... "More than 1000mm of annual rainfall is available."
・ Social/ Cultural ..................................... "Not many villagers migrate away for seeking jobs."
・ Institutional .......................................... "Farmers' Cooperative Organizations continue to exist."
・ Economic ............................................... "Market price of rice is maintained."
Projects or activities that are included in the Logic Tree, but are not included in the Narrative
Summary of the Log-frame, are also candidates for Important Assumptions. When there are other
organization(s) to do projects influencing the program, the projects should be considered as one of the
Important Assumptions. This will help in the later stages for analysing the attribution of the program.
In the Amelia case, the project conducted by Country M can be considered as an Important
Assumption.
"Pre-conditions" are the conditions that should be realised before the commencement of the program.
In practice, however, it is a rare case to need the pre-conditions because LEAD evaluation is conducted
mostly for on-going programs.
2-4-7. Setting Inputs The "Inputs" cell consists mainly of human resources, facilities and materials, and other operating
costs. If the inputs are diverse in various operating organizations, the inputs to be divided throughout
the organizations should be so described. This will help the later analysis of the contribution level of
each organization. In practice, the evaluator determines the method of setting the Inputs, but it is
recommended to clarify at least the main components of inputs (e.g. types and amounts of human
resources, facilities, materials and other costs) by each organization. When analysing the efficiency of
the program, it is recommended to put these inputs into each group of Activities or to enumerate the
inputs into monetary values.
©FASID 33
Well Done!
You have successfully achieved the halfway point to the final goal!
Relax for a while......
©FASID 34
3. Evaluation Phase
3-1. Evaluation Study Design
a) Outline
Objective: To clarify what is necessary to be studied and how it is conducted, in order to achieve the evaluation purpose.
Procedures: 1. Reconfirm the evaluation purpose from Scope of Evaluation
and Log-frame.
2. By defining the evaluation purpose, set evaluation criteria and evaluation questions.
3. Set study methods to answer the evaluation questions.
4. Compose the above information into a matrix named "Evaluation Grid", which should be shared with stakeholders.
Output:
Evaluation Grid
©FASID 35
b) Output Sample: Evaluation Grid Example of Evaluation Grid
Evaluation Purpose
Evaluation Questions (Criteria)
Evaluation Questions (Level 1)
Evaluation Questions (Level 2)
Research Methods Information Source
1-1-1. Whether the rate of secondary education enrolment in the capital city is increased up to 90%.
Analyse the past trends by using statistics on secondary education in order to predict future trends.
Statistical Document published by government.
1-1. Is the Overall Program Objective achieved?
1-1-2. Does ratio of graduation/ entrance increase up to 0.9 as planned?
Analyse the past trends by using statistics on secondary education in order to predict future trends.
Statistical Document published by government. Results from
sample survey.
1-2-1. To what extent J’s development assistance contributed to the increase of the education continuance rate?
Examine the association between the enrolment rate and J’s assistance by using regression model.
Document on J’s assistance in each school.
1-2-2. Which programs are significantly effective or ineffective?
Conduct questionnaire among stakeholders particularly on effective programs, including questions on the reasons why they think that is so.
Contract out the survey to a local consultant. Conduct interviews by the evaluation team.
Conduct qualitative evaluation of the impacts of the Country M’s education project based on its evaluation report.
Program evaluation report done by Country M.
1. Is the program effective?
1-2. Is the achievement of overall objectives attributed to Country J’s assistance? How much is its contribution?
1-2-3. To what degree was the contribution of assistance from other donors or agencies?
Conduct questionnaire and interviews among stakeholders on other factors influencing the school enrolment.
Contract out the
survey to a local
consultant.
Conduct
interviews by the
evaluation team.
2-1-1. Is there any redundant program or project?
Check with the Logic Tree Logic Tree.
To conduct comprehensive evaluation of middle school education support programs, which are currently implemented in the capital of Amelia, to modify these in order to make them more effective and efficient.
2. Is the program efficient?
2-1. Is there any redundant or overlapping program?
2-1-2. Is there any program that has identical inputs?
Lay out an actual use of inputs to identify redundancies.
Overall Inputs Matrix.
©FASID 36
2-2-1. Are there any inputs that have significantly low or high level utilization?
Ask stakeholders to rank the order of utilization of inputs and also ask for the reasons.
Contract out the survey to a local consultant.
2-2. Are inputs sufficiently utilized and effectively transformed to outcomes?
2-2-2. How is the transformation from inputs to outcomes?
Ask stakeholders to raise indicators to assess the contribution of inputs to outcomes, and analyse them by running a regression analysis.
Contract out the survey to a local consultant. Analyse the results of the survey by using in-house staff.
©FASID 37
c) Explanatory Notes 3-1-1. Evaluation Purpose and Evaluation Criteria In order to make the evaluation more effective (i.e. to achieve the purpose) and more efficient (i.e. to
consume the least cost), it is crucial to focus the evaluation purpose before designing the evaluation
study. LEAD requires the clarification of the evaluation purpose at the initiation phase (See 1-1
"Scope of Evaluation"), so the evaluation purpose is reconfirmed at this step. If necessary, the
evaluation purpose can then be adjusted and defined more clearly.
For evaluation of international cooperation projects, it has been recommended to consider the five
evaluation criteria of efficiency, effectiveness, impact, relevance and sustainability. In practice,
however, applying all five criteria simultaneously is not recommended when there is limited time and
resources. There is also a risk that all of the five may be outside of the clients' interests, so that the
evaluation would be useless in the end.
With limited time and resources, the evaluator should consult with the clients or the users of the
evaluation, and clarify their interests in the evaluation. For customisation, any basic improvements
needed should be clarified at this stage, such as whether the client wants to improve the effectiveness
or the efficiency of the program, to thereby alleviate the negative impacts, etc. Of course, there can be a
desire to pursue the improvement of every aspect of the program, however such an ambitious (but
ambiguous) comprehensive evaluation tends to be obligated with enormous amount of tasks, and with
a tendency to end with unsatisfactory results. If there is a request for several
evaluation criteria to be evaluated, then these should be prioritised by
ranking them with the stakeholders. The designing of the Evaluation Grid
can be commenced from the high ranked evaluation criteria.
3-1-2. Program Level Evaluation A project evaluation in general considers the whole effect of the project, while a program level
evaluation, in which a program includes several projects, needs to have some evaluation added on the
structure of the program and on the relationships among projects as sub-programs.
Especially for LEAD, which composes the proxy-program containing projects as sub-programs to be
evaluated, one of the most probable recommendations resulting from the LEAD evaluation is to
re-construct the program by adjusting and coordinating existing projects. Thus, the question about
which component (project) should be improved by adding, deleting and modifying is very crucial, so the
evaluation on each component within the program structure is indispensable in any evaluation criteria
selected.
©FASID 38
The Concept Map of Program Evaluation
Overall Goal (Program Goal)
Sub-program
Goal 1
Sub-program
Goal 2
Sub-program
Goal 3
Sub-program
Goal ….n
Activities Activities Activities Activities Activities Activities Activities
Inputs Inputs Inputs Inputs Inputs Inputs Inputs
Evaluation of Program Structure
Evaluation of Program
©FASID 39
Evaluation topics for each of the five evaluation criteria can be described as follows.
Five Evaluation Criteria in Program Level Evaluation
Evaluation Criteria Basic Approach (As a whole)
Advanced Approach (About the Structure)
Efficiency
Consider the Productivity in
implementation. How efficient is
the manner in which the outputs
are produced by the inputs? Are
any inputs wasted? What about
the adequacy of the inputs,
considering the outputs and
outcomes achieved?
Consider adequacy in the use of the inputs
provided by different agencies. Are there
any redundant inputs? What about
adequacy in the sequence of executed
projects? Is there any efficiency-making
effort, e.g. to share the same input by
different agencies?
Effectiveness
Consider the achievement level of
the Overall Goal. Consider as to
whether the Intermediate
Objectives within the program
contribute the achievement of the
Overall Goal.
Consider which component projects are
effective or ineffective to the achievement
of the Overall Goal. Consider the
synergistic effects among the component
projects.
Impact
Consider positive and negative
impacts (changes) accrued directly
or indirectly by the program. The
impacts that are not expected from
the planning are also included.
Consider which component projects are
the most or least influencing in causing the
impacts, in a positive or negative manner.
Relevance
Consider the relevance of the
Overall Goal to the needs of the
beneficiaries, the national and
international goals, socio-
economic trends, etc.
Consider the relevance of component
projects and activities to achieve the
Overall Goal. Consider the coherence
among the component projects. Consider
the maturity of the proxy-program as a
genuine program.
Sustainability
Consider the sustainability of the
impact and benefits accrued by the
program.
Consider which component projects
provide the most or least influence to the
sustainability of the program.
©FASID 40
The above definitions are rough examples to provide some hints for creating evaluation questions.
The evaluator should communicate with the clients and other stakeholders in order to clarify the
evaluation criteria and items to be studied in accordance with the Scope of Evaluation.
3-1-3. Designing Evaluation Study The purpose of the evaluation study is to satisfy the needs of evaluation. Thus, the evaluation study
should be designed by considering how to collect the necessary information in order to address the
needs. The process of designing can be defined as follows.
It is recommended to undertake the design by using the following matrix, known as the “Evaluation
Grid”, which contains the evaluation questions, methods and other necessary information.
Evaluation Grid: Example
Evaluation Purpose
Evaluation Questions
(Criteria Level)
Evaluation Questions
(Level 1…n)
Research Methods
Information Source
If the issues and questions are not clearly defined, specify these by breaking them down into smaller questions
Research methods to answer the evaluation questions in the cell to the left cell
Resources (cost, information) to meet the research methods
Issues and questions to be answered in order to meet the evaluation needs. As the starting point, selected evaluation criteria is clarified at this level
Needs for the present evaluation derived from the scope of evaluation
* Our Output Sample has Level 2 Questions.
For each evaluation criteria, several evaluation questions should be created from which the criteria
can be defined in a concrete manner. In cases where the question is broad and ambiguous, it is
What are the needs for evaluation (from the scope of evaluation)
What you should know to meet the needs?
What and how you should examine to answer the questions
The research method is both effective and efficient?
① ② ③ ④
Broad question
Middle question
Middle question
Narrow question
Narrow question
Narrow question
Narrow question
©FASID 41
recommended to decompose the question into a handful of sub-questions that are more easily
answerable in practice, such as questions at level 1, level 2, level 3 and so on. The method to collect
the information for answering each evaluation question is stated in the column to the right of each
question. The number of these research methods for each question can be more than one. The other
necessary information, such as the names of required documents can be added in the next column. As a
result, the Evaluation Grid gains more rows (or layers) as you move towards the right. The Evaluation
Grid can be used not only for designing the evaluation study, but also for placing the results of the
evaluation study within it.
3-1-4. Setting the Methods of Data Collection Setting the methods of collecting data should need consideration not only of the accuracy and
reliability of the data, but also of the time and resources that will be available in reality.
There are several options for data collection methods as follows.
Method Description Advantages Remarks 1. Literature
Survey
Collection and analysis of existing statistics, survey reports, work records, research data, etc.
Economical and efficient Difficult to verify the reliability of materials
2. Direct Measurement
Collection of data that can be measured and classified based on actual surveys
Provides accurate and reliable data; relatively low cost.
Facts only, no explanations
3. Sample Survey
An interview or questionnaire-based survey, targeting a sample population of a sufficient size with regard to the parent population.
Provides quantifiable and reliable data
High cost
4. Case Studies A detailed study of a small number of cases.
Provides self-contained detailed information
Difficult to generalize
5. Observation
Undertaken using a checklist or survey sheet. May also involve simple interviews.
Little need for preparation. Particularly suitable for a preliminary study.
Results can vary widely depending on the observer.
6. Key Informant Interviews
A technique often used in an evaluation. After selecting persons well-versed in the matters concerned, detailed interviews are conducted.
Low cost and easy to implement. Provides useful information and sometimes draws attention to unexpected problems.
Risk of bias due to the personal views of interviewer and interviewee. Data cannot be quantified. Time consuming
7Group Interviews Interviews with representatives of the community or groups (5-6 people each) in the form of an open meeting.
Background information can be obtained from interviewees’ attitudes and reactions.
There is a risk of creating biased conclusions through manipulation in the interviewee selection process.
©FASID 42
Although the selection of data collection methods is limited to available time and resources, it is
recommended to use the following selection criteria in order to select the best methods.
・Validity ·························Do the evaluation questions cover what should be studied?
・Importance ····················How important are the answers of the evaluation questions to the
evaluation ?
・Reliability······················How much are they reliable in their sources of information and methods?
・Accessibility··················· Is the necessary information easily accessed?
・Cost ······························ Is the cost of conducting the method appropriate?
In the process of designing the Evaluation Grid, participation of stakeholders including the clients is
important. The evaluator should share the process of designing the Evaluation Grid with stakeholders
from the draft version to the final version. This makes the evaluation more realistic and acceptable in
its conclusions and recommendations at the end.
3-1-5. Data Collection for Factor Analysis The designing process of the Evaluation Grid should consider the issue of a factor analysis on the
results of the evaluation study. Evaluation about effectiveness is concerned with the achievement of
the Overall Goal, and a factor analysis needs information about all factors (positive and negative)
influencing the Overall Goal’s achievement, such as the achievement of Intermediate Objectives,
Activities, Inputs, and relative Important Assumptions.
Such sub-topic information should be collected together with the main topic information for greater
efficiency. Therefore, the evaluator should include the data collection for factor analysis at the
Selection of Methods
Cost Accessibility
Validity Importance
Reliability
Selection Criteria of Research Methods
©FASID 43
beginning by examining the Log-frame or Logic Tree, having preliminary interviews with stakeholders,
and predicting the results with experiences of similar programs.
In practice, the following viewpoints are generally needed in collecting data for factor analysis. They
are (1) the achievement level of each components of the Log-frame, (2) the conduct of management in
Activities, and (3) cross-cutting issues.
“Degree of Achievement of the Log-frame” is checked at each level of the Narrative Summary,
such as Intermediate Objectives, Activities, Inputs and relative Important Assumptions.
“The conduct of management in activities” concerns the way of conducting the planned activities,
daily management style, and relationship among stakeholders in reality.
“Cross-cutting issues” consists of aspects of policy, technology, environment, society, institution,
and finance (or economy) that influence the program performance.
For example, the sample Evaluation Grid contains the evaluation question to check ,by conducting a
questionnaire-based survey, as to whether there are other social factors or situations influencing the
secondary school enrollment. This is one of the examples of collecting factor analysis data from the
view of cross-cutting issues in “society” and “environment”.
Factor Analysis
Degree of Achievement of the Log-frame
What are reasons for failure?
Process of Activities (i.e. Stakeholder Relationship)
Why are they achieved? Why are they not achieved?
YES NO
Are overall goals achieved?
Cross-Cutting
Issues
What are reasons for success?
©FASID 44
3. Evaluation Study Phase
3-2. Evaluation Study Implementation
a) Outline
Objective:
To collect the necessary information by conducting the planned evaluation study.
Procedures:
1. To reconfirm the evaluation questions in the Evaluation Grid.
2. To conduct the planned evaluation study using the designed methods.
3. To describe and sort the results of the evaluation questions into the Evaluation Grid.
4. To inform the progress of the evaluation study.
Output:
Evaluation Grid with Results
©FASID 45
b) Output Sample: Evaluation Grid with Results Evaluation Grid with Results: Example
Evaluation Purpose
Evaluation Questions (Criteria)
Evaluation Questions (Level 1)
Evaluation Questions (Level 2)
Research Methods
Results
1-1-1. Whether the rate of secondary education enrolment in the capital city is increased up to 90%.
Analyse the past trends by using statistics on secondary education in order to predict future trends.
The continuance rate has been increasing year by year, but the rate stays between 60% & 70% in the year XXXX. The rate of students from a poor background remains in the low level.
1-1. Is the Overall Program Objective achieved?
1-1-2. Does ratio of
graduation/ entrance increase up to 0.9 as planned?
Analyse the past trends by using statistics on secondary education in order to predict future trends.
The ratio of graduation/ entrance is diverse depending on the school, but the average ratio is approximately 8 to 10.
1-2-1. To what extent J’s development assistance contributed to the increase of the education continuance rate?
Examine the association between the enrolment rate and J’s assistance by using regression model.
Based on the results of regression analysis, J’s overall contribution is R^2=70%, which indicates a positive result, but the scale of the contribution is greatly different among programs.
1-2-2. Which programs are
significantly effective or ineffective?
Conduct questionnaire among stakeholders particularly on effective programs, including questions on the reasons why they think this is so.
Most stakeholders point out that program ○○ is particularly effective but, on the other hand, program xx is not quite effective.
To conduct comprehensive evaluation of middle school education support programs, which are currently implemented in the capital of Amelia, to modify these to make them more effective and efficient.
1 Is the program effective?
1-2. Is the achievement of overall objectives attributed to Country J’s assistance? How much is its contribution?
1-2-3. To what degree is the contribution of assistance from other donors or agencies?
Conduct qualitative evaluation of the impacts of the Country M’s education project based on its evaluation report.
A primary goal of M’s project is to give training on public administration, and therefore there is no direct impact.
©FASID 46
Conduct questionnaire and interviews among stakeholders on other factors influencing the school enrolment.
Educational support activities for women run by a local NGO have as big an impact on broadening educational opportunities for women as does J’s assistance.
2-1-1. Is there any redundant program or project?
Check with the Logic Tree.
Activities of A and B are partially redundant.
2-1. Is there any redundant or overlapping program?
2-1-2. Is there any program that has identical inputs?
Lay out an actual use of inputs to identify redundancies.
Although Programs A and B have almost the same items of inputs, they have been arranged independently and neither have been shared nor coordinated.
2-2-1. Are there any inputs that have a significantly low or high level of utilization?
Ask stakeholders to rank the order of utilization of inputs, and also ask for the reasons.
Opinions from Stakeholders reveal that inputs XX are the best to use, and for the reason _____
2. Is the program efficient?
2-2. Are inputs sufficiently utilized and effectively transformed to outcomes?
2-2-2. How is the transformation from inputs to outcomes?
Ask stakeholders to raise indicators to assess the contribution of inputs to outcomes and analyse them by running a regression analysis.
Input-Outcome Analysis says inputs ○○& �� has the highest transformation rate of all inputs. The overall transformation rate is XX %
©FASID 47
c) Explanatory Notes 3-2-1. Implementation of the Evaluation Study The evaluation study is conducted in accordance with the Evaluation Grid. Sub-contractors can be
procured to undertake the simpler tasks such as a questionnaire survey, so that the evaluator can
concentrate on the whole picture of the evaluation. In using the sub-contractors (e.g. surveyors or
private consultants), the scope of work of the contract should be linked coherently with the Evaluation
Grid, which is shared with the sub-contractors, so that the sub-contractors should conduct only but the
necessary tasks.
When progressing the evaluation study, there may be a need to alter some components of the
Evaluation Grid . In this case, the Scope of Evaluation should be reviewed before the identified
adjustment is accepted. In other words, “the Scope of Evaluation”, “the Evaluation Grid” and “the
Scope of Work with sub-contractors” should be integrated and managed consistently.
3-2-2. Question of Attribution One of the most challenging tasks of the evaluation process is the factor analysis to clarify the level
of attribution of each factor to the results. The following diagram depicts the hypothetical model of
attribution.
In the above diagram, there are three factors at the Intermediate Objectives level to attribute the
achievement of the Overall Goal. It is important to know which of these factors are the greater or least
contributors to the Overall Goal. It is, however, a daunting task to pursue the answer in an objective
manner. Currently, there are several techniques to do this type of analysis as follows.
The literacy of students from
District A is improved
School facility is renovated.
Teacher’s capability is
improved
Intermediate
Objectives
Overall Goal
1 2 3
Teaching materials are improved
©FASID 48
Techniques Description Remarks
Comparative Analysis
with a similar program
Compare with the similar program which particular component
projects or activities are not included. From the comparison,
estimate the negative effects of the lost part of the program. In
another word, benefits accrued from the program of our interest is
equivalent to the negative effects derived from the estimation.
It is rare to find a similar program.
Comparative Analysis
in Time Series
Measure the degree of the change of the achievement level of the
Overall Goal between before and after the target project or activity
started.
It is difficult to distinguish the sole effects from the others that have already existed.
Regression Analysis Enumerate the achievement of each item in the program and apply
the variables into the regression model to calculate the coefficients
(a1-aN).
(Overall Goal) = a1 x (IO1) + a2 x(IO2)+…+ aN x(ION)+ e
where “IO” means “Intermediate Objectives”
This can be applied into panel data or time series data.
Enumeration and modelling are difficult.
Qualitative Analysis Qualitative judgment made by stakeholders whom are well-versed.
May be quantified by making questionnaire surveys and analysed
with statistical techniques (Average and Variance).
Proper sampling is difficult. Result is still subjective.
Experimental
Techniques
From the beginning, apply the various sets of components in
various areas and monitor the difference of the results.
Not possible to apply in ex-post evaluation. Basically denied from the ethical point of view.
*Details of these techniques can be referred to in books listed in the annex.
There are other techniques applicable for attribution analysis, but there are no standardized
techniques in reality. From the other point of view, it is not recommended to use a highly complex
technique that is understandable only to a limited number of specialists, because such a complex
technique obstructs the understanding of the stakeholders, most of which are not experts on the
technical issues. The reduction of understanding among the stakeholders will be more harmful than
losing the technical robustness of evaluation, in the sense that such common stakeholders are the
users of the evaluation in practice. If the user cannot understand the results, the recommendations
won’t be accepted. Using highly complex techniques and modelling is also harmful in the sense that it
©FASID 49
would make the stakeholders, as well as the evaluator, lose the whole picture by focusing excessively
on detail.
Thus, when conducting the evaluation study, the evaluator should remember the needs and purpose
of the evaluation. The evaluation study should be implemented in order to make good
recommendations for improvement, and to give useful information for the users of the evaluation at
the end.
3-2-3. Sorting the Results of the Evaluation Study
Results of the evaluation study are put into the Evaluation Grid.
The style chosen for the description of the results is part of the evaluation process to be decided. The
results are placed to the right of the description of the research methods in the Evaluation Grid. If the
information in the results column is too extensive, the results should be summarised with the detailed
information being attached in an annex.
Evaluation Grid with Results
Evaluation
Purpose
Evaluation Questions (Criteria)
Evaluation Questions
(Level 1...n)
Research
Methods
Results
Notes
If the issues and questions are not clearly defined, specify these by breaking them down into small questions.
Research methods to give answers to the evaluation questions in the cell to the left.
Summary of Evaluation Results
Documents related to the results and their serial numbers.
Issues and questions to be answered in order to meet the evaluation needs. (Evaluation Criteria selected at this level)
Needs for the evaluation derived from the Scope of Evaluation
The attached documents should be written in a detailed but understandable form by using charts
and graphs. These documents do not just form a record. The documents in the evaluation should be
always be written by bearing in mind that they will be read by important stakeholders.
©FASID 50
3. Evaluation Study Phase 3-3. Conclusions, Recommendations and Lessons Learned
a) Outline
Objective::
To draw recommendations and lessons learned with overall
conclusions.
Procedures:
1. Draw overall conclusions by reviewing the results of the
evaluation study.
2. Draw recommendations.
3. Draw lessons learned.
4. Make presentation materials by summarizing the progress and results of the evaluation.
Output:
Conclusions, Recommendations and Lessons Learned
©FASID 51
b) Output Sample :”Conclusions, Recommendations and Lessons learned”
アメリア国中等教育支援プログラムの総合評価取りまとめ例
評価結果取りまとめ
総合評価
全体目標の達成度については堅実に中等教育の進学率は伸びてきて
おり、さらに卒業率の改善も目覚しい。一方で、その改善は貧困層
に属する学生とっては限られたものである傾向があり、また有効性
からみても、貧困層や女性をターゲットにした事業が十分にその効
果を発揮していないことが懸念される。特に貧困層をターゲットと
した一部の事業の有効性が低かったことは、貧困が一部の事業で対
応するには大きすぎる問題であることと、事業単位で実施されてお
り、J 国の他の事業や NGO など既存の活動との総合的なアプローチ
がとられていないことに起因するものと考えられる。
効率性の観点からも、同じような事業や活動をしている場合も多く、
また他の活動の経験や教訓などが十分に共有されていないなどの問
題が指摘されうる。投入財についてはおおむね効率的に成果に転換
されていると判断されるが、より事業を統合化し、共有するべきと
ころは共有するなどすれば、さらなる効率化が図られる可能性があ
る。
提 言:
(J 国に対して)
・ 貧困層支援の視点から、J国の各プロジェクトの組み合わせを見
直し、ロジック・モデル及びログフレームを改善することを提
言する。
・ 重複している活動については統合し、連携を強める。
・ 今後はそのログ・フレームを元に、各プロジェクトが進捗状況
や情報を共有し、連携を保ちながら実施する。
・ 上記プログラムを運営管理するための担当者を任命する。
(アメリア国に対して)
・ 以下の情報を、J 国のロジック・モデル作成の際には、それを提
供する。
- 中等教育に係る政府の中期戦略
- 中等教育にかかる現地 NGO の活動
- 中等教育にかかる他のドナ-の活動
教 訓:
中等教育に関する計画を検討する際には、貧困層への十分な配慮が
必要であり、NGO などのコミュニティーの協力は不可欠である。
Summary of Evaluation Results for Secondary Education Assistance of Country J Title Contents
Summary and
Conclusions
The degree of the achievement of the Overall Goal indicates that the enrolment of the secondary school has been improved, and especially that the graduation rate has been increased dramatically. There is, however, concern that the improvement seems to be enjoyed solely in the area populated mainly by middle income class households, and that the improvement is limited in the “poor” area, where the population of the residents have the lowest level of income. In the gender perspective, the improvement of the state of the female students is not significant compared with the male students.
The tendency for benefits not to be received by poor or female students should be of great concern, in the sense that Country J’s assistance is one of the programs for the poverty alleviation policy. There has been no significant improvement observed for the poorest group of students. There are some reasons for this. Firstly, poverty itself is the complex and difficult problem that the secondary school program can not solve on its own. Secondly, the projects have been conducted independently in practice and there is no effective coordination system among them. Thirdly, there has not been any cooperation with other active supporters, such as local NGOs and private schools.
The last two matters are concerned from the point of view of efficiency. Several inputs and activities are overlapped but neither shared nor coordinated. The lessons learned from a project did not get fed back to other projects, thus the same type of mistakes have been repeated. All inputs are efficiently utilized to the full, but a coordination of projects will increase the efficiency of the whole structure of Country J’s assistance.
Recommendations
�
(For Country J) ・ The coordination of projects should be fundamentally reviewed and
reconstructed from the view of assisting the poor, by improving the Logic Tree and the Log-frame.
・ The overlapped or very similar activities should be combined and coordinated consistently.
・ The multiple projects should be managed as a program by using the improved Log-frame.
・ A full-time program manager for managing the program should be assigned, while the project-level management should be conducted by the project leaders, and not by the headquarters office.
(For Amelia government) In the improvement process, the government of Amelia should provide the information as follows to Country J.
- The mid-term strategy for secondary education. - The detailed information of the local NGOs concerned.
- The detailed information of the main donors concerned.
Lessons
Learned
In planning the assistance of secondary education in Amelia, it is indispensable to consider the situation of poor people, and to pursue the support from the community such as local NGOs.
©FASID 52
c) Explanatory Notes
3-3-1. Drawing Conclusions Conclusions are not the summary of the results from the evaluation. They consist of the following
points.
・ How are the results relating to the needs and purpose of the evaluation expressed in the Scope
of Evaluation?
・ In the case of setting several evaluation criteria, which of the criteria are the results considered
to be very successful or unsuccessful?
・ Is there any underlying common factor which boosts or hampers the results?
・ In the cases of where problems have been found, is it necessary to implement any improvement
measures?
The conclusions are a basis from drawing the recommendations, addressing any problems pointed
out in the conclusions.
3-3-2. Drawing Recommendations Recommendations are drawn from the conclusions in order to improve the situation. In practice,
drawing recommendations can be promoted by using a participatory approach involving the relevant
stakeholders, including the program implementer who will be given the recommendations. Through
open discussions for giving ideas about possible recommendations, the final version of these
recommendations can be drafted and their feasibility improved. The following remarks should be
reiterated.
・ Recommendations should be drawn from the conclusions. Recommendations without any link
with conclusions are not accepted.
・ Clarify with whom the recommendations are made.
・ Avoid using ambiguous expressions, and make them as concrete as possible.
・ The number of the recommendations should be a handful, and less than seven.
・ In order to realize the recommendations, the recommendations can address the need for
structural reform of an organization or a detailed action plan, if necessary.
©FASID 53
3-3-3. Drawing Lessons Learned Lessons learned are lessons that should be considered when planning and implementing similar
programs. Like drawing the recommendations, a participatory approach is encouraged to be applied
in drafting the lessons, utilizing the various participatory methods such as brainstorming. The
finalizing of lessons learned should also be made with consideration to feasibility.
The following are critical points when drawing lessons learned.
・ Clarify “for whom” and “which kind of program” that the lessons learned contribute to.
・ Avoid using ambiguous expressions, and make them as concrete as possible.
・ The number should be limited to no more than three.
・ If the lessons learned can not be applied without a reform of the organisational structure, then
such a point should be included as a long-term recommendation.
3-3-4. Preparing Presentation Materials Preparing presentation materials is one of the most important tasks in evaluation, since an
evaluation that is not understood by the stakeholders is likely to be judged as a bad evaluation, even
though its methods and contents are correct. The Evaluation Study Phase is closed only after the
evaluation results are understood by the stakeholders, including the program implementers. Thus, the
documents of the evaluation should be carefully prepared for the purpose of presentation. The
presentation materials should consist, at least, of the following items.
・ Evaluation Report
Chapter 1 Program to be evaluated
Chapter 2 Evaluation Study Design
Chapter 3 Results of Evaluation
Chapter 4 Conclusions, recommendations and lessons learned.
・ Executive Summary of Report
Program to be evaluated
Conclusions, recommendations and lessons learned.
・ Slide show data with computer software (e.g. MS Power Point)
* In the cases where the client has a particular preferred format of reporting, then that should be
utilised.
©FASID 54
4. Improvement Phase
4-1. Feedback
a) Outline
Objective:: To acknowledge the results of the evaluation with the stakeholders Procedures:
1. Present the progress and results of the evaluation
2. Obtain comments from stakeholders
3. Discuss with stakeholders about improvement
4. Clarify the Commitment of Improvement with stakeholders
Output:
Commitment of Improvement
©FASID 55
b) Output Sample: Commitment of Improvement
Title: Commitment of Improvement for Country J’s assistance for the improvement of secondary education in Amelia.
Items Contents
Recommendation LEAD evaluation, conducted in the year XXXX, reported the following recommendations for Country J’s assistance for the improvement of secondary education in the Republic of Amelia:
1. The coordination of projects should be fundamentally reviewed and reconstructed from the view of assisting the poor, by improving the Logic Tree and the Log-frame.
2. The overlapped or very similar activities should be combined and coordinated consistently.
3. The multiple projects should be managed as a program by using the improved Log-frame.
4. A full-time program manager for managing the program should be assigned, while the project-level management should be conducted by the project leaders, not by the headquarters office.
Commitment of Improvement
In response to recommendations, the country of J will implement the following actions. 1. Hold a LEAD workshop to re-build a LEAD Logic Tree in reviewing
the combination of several projects, except for Project A. Also hold a discussion to construct the LEAD Log-frame.
2. Take necessary measures to cut off redundant activities based on the improved Logic Tree.
3. It is difficult to administer Log-frame management in J’s assistance where project-based management is done. Log-frame management should be implemented in a few years time, but is referred to in the comprehensive review.
4. Appoint a commissioner to promote project harmonisation between the country of J and the Republic of Amelia, and initiate research that helps to implement it in a few years time.
Responsible Personnel/
Agency
For actions 1 to 3, Department of XX, J’s Agency for Technical Assistance For 4, appoint a responsible person for coordination by XX, XXXX.
Others Next action: Run a preparatory committee for the workshop stated in the first action point in XX, XXXX.
Confirmed and signed by: Mr. X Ms. Y Mr. Z (J’s Agency for Technical Assistance)
©FASID 56
c) Explanatory Notes
4-1-1. Commitment of Improvement Unfortunately, there are a lot of cases that good recommendations from correct evaluations are not
implemented in reality. One of the main causes of this failure is the lack of follow-up action to ensure
the implementation of recommendations, while there are other causes such as the giving of useless
recommendations that are ambiguous, unfeasible and irrelevant to the needs of the program
implementers. Recommendations should be considered as a request for improvement that needs a
reply from the concerned party. The concerned party (the program implementer in most cases) that
receives the recommendations should clarify their commitment to the request for improvement by
issuing the Commitment of Improvement.
The Commitment of Improvement expresses the basic strategy to ensure the implementation of the
recommendations, and acknowledge the respective roles of the stakeholders. It is not necessary to
determine a detailed plan for implementation. Such a plan will be constructed in the next step.
The basic contents of the Commitment of Improvement are as follows (although they can be modified
if necessary).
Title Contents
Recommendations from the last
evaluation
State the recommendations and clarify them here if some ambiguity is
found
Basic Strategy of Improvement Clarify which of the recommendations are accepted and complied
with. Also clarify the measures taken concerning the
recommendations that are not accepted. Decide whether the
Log-frame should be changed or not.
Implementers
Clarify who is responsible for leading the improvement.
Schedule State when and how the first action is commenced, such as the initial
meeting.
Main Stakeholders Describe other stakeholders such as institutional and financial
supporters, and their roles.
Monitoring
How is the progress of the improvement monitored?
©FASID 57
Commitment of Improvement is the policy level document that describes a rough sketch of the basic
strategies and roles of the stakeholders. Individual stakeholders consider their own detailed
implementation plans.
4-1-2. A Participatory Approach in Making the Commitment of Improvement It is recommended to apply a participatory approach in making the Commitment of Improvement,
by holding a meeting of stakeholders after the presentation of evaluation results. In the meeting,
decisions on several topics such as whether all or part of the recommendations are accepted, and who
will take the leading role of implementation, are discussed and determined. Such a meeting will enable
the following actions to be implemented more realistically. The chairperson of the meeting should be
the representative of the concerned party for whom the recommendations are intended. In practice, the
meeting may need a facilitator or a consultant from the evaluation team, in order to support a smooth
procedure at the meeting.
The representatives of the concerned party produce the draft version of the Commitment of
Improvement. However, it should be shared and agreed by consensus of all the main stakeholders
before final authorization is given.
©FASID 58
4. Improvement Phase
4-2. Improvement Planning
a) Outline
Objective: To improve the Log-frame in line with the Commitment of Improvement
Procedures:
1. Reconfirm the Commitment of Improvement
2. Improve the Logic Tree through a participatory planning workshop
3. Form the Log-frame from the improved Logic Tree
4. Hold a meeting to discuss the feasibility of the improved Log-frame
5. Authorize the improved Log-frame to be shared with stakeholders
Output:
Improved Logic Tree Improved Log-frame
©FASID 59
b) Output Sample: Improved Log-frame Amelia: J’s assistance for the improvement of secondary education: Improved Log-frame
Target Area: Capital of Amelia Month/Date/Year Narrative Summary (of
Program/Policy) Objectively Verifiable
Indicators Means of
Verification Major Important
Assumptions Super Goal: ・ Poverty in the capital of
Amelia is reduced
・ The population below the poverty
line is less than 10% in the capital of Amelia by the year of XXXX
・ National statistics
derived from the Ministry of Social and Health
・ UNDP Statistics Year Book
Overall Goal: ・ The secondary education is
more accessible for the poor in the capital of Amelia.
・ The rate of poor population receiving secondary education is increased more than 80% in the capital by the year XXXX.
・ The ratio of graduation/entrance of secondary students in Special Assistance Districts is more than 1 to 10 by the year XXXX.
・ National statistics
on education derived from the Ministry of Education.
・ Social Statistics
・ Government policies
on secondary education are kept unchanged.
・ Other poverty reduction programs are implemented.
Intermediate Objectives: In Special Assistance Districts of the capital of Amelia- 1 The number of school
facilities with sufficient educational function is increased.
2 School curriculum is
improved. 3 Teachers’ skills and
knowledge is improved. 4 Students’ morale toward
schooling (learning?) is increased.
5 Accessibility of girls’
schooling is improved.
1-1.More than 90% of schools meet the
national school facility standards level B in the special district.
2-1 An improved school curriculum is
endorsed by the educational committee and by the third party by the year of XXXX.
2-2 More than 80% of schools in the city
introduce an improved curriculum by the year of XXXX.
3-1.More than 70% of all teachers in the
district have a license with which they can teach the program by the year of XXXX.
4-1.The rate of students from the special
school district who think they enjoy schooling increases to more than 90%.
5-1.The rate of women who think girls
have limited educational opportunities decreases to less than 10% by the year of XXXX.
1-1 Results of audit of the national standards of school facilities 2-1 Minutes of the Boards of Education 2-2 Statistics on education (the Ministry of Education) 3-1 Statistics on education (the Ministry of Education) 4-1 Results of Questionnaire and Survey 5-1 Results of Questionnaire and Survey
・ Trained teachers
continue working for the school
・ School buildings constructed by the project are adequately maintained
・ Social condition surrounding students do not worsen.
・ M’s projects to improve secondary school education are implemented.
Activities: 1. Provide school facility in a
special district 1-1. Select a school for the
program 1-2. Construction of a school
building 1-3. Equip the facility with a
bathroom 1-4. Equip with a water tank 1-5. Training on how to manage
and maintain the facility 1-6. Procure teaching materials 1-7. Audit the national facility
standards (The rest is omitted)
Inputs: Project Name ・ Secondary school reconstruction project ・ Secondary school teacher training project ・Secondary school curriculum development project ・Secondary school scholarship program Total Amounts: ooooooooo Yen Major Inputs: ○○、��、×× Stakeholders: Amelia: the Ministry of Education, Ministry of Social Insurance,
Ministry of Justice Local Public Agency: Department of Secondary Education
Pre-conditions:
Remarks: After conducting the participatory workshops and social surveys, the program is improved to thereby be focusing more on the poor.“ Special Assistance Districts” are areas where the average income of households is less than 30% of the average of the total population of the capital.
©FASID 60
Output Sample 2: Improved Logic Tree created for forming the Log-frame
Poverty
reduction in the capital
city of Amelia is promoted.
Enrollment rate for secondary education from
the poor families in
capital city of Amelia
increases.
The number of good school buildings increases
Teaching curriculum improves
Teaching methods improves
Student's morale for schooling improves
Female access to education increases
Activities Activities Activities Activities Activities Activities Activities Activities Activities・Omitted・ Activities ・Omitted・
©FASID 61
c) Explanatory Notes 4-2-1. Cases Where an Improvement of the Log-frame is Not Necessary In cases where the Commitment of Improvement does not include the restructuring of the Log-frame
in its basic strategy, those procedures in this stage can be skipped. In such a case, the existing
Log-frame is accepted as the base plan and the action plan is formed for the necessary improvements.
4-2-2. Workshop for Improving the Log-frame In cases where an improvement of the Log-frame is needed, it is recommended to hold a
participatory workshop with the stakeholders, rather than planning this at a desk or in a
meeting attended by a limited number of planners.
A participatory workshop has several advantages as follows.
・ It nurtures and encourages the ownership of the stakeholders for the Log-frame and the
action plan, which will increase their feasibility.
・ It improves the effectiveness of the plan by considering and adding various ideas from
multiple views of stakeholders.
・ It promotes team building among the stakeholders through such teamwork.
There are, however, some disadvantages with the participatory workshop, such as “the planning is
biased towards a few powerful participants”, “analysis tends to be short-sighted”, “the structure of the
plan is inconsistent”, etc. To avoid these weaknesses of the participatory workshop, it should be
carefully planned and managed by an experienced workshop facilitator.
In the participatory workshop for considering improvements, the participants are the stakeholders
indicated within the Commitment of Improvement. Discussions and analysis in the workshop can be
progressed smoothly by using paper cards (half the size of A4 paper) and larger pieces of paper in order
to visualize the discussions and analysis.
The goal of the workshop is to improve the Log-frame as the base plan.
Steps in the Participatory Workshop for Log-frame Improvement ① Review of the existing Logic Tree
- To review the Logic Tree critically from the Overall Goal, and identify the problems and
shortcomings
② Improvement of the Logic Tree
- To refine the structure of the Logic Tree to attain the Overall Goal effectively.
③ Formation of the Log-frame improved.
©FASID 62
The above sequence of procedures is a basic example, with the program of the workshop being
customized according to needs and constraints. When planning a participatory workshop of LEAD, it is
recommended to read the guidebook about the Participatory Planning method of PCM, since the PCM
is the foundation of the LEAD method. It is also a good idea to consult experienced moderators of the
PCM workshop first of all.
The evaluation study must have produced documents containing useful information. The results and
other information from the last evaluation should be fully utilized within the planning process,
especially when selecting the alternatives.
4-2-3. Ex-ante Evaluation of the Improved Log-frame After drafting the first version, the improved Log-frame is assessed and adjusted with the following
evaluation criteria.
・ Feasibility of the Plan
Is the program logic (or theory) well structured?
What is the reality of the setting of each component (especially the inputs)?
・ Five Evaluation Criteria
Is it “relevant” to the national policy and the needs of the beneficiaries?
Is it effective to achieve the Overall Goal?
Is it an efficient way for using the least inputs to produce the best outputs?
Are there any positive or negative impacts to be concerned?
Is it sustainable in its effects and good impacts?
Through the above ex-ante evaluation, the program expressed in the Log-frame is adjusted and
improved before commencement if necessary. The ex-ante evaluation is recommended to be a part of
the participatory planning workshop. For increasing the credibility of the plan, some of the ex-ante
evaluation should be conducted by the third party.
©FASID 63
4. Improvement Phase 4-3. Improvement Implementation
a) Outline
Objective: To implement the improved Log-frame with a monitoring system, which enables the continuous improvement of the program.
Procedures:
1. Implement the action plan that complies with the improved Log-frame.
2. Conduct monitoring on the progress of the plan.
3. Implement a periodical evaluation study.
4. Iterate the improvement phase with the periodical evaluation study.
Output:
Continuous Improvement within the Improvement Cycle
©FASID 64
b) Output Sample: Improvement Cycle
Commitment of Improvement
Recommendation
Implementation
(Do)
Evaluation
(See)
Planning
(Plan)
Regular Evaluation
Implementation of Activities
Planning of Activities
Planning of Improved Log-frame
Monitoring
Commitment of Improvement
Start from here!!
©FASID 65
c) Explanatory Notes 4-3-1. Progress Monitoring Progress monitoring is one of the most important activities after planning. Objectives of the progress
monitoring are to confirm the progress towards the Overall Goal, and to conduct a proper adjustment
if necessary. Thus, monitoring takes care not only of the Overall Goal, but also other components of the
Log-frame such as the Intermediate Objectives, Activities, Inputs and Important Assumptions. The
components to be monitored can be selected by referring to the Log-frame, in which the components
are structured in the “Means-Ends” relationship.
The most important item to be monitored is the achievement of the Overall Goal. It is, however, very
rare that effects on the Overall Goal can be measured during the early stages of the program.
Therefore, in the early period of the implementation, monitoring is concerned only with achievement of
Inputs and Activities. As the program is progressed, the effects on Intermediate Objectives can be
measured. Then finally the effects on the Overall Goal are reached, which is the final stage of the
program.
In analysing the causes of delay in progress, the “means-ends” relationship in the Log-frame can be
very useful. When finding that the effect on one of the Intermediate Objectives is not confirmed as
planned, the achievement of the activities is firstly examined as the means for the Intermediate
Objective. Next, the realisation of the relevant Important Assumptions to the Intermediate Objectives
can be referred to. If finding that some of the activities were not implemented, the cause can be
analysed by examining the achievement of the planned components in the Inputs section or the
realisation of the relevant Important Assumptions.
Narrative
Summary (of
Program/Policy)
Objectively
Verifiable
Indicators
Means of
Verificati
on
Major
Important
Assumptions
Overall Goals
(Program
Goals):
Intermediate
Objectives
( Sub-program
Goal)
Activities:
Inputs: Pre-condition
s
”Means-Ends”
relationship
©FASID 66
If these problems are found, it requires counter-measures such as adjustment, avoidance, mitigation,
and so on. The counter-measures are different in each situation, but it is important to plan for the
counter-measures within the basic organisation.
4-3-2. Monitoring System Monitoring should be conducted systematically within a monitoring system, which must consist of
“data collection”, “aggregation”, “decision” and “communication” as its components. The personnel and
organizations responsible for this should be clarified for each component. It helps the understanding if
the monitoring system is described in a diagram or a table as follows.
Example of a Monitoring System Narrative summary
Objectively Verifiable Indicators
Means of Verifi- cation
Data Collection ・ personnel ・ schedule ・ method
Aggregation ・ personnel ・ schedule ・ method
Decision ・ personnel ・ schedule ・ method
Method of counter- measures
(Intermediate Objective: 1) Literacy rate of women in village A is increased.
By the year XXXX, more than 80% of adult women in village A can read basic texts.
Monitoring report of the program
・ Chiefs of women’s groups
・ Every 3 months
・ Sample survey
・ Leader of literacy program
・ Every 3 months
・ Aggregate data
・ Steering Committee
・ Every 3 months
・ Discuss and vote
Necessary counter- measures can be conducted by the implementers, if they are at the activities level.
Intermediate Objectives are not achieved
Are activities successfully
implemented?
Are major important assumptions
realized?
Are inputs successfully
projected?
Do other activities achieve their
goals?
Are major important
assumptions realized?
©FASID 67
The above table is just an example. The monitoring system should be customized with the needs of
the stakeholders and based on utility.
4-3-3. A Periodic Evaluation for Further Improvement A periodic evaluation is necessary in order to improve fundamentally the structure of the program
as the program is being progressed. The periodic evaluation starts by creating a Scope of Evaluation,
and with the procedure then just following the steps from the evaluation phase of LEAD, as described
previously. Conducting the LEAD procedure the second time will be much easier than the first, since
the Log-frame has already been constructed and the stakeholders will be more experienced.
©FASID 68
5. Epilogue: Everlasting Cycle of Improvement
The concept of PCM (Project Cycle Management) is to manage the entire cycle of a project, consisting
of planning, implementation and evaluation, consistently with a prepared plan named PDM (Project
Design Matrix), which is the Log-frame.
In contrast, LEAD (Log-frame Evaluation Application Design) starts from the evaluation by
combining related projects and activities into a proxy-program into a Log-frame, which is then
evaluated as a program. LEAD continues to the following improvement phase, in which the
recommendations from the evaluation lead to the improvement of the existing program by a
re-planning of the Log-frame. The improved Log-frame will be used in the management of the program
cycle. LEAD results in management via a program cycle of continuous improvement, known as
“KAIZEN” in Japanese.
The basic concept of the cycle of continuous improvement is the same as that of the PCM, such as a
cycle of “plan”- “Do”- “See”. LEAD places greater emphasis on improvement, so that a PDSA
(Plan-Do-See-Action) cycle applied in Japanese TQC (Total Quality Control) is more suitable to LEAD.
The message of the PDSA cycle is that the evaluation should never be conducted only just for the
purpose of evaluation, and that the evaluation must be strongly linked with further actions to improve
the situation.
The PDSA cycle never stops, with improvement being continuous. The objective of the LEAD
evaluation is not to give recommendations that are only tentatively worthy. Rather, it is to install a
cycle of continuous improvement by the use of a Log-frame. LEAD leads you to establish such an
integrated management system with a Log-frame.
Program A
Integrating as Program
Evaluation
Improved Plan
Implementation of Plan
Regular Evaluation
Cycle of Improvement
Program B
Program C
Program D
©FASID 69
Reference Books (not completed, but useful) ・ FASID (2002), PCM Project Cycle Management Participatory Planning,
FASID ・ FASID (2000), PCM Project Cycle Management Monitoring and Evaluation,
FASID ・ Rossi, Peter H, Forward E. Freeman, and Lipsey, Mark W.(1999),
Evaluation: A Systematic Approach, 6th ed., SAGE Publications ・ Weiss, Carol H.(1998), Evaluation: methods for studying programs and
policies, 2nd ed.., Prentice Hall ・ Wholey Joseph S., Harry, and Kathryn E. Newcomer, eds(1994), Handbook
of Practical Program Evaluation, Jossey-Bass ・ Patton, Michael Quinn. (1997). Utilization-focused evaluation: The new
century text (3rd ed.). Thousand Oaks, CA: Sage
©FASID 70
Member List Members of Study Group for Evaluation on Public Policy and Programs - Ryo SASAKI - Keiko NISHINO - Kiyoshi YAMAYA - Masaki SAITO - Yujiro HAYAMI - Masafumi YAGI - Yoko ISHIDA - Naonobu MINATO Members of PULUG (Production Unit for LEAD Users Guide) - Takahiro MIYOSHI - Yasushi HAYASHI - Mari FURUYA - Mitsuru KAYAMA All comments and inquiries must be received by Naonobu Minato. The contact address is [email protected] Foundation for Advanced Studies on International Development (FASID) Chiyoda Kaikan Building (4th and 5th fl. ) 1-6-17, Kudan-Minami, Chiyoda-ku, Tokyo 102-0074, Japan