Upload
joie
View
34
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Suggestions for the Evaluation of Transitional Justice. Angle Taken in this Presentation. Perspective of an evaluator of conflict related programmes, of which TJ is one field. The experience of TJ is drawn from: - PowerPoint PPT Presentation
Citation preview
Suggestions for the Suggestions for the EvaluationEvaluation
of of Transitional JusticeTransitional Justice
AngleAngle Taken in this Taken in this PresentationPresentation
Perspective of an evaluator of conflict related Perspective of an evaluator of conflict related programmes, of which TJ is one field. The programmes, of which TJ is one field. The experience of TJ is drawn from:experience of TJ is drawn from:
Evaluation of EC support to human rights Evaluation of EC support to human rights and the rule of law, thematic 2005, and and the rule of law, thematic 2005, and support to ICC ratificationsupport to ICC ratification
Evaluation of USAID peace-building and Evaluation of USAID peace-building and justice support programmes in Burundi, justice support programmes in Burundi, Senegal (Casamance) and Uganda (north).Senegal (Casamance) and Uganda (north).
Evaluation of a World Bank and of a Danida Evaluation of a World Bank and of a Danida funded project in Burundi concerning the funded project in Burundi concerning the repatriation of refugees with a focus on repatriation of refugees with a focus on restorative and traditional justice.restorative and traditional justice.
WhatWhat is Evaluation ? is Evaluation ?
Evaluation is…Evaluation is… To assess or judge the value or worth of To assess or judge the value or worth of
something (its relevance, effectiveness, something (its relevance, effectiveness, efficiency, sustainability)efficiency, sustainability)
It is Not MonitoringIt is Not Monitoring::Continual self-evaluation of changes to generate Continual self-evaluation of changes to generate
a lateral view, through formal and informal a lateral view, through formal and informal processesprocesses
It is Not Impact AssessmentIt is Not Impact Assessment: : Tracking of change in a society as it relates to an Tracking of change in a society as it relates to an
identifiable source or set of factors identifiable source or set of factors
Why Why Evaluation?Evaluation?
WhyWhy Evaluation ? Evaluation ?
AccountabilityAccountability LearningLearning
… … However evaluation comes at a However evaluation comes at a greater cost (financial or not) greater cost (financial or not) than is acknowledged, and are than is acknowledged, and are used sparingly.used sparingly.
WhyWhy Evaluation? Evaluation?
Evaluation is part of the concept of results Evaluation is part of the concept of results based management: a verifiable argument based management: a verifiable argument about what worked and what did not, and about what worked and what did not, and why is it so.why is it so.
It guides the interface between a programme and a It guides the interface between a programme and a society- all too often this information is restrictedsociety- all too often this information is restricted
Creating an information and analysis Creating an information and analysis environmentenvironment
Creating a flow of decision-oriented informationCreating a flow of decision-oriented information Obtaining reliable information about the progress of Obtaining reliable information about the progress of
activites and their outcomes, the reasons for success activites and their outcomes, the reasons for success or failure, and the contexts in which activities are or failure, and the contexts in which activities are taking place. taking place.
HowHow to Evaluate: to Evaluate:
A Rule of Tumb: Keep Definitions Simple:A Rule of Tumb: Keep Definitions Simple: Activity: the process to deliver an output Activity: the process to deliver an output
(use made of an input or resources)(use made of an input or resources) Output: a product, usually a definable Output: a product, usually a definable
quantity (most easily monitored over time quantity (most easily monitored over time and space)and space)
Outcome: use made by the beneficiaries of Outcome: use made by the beneficiaries of an outputan output
Impact: the consequence of the outcome, Impact: the consequence of the outcome, or change triggered by the use of an or change triggered by the use of an output.output.
HowHow to Evaluate: to Evaluate:
Outcome MappingOutcome Mapping focuses in focuses in particular on changes in behaviour particular on changes in behaviour and relationships, and seeks to be and relationships, and seeks to be specific about changespecific about change
My preferred definition of outcomes My preferred definition of outcomes fits this concept, although in some fits this concept, although in some ways it is more specific even: the use ways it is more specific even: the use made of an output is a particular made of an output is a particular form of behavioural change.form of behavioural change.
HowHow to Evaluate: to Evaluate:
HowHow: Spheres of : Spheres of InfluenceInfluenceChange operates as a ripple of change around an Change operates as a ripple of change around an
actoractor
Objectives: Impact: general
population = sphere of indirect
influence
Results: Outcomes: level
of beneficiaries = sphere of direct
programme influence
Activities: Outputs: level of
implementing partners, = sphere of primary influence
HowHow to Evaluate: to Evaluate:
Change should ideally be mechanical like a spring. The Change should ideally be mechanical like a spring. The combination of different changes should be necessary combination of different changes should be necessary and sufficient to lead to a higher change. This calls for and sufficient to lead to a higher change. This calls for projects and programmes and sector-wide approaches projects and programmes and sector-wide approaches to be modest in the claims they make about impact.to be modest in the claims they make about impact.
ImpactImpact
OutcomeOutcome
OutputOutput
Activity Activity
Evaluation
HowHow to Evaluate: Plan & to Evaluate: Plan & FeedbackFeedback
Resources
Activity & Process (100% control)
Output (75% control)
Outcome (50% control)
Impact (25% control)
Detailed Operational Plan
Implementation
Results, Specific & General Objective
Improvements through M&E
HowHow to Evaluate: to Evaluate: Intentional DesignIntentional DesignThere are no set objectives, outcomes and indicators. These There are no set objectives, outcomes and indicators. These depend on depend on
the casethe case::For example: “Progress in adherence to the Rome Statute, a For example: “Progress in adherence to the Rome Statute, a
better public perception of the court’s role, and toward better public perception of the court’s role, and toward implementation of the Rome Statute into national implementation of the Rome Statute into national legislation”:legislation”:
Emergence of national lobby groups and coalitions to support Emergence of national lobby groups and coalitions to support ratification by a given state of the Statutes of international criminal ratification by a given state of the Statutes of international criminal courtscourts
Evidence of trust and active relations within a network of Evidence of trust and active relations within a network of professional experts in areas relevant to the Courts; professional experts in areas relevant to the Courts;
evidence of a strong community of purpose, in particular in the evidence of a strong community of purpose, in particular in the number of subscriptions to specialised newsletters, number of “hits” number of subscriptions to specialised newsletters, number of “hits” to a specialised website, participation in meetings, participation in to a specialised website, participation in meetings, participation in demonstrations.demonstrations.
Recurrence of key words in the national press, particularly the most Recurrence of key words in the national press, particularly the most popular media, and in specialised legal literature.popular media, and in specialised legal literature.
Judges and magistrates are selected in a public and transparent Judges and magistrates are selected in a public and transparent manner which is commented on in national communities.manner which is commented on in national communities.
WhatWhat to Evaluate: to Evaluate: Intentional Design Intentional Design
Set objectives, and detail changes which could be Set objectives, and detail changes which could be observed as a result of success (here based on F. observed as a result of success (here based on F. Oduro):Oduro):
Trust Trust restoredrestored
Forward Forward relationshrelationshipip
CooperatiCooperative ve endeavouendeavoursrs
BehaviourBehaviours and s and attitudesattitudes
StructuraStructural l transformtransformat°at°
Process Process (still (still outcome!)outcome!)
…… …… …… …… …… ……
WhatWhat to Evaluate: The to Evaluate: The Constraint of TransitionConstraint of TransitionThe The challengeschallenges of programmes in of programmes in situations of situations of
transitiontransition are unique and different from are unique and different from evaluations in other fields. This naturally evaluations in other fields. This naturally applies to TJ evaluations:applies to TJ evaluations:
The balance between reconciliation The balance between reconciliation and other objectives is rarely clearand other objectives is rarely clear
The constraint of The constraint of TransitionTransition
The balance between reconciliation The balance between reconciliation and other objectives is rarely clearand other objectives is rarely clear
Objectives as benchmarks is denied Objectives as benchmarks is denied by the fact that conflict by the fact that conflict programming requires above all a programming requires above all a response to opportunitiesresponse to opportunities
The constraint of The constraint of TransitionTransition
The balance between reconciliation and The balance between reconciliation and other objectives is rarely clearother objectives is rarely clear
Objectives as benchmarks is denied by Objectives as benchmarks is denied by the fact that conflict programming the fact that conflict programming requires above all a response to requires above all a response to opportunitiesopportunities
The justice and reconciliation objective The justice and reconciliation objective is not well articulated to the objectives is not well articulated to the objectives of a programme, often based on of a programme, often based on assumptions.assumptions.
The constraint of The constraint of TransitionTransition
The balance between reconciliation and The balance between reconciliation and other objectives is rarely clearother objectives is rarely clear
Objectives as benchmarks is denied by the Objectives as benchmarks is denied by the fact that conflict programming requires fact that conflict programming requires above all a response to opportunitiesabove all a response to opportunities
The justice and reconciliation objective is The justice and reconciliation objective is not well articulated to the objectives of a not well articulated to the objectives of a programme, often based on assumptions.programme, often based on assumptions.
Selecting indicators demands a strong Selecting indicators demands a strong understanding of the conflict and culture, understanding of the conflict and culture, and verification methods applicable to a and verification methods applicable to a conflict situation.conflict situation.
The constraint of The constraint of TransitionTransition
The balance between reconciliation and other The balance between reconciliation and other objectives is rarely clearobjectives is rarely clear
Objectives as benchmarks is denied by the fact that Objectives as benchmarks is denied by the fact that conflict programming requires above all a response conflict programming requires above all a response to opportunitiesto opportunities
The justice and reconciliation objective is not well The justice and reconciliation objective is not well articulated to the objectives of a programme, often articulated to the objectives of a programme, often based on assumptions.based on assumptions.
Selecting indicators demands a strong understanding Selecting indicators demands a strong understanding of the conflict and culture, and verification methods of the conflict and culture, and verification methods applicable to a conflict situation.applicable to a conflict situation.
Justice and reconciliation usually depends on the Justice and reconciliation usually depends on the identification of an end-state, an overall objective. identification of an end-state, an overall objective. The value given to this may not be shared by all The value given to this may not be shared by all stakeholders, even those within a single programme stakeholders, even those within a single programme or organisation.or organisation.
WhatWhat to Evaluate: to Evaluate: Opportunistic Design or Opportunistic Design or Evaluation Without Evaluation Without ObjectivesObjectives
Individual Individual Project Project OutcomesOutcomes
Analysis Analysis of of Societal Societal IssuesIssues
Testing Testing the the connectioconnection n outc./issuoutc./issueses
ReasonsReasons
…… …… …… ……
Outcome LevelOutcome Level: For Opportunistic : For Opportunistic Strategy or Sector-Wide ApproachesStrategy or Sector-Wide Approaches
E T
E
E
E
TT
T
Conflict Drivers Issue 1
Issue 2
Issue 3
Outcomes Outcomes Outcomes
Relevance x Extent x Duration
WhoWho Evaluates? Users Evaluates? Users and Rolesand Roles All evaluations of TJ I have done were All evaluations of TJ I have done were
commissioned by donors, and were not commissioned by donors, and were not well disseminated to the beneficiaries.well disseminated to the beneficiaries.
The link between general public The link between general public perception, research on one side, and perception, research on one side, and evaluation on the other, is very tenuous.evaluation on the other, is very tenuous.
Victims, support groups, civil society, Victims, support groups, civil society, need to intervene in what questions are need to intervene in what questions are asked at the outset, for evaluations to be asked at the outset, for evaluations to be relevant to them.relevant to them.
WhoWho Evaluates? Evaluates? Participatory EvaluationParticipatory EvaluationOnce stakeholders have been identified, Once stakeholders have been identified,
agenda, objectives and methodology agenda, objectives and methodology should be agreed with them. The skills are should be agreed with them. The skills are the ability to ask critical questions, and the ability to ask critical questions, and suspend personal judgement. The following suspend personal judgement. The following questions could guide preparation:questions could guide preparation:
What are the questions to be answered?What are the questions to be answered? What will the evaluation be used for?What will the evaluation be used for? Who must be heard and how to create the Who must be heard and how to create the
conditions for them to speak freely and conditions for them to speak freely and openly?openly?
Evaluation of Evaluation of Transitional JusticeTransitional Justice
When the programmes are closer to When the programmes are closer to a classical development programme a classical development programme I would argue that evaluation is not I would argue that evaluation is not very different from a general form.very different from a general form.
When on the other hand the When on the other hand the objectives are not clearly defined, it objectives are not clearly defined, it is best to approach evaluation by is best to approach evaluation by focusing on outcomes and on their focusing on outcomes and on their connection to the context and needs.connection to the context and needs.