View
21
Download
0
Category
Tags:
Preview:
DESCRIPTION
Four Rules of the Art AND SOME POKING AROUND FOR PRACTICAL SOLUTIONS TO THE CHALLENGES OF JOINT PEACE INTERVENTION EVALUATIONS. Emery Brusset NUPI/IPI Workshop, New York 7 & 8 May 2009. Four Theoretical Challenges to the evaluation of conflict interventions. 1. Seizing Opportunities - PowerPoint PPT Presentation
Citation preview
Four Rules of the Art
AND SOME POKING AROUND FOR PRACTICAL SOLUTIONS TO THE CHALLENGES OF
JOINT PEACE INTERVENTION EVALUATIONS
Emery BrussetNUPI/IPI Workshop, New York
7 & 8 May 2009
Channel Research 2
Four Theoretical Challenges to the evaluation of conflict interventions
1. Seizing Opportunities
• Programming requires a response to constraints and opportunities as they arise.
• These cannot be planned for in advance and plans are not a good evaluation reference
1. Seizing Opportunities
• Programming requires a response to constraints and opportunities as they arise.
• These cannot be planned for in advance and plans are not a good evaluation reference
2. Attribution
• It is particularly difficult to separate the impact of peace-building projects from other impacts on conflict
• The time required to cover other interventions is limited
2. Attribution
• It is particularly difficult to separate the impact of peace-building projects from other impacts on conflict
• The time required to cover other interventions is limited
3. Data Quandaries
• Selecting evidence requires prioritisation and demands a good understanding of the nature of the conflict
• The verification methods must be applicable in a conflict situation
3. Data Quandaries
• Selecting evidence requires prioritisation and demands a good understanding of the nature of the conflict
• The verification methods must be applicable in a conflict situation
4. End State?
• Problems in defining terms such as peace and conflict.
• Usually depends on the identification of an end-state.
• This may not be explicit or shared by all stakeholders or those within a single programme
4. End State?
• Problems in defining terms such as peace and conflict.
• Usually depends on the identification of an end-state.
• This may not be explicit or shared by all stakeholders or those within a single programme
Options
Channel Research 3
The Practical Challenge of Joint Evaluation: Evaluation needs to provide a convincing set of conclusions to a growing diversity and size of audience
“Ideal single actor evaluation” • TOR enable the
evaluator to identify unplanned and unintended effects
• A way is found to identify comparative weight of intervention
• Good conflict analysis is possible
• The nature of “peace” is agreed
“Reality of Joint Evaluation”
• Expectations evolve
• The questions focus on the management of interventions
• There is no strategic baseline or agreed assessment
• Peace is defined generically, or not at all
4 Rules of Good Evaluation:
1. Independence of team to focus on situation as it emerged
2.Team has means to analyse societal interface
3.Team is capable of participatory assessments
4.Conflict objectives are reconstructed
toFrom
Channel Research 4
• Challenge 1: Capturing unintended and unexpected aspects
Rule 1: The team is enabled to define what issues to focus on, and allows for Steering Committee consultation milestones to secure alignment of all stakeholders.
Assess
Adapt Act
Ongoing Analysis Rolling Assessment
Intensive ManagementOngoing Decision Making and
Documentation
RISK
Challenge 2: Evaluation questions tend to focus on institutional issues
Evaluation, particularly real time evaluation, tends to be linked to decisions and the need to understand and support the management of an intervention. This risks reducing the time available to analyse the impact .
Conflict Analysis Channel Research 6
Rule 2: Evaluations are given time to enable an iterative but systematic process (six months to a year?) – scope is contained. There is agreement that attribution of impact is replaced with contribution.
Threat
Threat
Threat
Threat
Threat
Threat
?$$
Success depends on sufficient resources
Challenge N° 3: Get the readers and audience to agree on a consistent analysis of the conflict, and on key drivers of peace
Map of situation
Intervention footprint
Situation withintervention factored in –
how does it affectsituation, how doessituation affect it?
Channel Research
Channel Research 8
Participatory Mapping of Immediate Conflict Environments (MICE) works by looking at linkages between key trends and events = drivers
Event: occurs ata point in time
Trend: changein a condition orcharacteristic
Arrow of change: increased probability orintensity
The elements of conflict mapping
GrowingUnemployment
Coup
Corruption more apparent
Officers inc. disgruntled
Battle in whichwar was lost
Growing leftistradicalisation
Dissentincreasinglysuppressed
Many insurgent Attacks in XX
Hypothetical and partial example: Country withmilitary regime in conflict with leftist insurgents
Channel Research 9
The aim is the identification of key issues that need to be addressed to mitigate conflict or instability
Growing unemployment
Coup
Corruption more apparent
Officers inc.disgruntled
Battle in whichwar was lost
Growing leftistradicalisation
Dissent increasinglysuppressed
Many insurgent Attacks in XX5
4
4
4
To identify priority issues, we count the linkages leading toand from relevant events and trends – those with the mostlinkages have the most reverberation in the situation
Priority issues:
1. Leftist radicalisation
2. Militarism, state repression, military dictatorship
CAVEAT
Conflict mapping is not root cause or systems analysis. Although corruption and unemployment might be major causes of leftist radicalisation, we do not identify them as priority issues – ranking is not intuitive at all
Channel Research 10
Representation is qualitative and culturally neutral. For example…
Militarycoup Leftist
radicalisation
Leftistuprising
MilitaryPoliticisation increases
Militaryrepression
Labouractivism
More Poverty
Peasantrevolt
Leftist – peasantcooperation
US companyoil contract
Leftistperception of
collusion
Companyunpopularityon ground
Rebel threatsto company
Strikes amongproject staff
Increasingemployment
Increasingstate revenues
Military regimecorruption
Human rightsNGO scrutiny
Shareholderscrutiny
US gov’tcriticism of
military regime
Pressure fordemocratisation
Military steps upproject security
Generalstrikes
Oil productionincreases
_
_
_
Channel Research 11
An example from a real mapping exercise in a post-conflict African country
Channel Research 12
Mapping of Cross-Border Issues in the Conflict in NIWorkshop 1 Glencree Centre for Reconciliation
Workshop 2 Monaghan
Workshop 3 Derry
Workshop 4 East Antrim
Workshop 5Letterkenny
Unionists feel under threat Republic’s population losing interest in Northern Ireland
Belfast/Good Friday Agreement
Creation of cross-border bodiesBritain and Ireland joining the EEC
Launch of the peace process in 19921970s further increase in violence
Partition (1920/21) Fear emerged (late 1960s/early 1970s)
Border region becomes established Belfast/Good Friday
Agreement1994 IRA ceasefire and negotiations /peace process
Crushing of civil rights movementIncreased disengagement by Irish Government
Abandonment of the border regionNorthern Ireland seen as a foreign country
Protestant / Unionist security and law enforcement
Civil rights movement/ marches (1960s)
B Specials introduced
Increasing involvement of people in paramilitary activity (especially hunger strikes (1981)
Violent British government response (late 1960s)
Closure of border roads (1970s)
Establishing checkpoints
Increasing segregation of housing – late 1960s/ early 1970s
Unionism increasingly associated with Loyalism and negative images
Working class ‘cut off’, paramilitaries
DUP and UUP responding to availability of peace money, e.g. Gatekeeping of the funding
Nationalist movement portrayed as freedom fighters
Belfast/Good Friday Agreement
92 miles of border in Donegal
Increased intimidation of businesses in Donegal
Maturity in cross-community relations
Both countries have run away from EU commitments on tariffs and trade agreements
Celtic Tiger exceptional performance of the economy
Channel Research 13
Challenge N° 4: objectives and their connection to peace are unclear or outdated
Project summary Verifiable indicators Means of verification Important assumptions
Gen. Objective/impact: Overall goal which this project will help to achieve
The evidence (quantitative / qualitative) which will be used to measure / judge the achievement of goal
Sources of information / data which will be used to assess the indicator(s)
(Goal to super goal)Main external factors necessary to sustain objectives in the long run
Purpose/Specific Objective/outcome: Immediate impact on the project area or target group, that is, the change or benefit to be achieved by the project
The evidence (quantitative / qualitative) which will be used to measure / judge the achievement of the purpose
Sources of information / data which will be used to assess the indicator(s)
(Purpose to goal) Main external factors necessary if project purpose is to contribute to reaching project goal
Outputs: The specific, deliverable results expected from the project to attain the purpose
The evidence (quantitative / qualitative) which will be used to measure / judge the achievement of the outputs
Sources of information / data which will be used to assess the indicator(s)
(Outputs to purpose) Main external factors necessary for outputs to achieve project purpose
Complexity in PLANNING is not well captured by prevailing tools
“Project”
Channel Research
LogFrame
“Strategy”
???
Rule 3: allow the team to innovate and interpret in mapping objectives and their meaning in terms of peace
“Project”
Channel Research
LogFrame
“Strategy”
“Effect Diagram”
Channel Research 17
The intervention tree in its simplest form – it is an adapted logic or decision tree
We need to achieve this, and to achieve this we need to have obtained…
… this, and to see this, we need to do…
… this, and to see this, we need to do…
… this, and to see this, we need to do…
This
This
This
This
This
This
This
This
This
Each new branch needs to be a separate and complete initiative to the extent possible, toensure no redundancy, and to spot any gaps
Poverty reduced
Alternative rural livelihoods created
Equitable access to productive assets, markets and services (e.g. microfinance) established
EC activities Outputs / Outcomes Impacts
Rural economy strengthened
Poppy production reduced or stabilizedState of law and order
improved (incl. drug enforcement
Sustainable management of natural resources achieved / improved
Vulnerability of rural population decreased
Rural production and gainful employment strengthened
More effective provincial / community based institutions developed
Governmental / admin. structure strengthened
Civil Service System improved (including police)
Governmental physical infrastructure re-created
Gov / ATA budget stabilized
National payment system created
Revenue position of Gov strengthened
Reform of civil service promoted
Access to BHCP improved / increased
Health sector management on central and provincial level improved
Positive tendency in health status of population (especially U5 and maternal mortality)
Rural Developm
entPublic Adm
inistration Reform
Foundation for public administration reform created (NOTE: in 2nd NIP, increased focus on sub-national level)
(Participation of) rural communities in dev. Strengthened
Health
Example: EC Effect Diagram Afghanistan
Channel Research 19
Actions OutcomesOutputs Global Impact
Conflicts in partner countries prevented
Regional co-operation and integration strengthened
Support measures for regional integration mechanisms and trade co-operation implemented
Direct Impact
CS
P used as central instrum
ent for m
ainstreaming conflict prevention
Trade links between countries strengthened.
Macro-economic environment in at risk countries stabilized.
Equal participation of men & women in social, economic & political life promoted (14).
1) Elements of Structural Stability: sustainable economic development, democracy & respect for Human Rights, viable political structures, healthy environmental & social conditions, capacity to manage change without resorting to conflict.
Potential for more favourable democratic environment increased(14).
Rule of Law, Good Governance & Civil Society strengthened & (13)
Security sector prepared for post-conflict situation
Peace in post-conflict countries consolidated (16).
Potential for renewed conflicts in post-conflict countries reduced (16).
Stru
ctu
ral S
tab
ility im
pro
ved
1)
Support to security sector increased (I.e. police training, HR training) (14).
Rehabilitation and reconciliation programmes implemented (16).
Targeted actions implemented to support electoral process, parliamentary activities, administration of justice (14).
Macro-economic assistance operations implemented (12).
Integ
rated A
pp
roach
Ow
nership of process by partner country established
Cross-cutting issues efficiently addressed in EC activities
Mechanisms developed to appropriately address cross-cutting issues in relevant areas:- social conditions (drugs, small
arms, Health / communicable diseases (HIV / Aids), Population flows / trafficking)
- environment & natural resources (water)
- economy (Private sector in unstable areas)
Early-warning mechanism & procedures created
EU decision-making & operational centres alerted to imminent crisis
EC enabled to formulate appropriate response to developing crisis situation (incl. preventive sanctions)
information on cause & potential consequences of conflict provided
EC enabled to react quickly to developing crisis situation
Rapid Reaction Mechanism (RRM) introduced EC enabled to carry out
wide-ranging short term missions on short noticeCo-ordination of Member States on
common political line intensified (incl. sanctions)(23).
More focused, time-flexible & robust political dialogue & diplomacy enabled (23)
Degeneration of political difficulties / tension into armed conflict prevented
Traditional community instruments optimised to respond to conflict (including ESDP)
Reform of aid management allows more responsiveness to needs in line with the performance of partner countries
Coherence of approaches for conflict prevention improved among partners (countries, international organisations, NGOs)
Dialogue with partners intensified & focused on conflict prevention (incl. NGOs)(25)
Common early warning systems, monitoring procedures & conflict indicators developedAppropriate mechanisms &
procedures for coordination with international partners developed.
Cooperation with international partners intensified
Tensions between countries reduced
Poverty reduced (14).
Economic growth stimulated
Societal post-conflict tension reduced
Potential for new conflicts in at at-risk countries reduced.
Potential for conflicts between partner countries reduced
Analysis of Effect Diagrams
What is their purpose?• Visualize hierarchy of objectives / expected effects• Reveal logical links, help to define certain Evaluation Questions
& assess internal coherence of strategy
What are their weaknesses?• Do not capture changes over time or feedback loops• “Other actor” / “Implementation” / “Context” not captured
Conclusion?• For complex interventions: can be used for sub-diagrams• Shows missing elements • Does not provide the final leap from objective to situation
change
Channel Research
The diagram can be turned into a narrative: theories of change
• Theories of change can take the simple format: “We believe that by doing X (action) successfully, we will produce Y (movement towards peace).”
• Or the chain of logic might involve multiple steps: “if we do X, that will lead to Y, which will encourage Z, which will exert an influence on key actor A.”
• In order to create, uncover or test a theory of change, it is essential to conduct a comprehensive analysis of the conflict, identify the key factors of conflict and the relationship between them, and spell out how the intervention (project, programme or policy) will create change.
• Making theories of change explicit is a central part of planning and programme design and can facilitate evaluation.
Channel Research 21
List of Identifiable Theories of Change:
• The Individual Change Theory: Peace comes through transformative change of a critical mass of individuals, their consciousness, attitudes, behaviors and skills. [Methods: investment in individual change through training, personal transformation/consciousness-raising workshops or processes; dialogues and encounter groups; trauma healing.]
• The Healthy Relationships and Connections Theory: Peace emerges out of a process of
breaking down isolation, polarization, division, prejudice and stereotypes between/among groups. Strong relationships are a necessary ingredient for peacebuilding. [Methods: processes of inter-group dialogue; networking; relationship-building processes; joint efforts and practical programs on substantive problems.]
• The Withdrawal of the Resources for War Theory: Wars require vast amounts of material
(weapons, supplies, transport, etc.) and human capital. If we can interrupt the supply of people and goods to the war making system, it will collapse and peace will break out. [Methods: anti-war campaigns to cut off funds/national budgets; conscientious objection and/or resistance to military service; international arms control; arms (and other) embargoes and boycotts.]
Channel Research 22
• The Reduction of Violence Theory: Peace will result as we reduce the levels of violence perpetrated by combatants or their representatives. [Methods: cease-fires, creation of zones of peace, withdrawal/retreat from direct engagement, introduction of peacekeeping forces/interposition, observation missions, accompaniment efforts, promotion of nonviolent methods for achieving political/social/economic ends.]
• The Root Causes/Justice Theory: We can achieve peace by addressing the underlying issues of injustice, oppression/exploitation, threats to identity and security, and peoples’ sense of injury/victimization. [Methods: long-term campaigns for social and structural change, truth and reconciliation; changes in social institutions, laws, regulations, and economic systems.]
• The Institutional Development Theory: Peace is secured by establishing stable/reliable social
institutions that guarantee democracy, equity, justice, and fair allocation of resources. [Methods: new constitutional and governance arrangements/entities; development of human rights, rule of law, anti-corruption; establishment of democratic/equitable economic structures; economic development; democratization.]
• The Political Elites Theory: Peace comes when it is in the interest of political (and other) leaders
to take the necessary steps. Peacebuilding efforts must change the political calculus of key leaders and groups. [Methods: raise the costs and reduce the benefits for political elites of continuing war and increase the incentives for peace; engage active and influential constituencies in favor of peace; withdraw international support/funding for warring parties.]
Channel Research 23
Channel Research 24
• The Grassroots Mobilization Theory: “When the people lead, the leaders will follow.” If we mobilize enough opposition to war, political leaders will have to pay attention. [Methods: mobilize grassroots groups to either oppose war or to advocate for positive action. Use of the media; nonviolent direct action campaigns; education/mobilization effort; organizing advocacy groups; dramatic events to raise consciousness.]
• The Economics Theory: As a politician once said, “It’s the economy, stupid!” People
make personal decisions, and decision-makers make policy decisions based on a system of rewards and incentives and punishment/sanctions that are essentially economic in nature. If we can change the economies associated with war-making, we can bring peace [Methods: use of government or financial institutions to change supply and demand dynamics; control incentive and reward systems; boycotts.]
• The Public Attitudes Theory: War and violence are partly motivated by prejudice,
mispercep tions, and intolerance of difference. We can promote peace by using the media (television and radio) to change public attitudes and build greater tolerance in society. [Methods: TV and radio programs that promote tolerance; modeling tolerant behavior; symbolic acts of solidarity/unity; dialogues among groups in conflict—with subsequent publicity.]
25OTI HQ
mission
OTI+I.P.
StateNGO
StateNGO
StateNGO
StateNGO
StateNGO
StateNGO
MoH Police CommunityLeaders
Families Banks PCs
Outcome Mapping: Focusing on actors and behaviour
Channel Research 26
Challenge N°5: Bringing it all back together
E T
E
E
E
TT
T
Linkage mapping Issue 1
Issue 2
Issue 3
Outcomes
Outcomes
Outcomes
Outcomes
Relevance
x
Extent
x
Duration
Focus of evaluationThe Significance of Impact methodology has been designed to provide solid evidence on the contribution which an actor makes to a particular issue
Issues identified with the mapping tools can provide the starting point
Priority issues
a. Poor relations with local community
b. Rivalry between central and provincial administration
c. Criminal economy
d. Ethnic cleavages lead to growing fear of threats
Extent and Duration
a. What proportion of the target population is covered?
b. Were the key people accessed?
c. How long did the influence last?
d. Can the influence be sustained after the ending of the intervention?
Duration
Extent
a
b
c
d
SIGNIFIC
ANCE
Priority risks
Leading to three dimensional assessments…
Priority issues
a. Poor relations with local community
b. Rivalry between central and provincial administration
c. Criminal economy
d. Ethnic cleavages lead to growing fear of threats
Extent and Duration
a. What proportion of the target population is covered?
b. Were the key people accessed?
c. How long did the influence last?
d. Can the influence be sustained after the ending of the intervention?
Relevance:
- What do others do?
- Are there other options?
- Could a clearer objective have been designed?
- Does the intervention correspond to the need assessed?
Degreeof
relevance
Extent and Duration
Extent
Duration
Significance
D
C
B
A
D. . . C. . .
B. . .
A. . .
The tools is designed to highlight what has been done and test its connection to the issues which are recognised as issues to be addressed to achieve ‘progress’
Channel Research 29
All levels of the result chain can yield important results or outcomes: demobilisation
Activities and process deliver outputOutputs are defined and easily verified
Example: “a workshop is held”
First levelOperational level
Specific objectives are the results of activities and ideally are how outputs are to be used by the beneficiary groups – user interface
Example: “skills and tools are taught”
Second levelProgramme
level
General objectives are the statement of overall verifiable change intended
Example: “Armed groups are able to design a new force concept”
Third Strategy Level
Society & State
Process can be the output
The issue can be that groups have lost contact and need to build mutual respect
Example: “Familiarity is achieved”.
New capacities can lead to changes in the command structure and the reintegration of groups.
Example: “Specialisation and opportunities are required for conversion to civilian life”
Changes in the overall environment can lead to a better notion of security based on smaller forces
Example: “Security imperatives require larger armed groups”.
Strategy Outcome
The Table of Contents of the Evaluation Report!
1. Issues / Entry Points have been identified
2.Outcomes achieved have been identified
3.Assessment of the value of the contribution is possible:• Relevance and efficiency• Effectiveness • Sustainability• Coherence with other interventions• Overall conclusions on « posture »• Recommendations
Channel Research 30
Recommended