Upload
daryl-colegrove
View
217
Download
1
Tags:
Embed Size (px)
Citation preview
Evaluating Humanitarian Aid:
November 2013Harry Shannon
McMaster University
Working with
• David Sanderson Director, Centre for Development and Emergency Practice (CENDEP), Oxford Brookes University
• Charles Parrack, CENDEP
Outline of talk
• What is (humanitarian) aid• Issues in evaluation of aid• Shelter after disaster• Proposed research agenda for evaluation• Discussion
Work in Progress!
Humanitarian imperative
• Helping fellow human beings in (desperate) need• “How we think about each other, our relationships,
our place in the ecosystem … are the most basic questions … We must recognise humility; trying and doing; governance … given them, don’t wait for the perfect answer.”– James Orbinski, former President of MSF
Maimonides: hierarchy of charitable giving
1. The lowest: Giving begrudgingly and making the recipient feel disgraced or embarrassed.
2. Giving cheerfully but giving too little.3. Giving cheerfully and adequately but only after being asked.4. Giving before being asked.5. Giving when you do not know who is the individual benefiting, but the
recipient knows your identity.6. Giving when you know who is the individual benefiting, but the recipient
does not know your identity.7. Giving when neither the donor nor the recipient is aware of the other's
identity.8. The Highest: Giving money, a loan, your time or whatever else it takes to
enable an individual to be self-reliant.
charitywatch.org
Definitions (though boundaries fuzzy)
• (Overseas Development) Aid: flows to developing countries and institutions whose aim is to promote development and welfare of the recipient countries.
• Humanitarian aid/action (HA): aid that follows a disaster (complex emergency), typically on a more urgent basis than regular ODA
Contrast with …
• Humanitarian intervention: the use of military force by a state to end human rights violations by a state (R2P)
Millennium Development Goals, 2000
By 2015:
1. Eradicating extreme poverty and hunger2. Achieving universal primary education3. Promoting gender equality and empowering women4. Reducing child mortality rates5. Improving maternal health6. Combating HIV/AIDS, malaria, and other diseases 7. Ensuring environmental sustainability8. Developing a global partnership for development
Millennium Development Goals
• Specific targets set, e.g., between 1990 and 2015, halve the proportion of people living on less than $1.25
• Ban Ki Moon: ‘The Millennium Development Goals (MDGs) have been the most successful global anti-poverty push in history.’
• However, progress has been uneven• Being replaced in 2015 by Sustainable Development
Goals
Development aid
• $140 billion / year• Dambisa Moyo, William Easterly - sceptics• Jeffrey Sachs – proponent• Paul Collier - ?intermediate• Esther Duflo, Abhijit Banerjee – RCTs
(Randomized Controlled Trials)
• All support humanitarian aid (HA)
Duncan Green is Senior Strategic Adviser at Oxfam GB
Duncan Green, From Poverty to Power
Sceptical view of Canada’s NGOs
Humanitarian aid ‘system’ - some numbers
• Estimated 4,400 NGOs worldwide doing some HA(Haiti sometimes called a ‘Republic of NGOs’)
• 2010: estimated 274,000 humanitarian workers worldwide
• Approx $14 billion / year
Humanitarian aid ‘system’
• UN humanitarian agencies• International Movement of the Red Cross and
Red Crescent• Five largest International NGOs
Médecins Sans Frontières (MSF) Catholic Relief Services (CRS) Oxfam International International Save the Children alliance World Vision International
• Increasing attention to ‘accountability’
Cluster approach – 2005 Inter-Agency Standing Committee (IASC)
CODE OF CONDUCT for the International Red Cross and Red Crescent Movement
and NGOs doing humanitarian aid activities
1) The humanitarian imperative comes first. 2) Aid is given regardless of the race, creed or nationality of the
recipients and without adverse distinction of any kind. Aid priorities are calculated on the basis of need alone.
3) Aid will not be used to further a particular political or religious standpoint. (Independence)
4) We shall endeavour not to act as instruments of government foreign policy. (Neutrality)
5) We shall respect culture and custom. 6) We shall attempt to build disaster response on local
capacities.
7) Ways shall be found to involve programme beneficiaries in the management of relief programmes.
8) Relief aid must strive to reduce future vulnerabilities to disaster as well as meeting basic needs.
9) We hold ourselves accountable to both those we seek to assist and those from whom we accept resources.
10) In our information, publicity and advertising activities, we shall recognise disaster victims as dignified human beings, not hopeless objects.
Emphases added
CODE OF CONDUCT for the International Red Cross and Red Crescent
Movement and NGOs doing humanitarian aid activities
What should humanitarian groups aspire to?
Some ethical dilemmas
• Do we respond in Japan, where it’s expensive, or in the Congo where we can do more for less?
• If we work with, e.g., the Syrian government, do we risk being co-opted into their strategy?
• If we bring food, do we drive down market prices and upset local agriculture?
• If we give out food at distribution points, do we put people at risk of attack when they walk home?
Ctd …
• What is the balance between paternalism and autonomy?
• Do we speak out about terrible things (only) we see, or stay quiet, do our work and save lives, and let others speak out?
• Relations with military?
These decisions are made under (severe) time pressure, and possibly armed force and persuasion/coercion.
Some ethical dilemmas (ctd)
Source: Hugo Slim
The Guardian, 28 August 2013
On 14 August, Médecins Sans Frontières (MSF) announced that, after 22 years, it was closing all its activities in Somalia as “the situation in the country has created an untenable imbalance between the risks and compromises our staff must make, and our ability to provide assistance to the Somali people.”
…
Of the 160 aid workers killed, 82 wounded and 99 kidnapped in Somalia since 1997, the majority have taken place since 2007 (106 killed, 53 wounded and 57 kidnapped), when the overall security environment deteriorated for both humanitarians and Somali civilians.
The Guardian, 28 August 2013
Report on aid worker securityOctober 2013
Some practical concerns with HA
• Little argument about the principle• Questions about how well HA is provided• Speed of response• Coverage of those in need• Duplication• Unnecessary help• Doing more harm than good (cholera in Haiti)
Critique of humanitarian aid
• Use of HA for political purposes has been going on for a long time
• British soldiers in Afghanistan – giving food in exchange for information on Taliban hiding places
• Colin Powell to NGOs: You are our force multipliers• In Darfur, aid workers have provided actionable
information for ICC• Secret services infiltrate NGOs• Embassies keep an eye on what ‘their’ NGOs are
doingSource: Antonio Donini
Other possible problems with aid
• NGOs attract talent from local governments• Effect of internationals on local prices• NGOs supplant government roles• Plethora of NGOs; potential competition• Government corruption• Qualifications to provide help• (Non) Involvement in politics• Promised money may not arrive
Port-au-Prince, October 2011
Humanitarian/development aid or conscience money?
Amira Hass in Israeli newspaper Haaretz, 6 Feb 2013
7 Golden Rules of Recovery Management1. Trust survivors and avoid paternalism2. Trust survivors and enable survivors to assess
their own needs3. Trust survivors and provide cash rather than kind4. Think locally5. Give priority attention to vulnerable groups6. Think ‘process’ not product, ‘sheltering’ not
shelters, ‘housing’ not houses7. Adopt a long-term perspective while addressing
short-term needsIan Davis + Anshu Sharma, 14 Nov 2013
Toby Ord: “Aid Works (on Average)”Talk in Oxford, 13 February 2013
• Cited Moyo: – Over 60 years, more than US$1 trillion in aid and not much good
to show for it– $1 trillion is nearly $1,000 per person on the planet today
• Ord:– Nonsense, actually 1 billion people in Africa– $1 tn / 440 mn (average number of people in Sub-Saharan Africa
over period)= $2,263 per person = $38 per person per year
Toby Ord: “Aid Works (on Average)” (ctd)
Successes include:Smallpox eradication Lives saved since eradication:
60-120 millionOral rehydration therapy for diarrheal diseases
Annual deaths down from 4.6 million to 1.6 million
Immunization Measles annual deaths down from 5 million to 1.2 million
Malaria Annual deaths down from 3.8 million to 0.7 million
Why evaluating ‘aid’ is difficult
• Reason aid is given• Reverse causation• Non-developmental aid • How/where aid is actually spent• Fungibility• Multiple donors• Remittances• Unpredictability of aid• Short- vs. long-term impact of projects
Aid or …?
The Guardian, 21 February 2013
Critique of methods in evaluating aidStuckler, McKee, Basu 2012
• Reviews are narrative not systematic• Evaluations don’t consider bias, confounding• Multiple outcome measures• Time lags• Data limitations• Power, Type II errors
They advocate:• Experimentation• Evaluation• Replication
• Process evaluation: how the intervention was done• Impact evaluation: longer-term, intended or
unintended • Indicator – measure of the effect
Evaluating aid - Definitions
Experiments
• Small-scale RCTs• Examples:– paying parents if child
attends school regularly
– giving food when children vaccinated
– microcredit
Increasing attention to accountability
• ALNAP - Active Learning Network for Accountability and Performance in humanitarian action
• Jim Kim, President of World Bank: “Science of delivery [of aid]”
• Various documents produced by different NGOs on how to evaluate
Evaluating the humanitarian system
ALNAP• Collates evaluations and produces summary
reports• State of the Humanitarian System:
‘assesses progress and performance in six areas: coverage/sufficiency, relevance/appropriateness, effectiveness, connectedness, efficiency and coherence.’
• Interactive online evaluation training materials
Difficulties in evaluating HA impact
• No baseline / control (or poor data)• Need for rapid information• Insecure / volatile environments• Surveys in settings with little information on
the target population• Impact of single intervention vs broader
impact of programme• Lack of capacity
Qualitative and quantitative methods
• “… much of the guidance for the evaluation of humanitarian action suggests that evaluations are more likely to provide robust evidence where they use ‘mixed methods’ approaches.”
• “However, in practice, humanitarian evaluation ‘uses mainly qualitative methods’ (Buchanan-Smith and Cosgrave 2012).”
• “… evaluations still tend to undervalue the experience of affected populations as a source of evidence …”
James Darcy and Paul Knox Clarke, 2013
Focus on Shelter
• Seen as human right• One of Clusters – incorporates settlement and non-
food items• Housing used for purposes other than purely shelter• Relatively little on shelter evaluation• ‘Providing adequate shelter is one of the most
intractable problems in international humanitarian response’ ‘(Humanitarian Emergency Response Review, 2011)
• ‘Building Back Better’
Housing and health
Shaw, Ann Rev Pub Health, 2004
Right to adequate housing
• Article 25 of Universal Declaration of Human Rights
• Adequacy includes– legal security of tenure– availability of services– affordability– habitability– accessibility– location– cultural adequacy
UNHCR, 1991
Section in Sphere Handbook
Proposed timing of shelter after disaster
Emergency shelter
Temporary / transitional shelter
Permanent shelter / reconstruction
0-2 weeks
2 years +
2 weeks – 2 years
Charles Parrack
Relief
Recovery
Reconstruction
Shelter damage in Port-au-Prince after 2010 Haiti earthquake
Damage to housing % 95% CI
Completely destroyed 24.4 22.1-26.9
Partly damaged 41.5 38.9-44.1
No visible damage 34.1 31.7-36.6
Kolbe, Hutson, Shannon et al., Medicine, Conflict and Survival, 2010
Shelter for displaced - refugees
Proposed agenda for evaluating shelter after disaster
• Step 1a: Systematically identify indicators• Step 1b: Improve list by consulting past and
potential beneficiaries• Step 2: Develop (short) questionnaire• Step 3: Conduct longitudinal study (RCT?)
after disaster
Housing trade-off: durability vs comfort
© David Sanderson; with permission
Step 3: Conduct longitudinal study/RCT after disaster
• Timing: – start 1-2 weeks after disaster – follow up at 6 months, 1 year and 2 years
• Allocate areas/communities/villages to different approaches to housing, i.e., cluster allocation/randomization
• E.g., ‘T-shelters’ vs. housing in camps vs. building materials and tools
• Mixed methods – quantitative and qualitative
Feasibility of RCTs?
Example of randomization at the individual level
Technical problems with RCTs include the need to deliver the treatment in a standardised way, the difficulties of maintaining stable membership in the treatment and control groups, and the expense and effort needed. These constraints led a review on evaluations of health promotion by the World Health Organization in Europe to find that the use of RCTs for this was ‘in most cases, inappropriate, misleading and unnecessarily expensive’ (WHO Regional Office for Europe, 1998, p. 5). RCTs also require that the evaluation be designed from the start of the project with random selection of control and treatment groups. For all of these reasons, RCTs are rare in humanitarian evaluation.
(emphasis added)
ALNAP EHA Pilot Guide p.113
Example of randomization at the community (cluster) level
Discussion