Improving quality of humanitarian programmes through the use of a scoring system: the Humanitarian Indicator Tool

Dr Vivien Margaret Walden and Nigel Timmins

Page 2

Acknowledgements

• We wish to acknowledge the consultants, Andy Featherstone, Peta Sandison, Marlise Turnbull and Sarah House who helped refine the tools and the 10 countries offices who have been through the process and have hopefully come out of it unscathed.

• Photographs used in this presentation are by Caroline Gluck, Oxfam

Page 3

Why a new tool?

• In 1995, the ODI evaluation of the response to the Rwanda crisis raised concerns around the quality of service delivery

• As a result of findings, the Sphere project was started

• In 1997 the People in Aid Code of Best Practice was published

• In 2000 the Humanitarian Ombudsman became HAP

• In 2001 Griekspoor and Sondorp wrote a paper highlighting the various quality assurance measures and posed the question “Do all of the developments in the field of accountability and performance actually improve overall performance?”

Page 4

Why a new tool?

• Oxfam’s hypothesis – by improving the quality of a humanitarian programme, the likelihood of positive impact is increased

• The tool does not try to prove the link

• Oxfam has introduced the Global Performance Framework as part of reporting to DFID

• The framework has global indictors of which the Humanitarian Indicator is one

• Whereas the other indicators measure impact, the humanitarian indictor looks at quality of the response

Page 5

Oxfam’s Global Performance Framework

Livelihood Enhancement Support:# of women and men directly supported to increase income via enhancing production and/or market access

Humanitarian Assistance:Total number of people provided with appropriate humanitarian assistance, disaggregated by sex

Campaigning and Advocacy:#of campaign actions directly undertaken or supported, e.g. contacts made with policy targets, online and offline actions taken, media coverage, publications, etc.

Citizen Mobilisation:# of a) citizens, CBO members and CSO staff supported to engage with state institutions/other relevant actors; and b) duty bearers benefiting from capacity support

Adaptation and Risk Reduction:# of people supported to understand current and likely future hazards, reduce risk, and/or adapt to climatic changes and uncertainty, disaggregated by sex

Women’s Empowerment:# of people reached to enable women to gain increased control over factors affecting their own priorities and interests

% of supported households demonstrating greater income, as measured by daily consumption and expenditure per capita

Improved quality of life for poor

women and men

Degree to which selected interventions have contributed to affecting outcome change, as generated from findings of rigorous qualitative evaluations

Degree to which selected interventions have contributed to affecting outcome change, as generated from findings of rigorous qualitative evaluations

% of supported households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends & uncertainty

% of supported women demonstrating greater involvement in household decision-making and influencing affairs at the community level

Degree to which humanitarian responses meet recognised quality standards for humanitarian programming (e.g. Sphere guidelines)

Global Output Indicators Global Outcome Indicators

Degree to which selected interventions meet recognised standards for accountable programming

Extent to which selected project delivery good value for money

Page 6

Page 7

The indicator

• Output indicator – the total number of people provided with appropriate humanitarian assistance, disaggregated by sex

• Data collected annually through online system

• Outcome indicator – the degree to which humanitarian responses meet recognised quality standards for humanitarian programming

• Data collected through HIT

Page 8

The tool – for rapid onset emergencies

Global Humanitarian Indicator: Degree to which humanitarian responses meet recognised quality standards for humanitarian programming

RAPID ONSET EMERGNECY – EARTHQUAKE, SUDDEN FLOODS, TSUNAMI, CYCLONES, TYPHOONS, HURRICANES, SUDDEN CONFLICT WITH DISPLACEMENT, AWD OUTBREAKS Number Quality standard Met

(score6) Almost met (4)

Partially met (score 2)

Not met (score 0)

1 Timeliness - rapid appraisal/assessment enough to make decisions within 24 hours and initial implementation within three days

2 Coverage uses 25% of affected population as an planned figure (response should reflect the scale of the disaster) with clear justification for final count

3 Technical aspects of programme measured against Sphere standards

Number Quality standard Met

(score3) Almost met (score 2)

Partially met (score 1)

Not met (score 0)

4 MEAL strategy and plan in place and being implemented using appropriate indicators

5 Feedback/complaints system for affected population in place and functioning and documented evidence of information sharing, consultation and participation leading to a programme relevant to context and needs

6 Partner relationships defined, capacity assessed and partners fully engaged in all stages of programme cycle

7 Programme is considered a safe programme: action taken to avoid harm and programme considered conflict sensitive

8 Programme (including advocacy) addresses gender equity and specific concerns

Page 9

The benchmarks • Timeliness - rapid appraisal/assessment enough to make

decisions within 24 hours and initial implementation within three days

• Coverage uses 25% of affected population as an planned figure (response should reflect the scale of the disaster) with clear justification for final count

• Technical aspects of programme measured against Sphere standards

• MEAL strategy and plan in place and being implemented using appropriate indicators

• Feedback/complaints system for affected population in place and functioning and documented evidence of information sharing, consultation and participation leading to a programme relevant to context and needs

Page 10

The benchmarks • Partner relationships defined, capacity assessed and partners

fully engaged in all stages of programme cycle

• Programme is considered a safe programme: action taken to avoid harm and programme considered conflict sensitive

• Programme (including advocacy) addresses gender equity and specific concerns and needs of women, girls, men and boys and vulnerable groups

• Evidence that preparedness measures were in place and effectively actioned

• Programme has an advocacy/campaigns strategy and has incorporated advocacy into programme plans based on evidence from the field

Page 11

The benchmarks

• Country programme has an integrated approach including reducing and managing risk though existing longer-term development programmes and building resilience for the future

• Evidence of appropriate staff capacity to ensure quality programming

Page 12

The methodology

• Done by an external consultant – although preferably one who has a knowledge of Oxfam

• Done as a desk study using documentation and some telephone/Skype interviews

• Follows a pre-determined scoring system and list of documents

• Has to be commented on and accepted by the country

• The country writes a management response

• All reports and a summary for each are published on the Oxfam website – www. Oxfam.org

Page 13

The scoring

Quality standard Evidence needed Met (score 6) Almost met (4) Partially met (score 2) Not met (score 0)

3 Proposals MEAL strategy and plans PH and EFSL strategies Technical adviser visits Training agendas and presentations LogFrames and monitoring frameworks donor reports RTE and other evaluation reports learning event or review reports

Sphere standards proposed and put in place with adjusted standards for context Training in standards carried out for staff and partners Indicators use standards and monitoring against standards takes place regularly Standards evaluated

NA Sphere standards proposed and adjusted to context Standards mentioned in proposals and LogFrames but not monitored against Some evidence of training but not widespread (staff but not partners or only in one area)

Standards only mentioned in proposals but not replicated in plans Or No mention of Sphere in any document

Page 14

Instructions for use

Benchmark Evidence Quality check Benchmark 3 Technical aspects of

programme measured against Sphere standards

Proposals MEAL strategy and plans PH and EFSL strategies Technical adviser visits Training agendas and presentations LogFrames and monitoring frameworks donor reports RTE and other evaluation reports learning event or review reports

Check proposals and strategies to see if standards are mentioned not just as a possibility but that they are considered in the context of the response – this might mean that Sphere has been adapted to suit the context The indicators on the LogFrame for technical areas should reflect Sphere standards The MEAL strategy should have Sphere as indicators and for data collection methods Check adviser reports for mention of standards and how these were implemented Check the RTE report for mention of Sphere standards Check WASH and EFSL strategies and adviser reports to see if any training was carried out for staff and partners Check review and evaluation reports for mention of standards

Page 15

Final score

• First year total score could be 30

• Adjusted for non-applicable benchmarks

• First year scoring

• Somalia – 17/28

• Kenya – 24/30

• Ethiopia – 9/28

• Pakistan – 19/30

• Colombia – the test case for the HIT – 18/26

• Second year – only South Sudan is complete – 21/30

Page 16

The findings – Somalia

• Benchmark 5 on accountability

• Score Met – 2/2

• “Oxfam’s partners appeared to differ in the level of beneficiary participation in design and delivery. Some documented highly participatory process, with qualitative and quantitative data. As well as gathering information, rapid assessments were done to establish VRCs to improve participation (criteria, entitlements, payment points, registration, complaints and feedback). Mobile phone hotlines were set up where possible, with feedback protocol to guide staff on how to register and follow-up complaints”

Page 17

Pakistan

• Benchmark 7 – protection and gender

• Partially met – ½

• Some protection concerns were identified relating to security for staff and women and girls using WASH facilities. Some actions were taken responding to dignity and protection including involving women in different activities. Post-distribution monitoring investigated some security concerns related to cashing cheques and distribution points, including analysis of responses from women. No protection problems observed were communicated to agencies or authorities responsible for, or specialising in, protection

Page 18

Ethiopia

• Benchmark 11 – advocacy

• Not met – 0/2

• Advocacy activities were clearly part of the intended response and there is a regional advocacy action plan with Ethiopia objectives and a media, advocacy and campaign strategy which includes a number of plans for Ethiopia. However, no Ethiopia country advocacy strategy was provided for the evaluation. The sitreps do mention Oxfam’s participation in influential meetings, but are not tied to an explicit strategy. There is no record of the impact of Oxfam’s advocacy activities

Page 19

Lessons learnt

• There are limitations to doing a desk study – it relies heavily on documentation and scores do not reflect absence of documentation or actual absence of good programming

• Several country teams objected to being judged solely on “little pieces of paper”

• There is no opportunity to get the views of the affected population unless this is already documented

• Telephone/Skype interviews can be biased – it is sometimes difficult to triangulate

• The process needs the goodwill and buy-in from the country team

Page 20

Advantages

• The process is fairly inexpensive – under £6000

• The country does not have to host a consultant

• The methodology can be used to track progress in subsequent responses in one country

• The scores are comparable across programmes (although context should be considered)

Page 21

She deserves the best we can give – thank you!