Evaluation of the Bangladesh Rohingya Response Phase II

Terms of Reference – June 2020

Page | 1

Table of Contents

Introduction ..................................................................................................................... 2 Background ...............................................................................................................................................2 AHP Activation ..........................................................................................................................................2 Project Overview .......................................................................................................................................2 AHP Bangladesh Response - Phase Three ................................................................................................4 COVID-19 ..................................................................................................................................................4

Scope................................................................................................................................ 5

Methodology .................................................................................................................... 5

Key Evaluation Questions.................................................................................................. 7

Evaluation Steering Committee ......................................................................................... 8

Evaluation Team ............................................................................................................... 8 NGO involvement on the team .................................................................................................................9

Evaluation Utilisation ........................................................................................................ 9

Key Documents ................................................................................................................. 9

Evaluation Timeline ......................................................................................................... 10

Team Leader Selection Criteria ......................................................................................... 11 Required skills, qualifications and experience ....................................................................................... 11 Desirable skills, qualifications and experience ...................................................................................... 11

Annex 1 – DFAT Monitoring and Evaluation Standards – Independent Evaluations ........... 13

Annex 2 – Evaluation Rubric used for the Mid-Term Evaluation of the AHP Building Peaceful Futures Program Iraq, 2020 ................................................................................ 24

Page | 2

Introduction

The Australian Humanitarian Partnership (AHP) Rohingya Humanitarian Response Phase II is a one year response funded by the Department of Foreign Affairs and Trade (DFAT) as part of DFAT’s broader Bangladesh Humanitarian Assistance Package to address the ongoing humanitarian needs of displaced Rohingya and host communities. The response is implemented by Plan International Australia, World Vision Australia, and Save the Children in consortium with Oxfam and CARE Australia.

DFAT and the AHP NGO partners are conducting an independently-led joint-evaluation of the Rohingya Humanitarian Response Phase II as the response is nearing completion. As per the AHP Evaluation Policy, an independently-led evaluation is required for any activation over $3m and evaluations are undertaken as a joint exercise between DFAT and the AHP NGOs to support learning and uptake of the recommendations by AHP partners and to increase the accountability and transparency of the response.

The evaluation will assess the effectiveness, efficiency and relevance of the response. It will also assess the extent to which the response was inclusive of and met the needs of those most vulnerable, was accountable to the affected populations and how well a localisation approach, appropriate to the context, was implemented. In addition to evaluating the Phase II response, the evaluation will also identify early lessons from the implementing partners’ experience in responding to COVID-19, as funded from both the Bangladesh Phase II and Phase III packages. The evaluation is intended to provide both an important accountability mechanism to DFAT, as well as to support learning by both DFAT and the AHP NGOs, and other stakeholders in the humanitarian sector. The evaluation will also provide recommendations and background analysis to inform future programming, specifically the AHP Bangladesh Phase III Multi-Year package due to begin implementation in 2020, and also any future responses in regards to COVID-19.

Background

AHP Activation

The three NGO projects were selected for funding through the Australian Humanitarian Partnership (AHP).

The AHP is a partnership between the Australian Government and six pre-selected Australian NGOs - CARE, Caritas leading the CAN DO consortium of faith-based organisations, Oxfam, Plan International, Save the Children and World Vision. The AHP aims to save lives, alleviate human suffering and enhance dignity during and in the aftermath of conflict, disasters and other humanitarian crises by harnessing the networks and access of Australian NGOs. An external AHP Support Unit provides services to all partners, including assisting with, and overseeing, monitoring, evaluation and learning.

Project Overview

In late 2017 the Department of Foreign Affairs and Trade (DFAT) provided AUD 6 million to the Australian Humanitarian Partnership (AHP) to fund NGO activities to help address the ongoing humanitarian crises in Bangladesh in Rohingya refugee and host communities. Save the Children Australia and Oxfam Australia (in partnership with CARE) were selected as implementing partners.

Page | 3

Their work was evaluated in October and November of 2018 (see the following link for the full report: AHP Response: Rohingya Refugee Crisis Evaluation).

In 2019 DFAT announced an additional $10M for the AHP’s work in Bangladesh. Initially World Vision Australia and a consortium involving Save, Oxfam, and CARE were selected. Subsequently, additional funding was also provided to Plan International Australia. These activities commenced in August 2019, and are intended to be completed by November 2020, as per the following table.

AHP Implementing Partners

Locations Amount Funded Start – End Dates

Save the Children / Oxfam / CARE

Cox’s Bazar $4,950,000.00 August 2019 – November 2020

Plan International Australia Cox’s Bazar, Northern Districts

$2,475,000.00 August 2019 – November 2020

World Vision Cox’s Bazar $2,475,000.00 August 2019 – June 2020

Each of the NGO projects build upon existing operational capacities and programming within Cox’s Bazar and the Northern District, including previous interventions under the AHP through the first phase of the Bangladesh Rohingya Humanitarian Response in 2017-2018. A summary of the NGO projects is outlined below.

Save the Children / Oxfam / CARE

The overarching objective of the Consortium project is to implement an integrated and inclusive project with activities delivered across the following sectors: Health (Sexual and Reproductive Health, Mental Health and Psycho-Social Support); Education; Protection; and Water Sanitation & Hygiene (WASH) services. The project also aims to promote social cohesion between host and refugee communities, as well as support localisation. A total of 163,697 people in Cox’s Bazar are intended to benefit from the project activities, including 4,897 people with disabilities.

Plan International Australia

The Plan International Australia project focuses on adolescent girls and boys across the sectors of Education and Protection, specifically through social empowerment, education in emergencies and the provision of life-saving assistance under two key components:

1. Build the capacity of adolescents and youth, including those with disabilities to strengthen everyday peace building and community support empowerment.

2. Girls, adolescents, young women and their families are able to secure their rights to dignity, protection, and education through timely humanitarian assistance.

The project aims to support a total of 43,000 people, including 6,450 people with disabilities in Cox’s Bazar and flood affected locations in the Northern Districts of Bangladesh.

World Vision

The World Vision project aims to meet the immediate life-saving needs of Rohingya refugees and host community members, including persons with disabilities residing in Cox’s Bazar district through provision of emergency WASH and protection support with a specific focus on Gender-Based

Page | 4

Violence prevention. A total of 26,965 Rohingya refugees are intended to benefit from the project, including 1,349 people with disabilities in Cox’s Bazar.

AHP Bangladesh Response - Phase Three

DFAT and the NGOs are currently designing a third package of support to Bangladesh to be funded through the AHP. This will involve all of the six lead AHP NGOs including CAN DO which is currently implementing activities in Cox’s Bazar funded through other sources. The AHP Bangladesh Phase III Consortium program is due to commence in July 2020. During the first six months of the program agencies will work together to produce key documents to guide program implementation and quality.

These inception deliverables and corresponding deadlines for delivery are outlined below.

Inception Deliverable Due Date

MEAL Framework 31 January 2021

Communications and Advocacy Strategy 31 January 2021

Communications and Advocacy Activities Plan 31 January 2021

Multi-Year Joint Implementation Plan 31 January 2021

Governance Structure, including Terms of Reference (ToR) and standard operating procedures for decision making

31 January 2021

Risk Management Plan 31 January 2021

Localisation Action Plan 31 January 2021

Disability Inclusion Action Plan 31 January 2021

Gender Action Plan 31 January 2021

The evaluation of the Bangladesh Phase II Response is intended to provide important background analysis which will inform these inception deliverables, and also the broader programming under the AHP Bangladesh Phase III Consortium program.

COVID-19

In May 2020, in response to the COVID-19 pandemic, NGOs were given the opportunity to access early funding of up to $500,000 each from the Bangladesh Phase III package for COVID-19 preparedness and response activities in Cox’s Bazar. Of the six AHP NGOs, CARE, Save the Children, and World Vision opted in, each submitting proposals requesting $500,000. CAN DO, Plan International, and Oxfam elected not to use any Phase III funds for COVID-19 activities. Rather, Plan International and Oxfam will pivot their remaining Phase II activities, and CAN DO will utilise other funding sources for their COVID-19 activities. AHP COVID-19 response activities are expected to commence in June 2020. NGOs are expected to receive their COVID-19 funding in June 2020.

Page | 5

Scope

The evaluation will assess the Australian Humanitarian Partnership (AHP) Phase II Response to the Rohingya humanitarian crisis, and the implementing agencies early work in responding to COVID-19.

The evaluation will assess the relevance, effectiveness, and efficiency of the response. In addition, all AHP independently-led evaluations are required to investigate four common cross-cutting issues for AHP. These include:

• Inclusion (gender, disability and other marginalised groups)

• Transparency and accountability to affected populations and other stakeholders

• Localisation

• Cost effectiveness

These reflect the commitments of the Grand Bargain as well as DFAT policy priorities. The evaluation will consider these issues both in terms of specific outcomes achieved, and the extent to which the AHP mechanism is contributing to progress more generally in these areas.

These issues reflect high priorities for the humanitarian sector, but also areas of significant challenge in any response. Therefore, the common investigation of these cross-cutting issues will enable DFAT and the NGOs to progressively draw together lessons and insights from multiple evaluations, and to reflect those lessons in future humanitarian activities.

The evaluation will deliver a set of findings about the Bangladesh Rohingya Response Phase II, and will make recommendations regarding the AHP Bangladesh Phase III Consortium program, and the agencies work in relation to COVID-19, and more broadly for consideration by DFAT and the AHP NGOs. Recommendations must be practical in nature and focused on those which can feasibly be addressed by DFAT and or the NGOs.

Methodology

The Evaluation Team Leader will develop a comprehensive and rigorous evaluation methodology and will document this in an Evaluation Plan including the relevant data collection and analysis tools. The evaluation methodology will address the requirements within DFAT’s Monitoring and Evaluation Standards (see Annex 1). The approach to data collection should be comprehensive and in-depth and include the use of several different methods to triangulate data and gain a deeper understanding of the outcomes of the project. Throughout the evaluation there will be consistent attention to involving men, women, children, people with disabilities, host communities and households and minority groups. The evaluation will be designed and conducted with regards to high standards of ethical conduct and the approach to ethics and safeguarding will be documented in the Evaluation Plan.

It is recognised that there are numerous constraints to data collection posed by COVID-19 restrictions, including limited access to Cox’s Bazar and difficulties in using mobile phones to reach project beneficiaries. Consequently, it is likely that all data collection will need to be conducted remotely. The evaluation team will be asked to explore innovative ways to collect data directly from beneficiaries. However, given the constraints, the evaluation will most likely involve secondary data analysis from existing documentation, with additional interviews and surveys with a wide range of other stakeholders.

Page | 6

Responsibilities will include:

1. Developing a detailed evaluation plan, including data collection tools, interview guides, a framework for data analysis, and timeline. These will be updated and finalised based on feedback from the AHP NGOs, DFAT and the AHPSU.

2. Developing a rubric with input from key stakeholders identifying clear standards and expectations for each of the relevant evaluation questions for the evaluation team to make a transparent judgement about the projects’ effectiveness, inclusion etc. The Team Leader may develop an initial draft of the rubric following stakeholder interviews, and further develop this in consultation with DFAT and the partners. An example rubric is included in Annex 2, taken from a previous AHP evaluation in Iraq.

3. A desk review of background documents (approved proposals, Project Implementation Plans, needs assessments, baseline reports, progress reports, relevant monitoring data, external background documents, etc.).

4. Remotely test all data collection tools, before wide-spread use.

5. Developing a sampling frame and conducting interviews:

o Key informant interviews with stakeholders in Australia including DFAT and the implementing agencies;

o Key informant interviews with stakeholders in Bangladesh including DFAT Post, NGO staff, other implementing partners, communities, regulatory authorities and other stakeholders.

6. Comprehensive evidence-based and triangulated analysis against the evaluation questions and rubric.

7. Presentation of preliminary findings for verification with relevant DFAT representatives and the AHP staff and partners in Bangladesh.

8. Writing an evaluation report suitable for publication, to be published on DFAT’s official website, the AHP website and elsewhere.

9. Communication of the key findings through a verbal report to the Steering Committee members (DFAT and the AHP NGOs) (delivered remotely). This may include separate presentations in Australia and Bangladesh.

10. The Team Leader will also provide weekly updates to the AHPSU MEL Manager on progress and provide regular briefings to the evaluation steering committee as required.

The evaluation might also include a small light-touch trial of the SenseMaker monitoring tool. This could involve using SenseMaker to collect a small amount of data from the AHP agencies in country (e.g. on their experiences in relation to responding to COVID-19). This would provide data to inform the evaluation, as well as give the lead agency representatives exposure to how the SenseMaker tool works in practice. The AHPSU will provide support if this trial of SenseMaker goes ahead (e.g. by arranging access to the software, and contracting technical support for the design of the tool).

Page | 7

Key Evaluation Questions

A set of guiding evaluation questions provide a framework for the evaluation and are set out with reference to the relevant commitments and quality criteria of the Core Humanitarian Standard, ensuring alignment with global commitments. The Team Leader and the rest of the evaluation team will refine the evaluation questions in preparing the Evaluation Plan, based on the following indicative questions:

1. Was the response appropriate and relevant? (CHS 1, CHS 6)?

a) To what extent were the activities selected appropriate (i.e. did we select the right activities in the right locations on the right sectors?)

b) How well did the NGOs and their partners respond to needs assessment information provided (both initially in planning, and over the course of implementation), as needs have changed?

c) To what extent did the assistance align with Australia’s Humanitarian Strategy and other key Australian government policies/priorities such as gender equality, disability inclusion and other vulnerable groups?

d) How relevant and appropriate was the assistance provided by Australian implementing partners from the perspective of affected communities?

2. Was the response effective? (CHS 2)?

a) How clearly defined were the intended outputs and outcomes for the AHP response?

b) To what extent were intended outcomes achieved, and did any unintended outcomes eventuate, either negative or positive?

c) What were the barriers and enablers to effective and efficient program design and achievement of the outcomes?

d) To what extent did Australian-funded activities promote longer-term resilience of affected communities and support broader recovery and stabilisation efforts?

e) How adequate were the NGO’s M&E practices to measure outcomes, and to enable them to assess the effectiveness and inclusion of their response (e.g. are these practices triangulated, are the most marginalised reached through these processes)?

f) What are the existing strengths and weaknesses of each of the NGO projects as relevant to the inception deliverables for the Phase III response - e.g. in regard to M&E, localisation, disability inclusion, and gender inclusion?

g) How effectively did the NGOs report and manage risk, fraud and corruption?

3. How inclusive was the response?

a) To what extent were the needs of different groups of people (including age, gender, ethnicity etc.) considered in the design and implementation of the response, including in influence and decision-making roles?

b) What did the AHP investment achieve in terms of protecting the safety, dignity and rights of affected people, promoting gender equality and addressing barriers to inclusion, including for people with disabilities and minorities?

4. How efficient was the response (CHS 2, CHS 9)?

Page | 8

a) To what extent was the response implemented according to agreed timelines and budgets?

b) In what ways was the response implemented to achieve good value for money?

5. Did the response reinforce local capacity/leadership (CHS3, CHS 4, CHS6)?

a) To what extent did the AHP investment support and strengthen local partners, including civil society (e.g. local women’s organisation, disabled people’s organisations) and local government, and include their participation in coordination fora?

b) What were the main barriers to involving local actors in the provision of assistance?

c) How were implementing partners engaged with affected communities, local government and coordination mechanisms?

d) What evidence is there of genuine and diverse local involvement in the planning, management and implementation of the response, including in influencing and decision-making roles?

6. How transparent and accountable was the response (CHS 4, CHS 5)?

a) To what extent were implementing partners sufficiently accountable to, and engaged with, affected communities?

b) What evidence exists of programs having been influenced by effective communication, participation and feedback from affected people and communities?

7. What can be learnt from the agencies’ early work in relation to COVID-19?

a) To what extent have the agencies integrated COVID-19 considerations into their response?

b) What are the early successes, challenges and lessons regarding integrating COVID-19 that could help to inform the agencies response in Phase III?

c) To what extent did the agencies COVID-19 assistance align with the Australian Government’s COVID-19 Aid Strategy, ‘Partnership for Recovery: Australia’s COVID-19 Development Response’?

Evaluation Steering Committee

The AHP Evaluation Steering Committee will oversee the evaluation, including approval of this terms of reference, and the selection of the Team Leader. The Steering Committee will be required to endorse the major outputs from the evaluation team: the evaluation plan and the final evaluation report (see below). The AHP Support Unit will facilitate this process and support the Steering Committee to provide its endorsement in a timely way. The evaluation steering committee will initially involve the Australian NGO representatives, but will quickly look to increase the engagement with the ANGO’s country office representatives where possible.

Evaluation Team

The evaluation Team Leader will be a senior evaluation specialist with the experience and skill specified below. The evaluation Team Leader would be supported by several technical specialists. These would include professionals with technical expertise in specified areas of need such as gender equality, localisation and M&E (if these skills are not already held by the Team Leader).

Page | 9

The AHPSU will endeavour to find a local evaluation or research consultant or team based in Bangladesh to lead the evaluation. If required, the evaluation could also be supported remotely by an international evaluation specialist. If a local evaluation firm is engaged, it is preferable that a researcher with knowledge of local languages works with the Team Leader to collect data directly from beneficiaries.

All consultants would be engaged by the AHPSU.

NGO involvement on the team

The team will also include a representative from each of Save the Children, Plan International Australia, World Vision and DFAT, who will bring required technical, country and contextual expertise. Staff from each of the NGOs will support the in country field work and will participate in the collection of data in ways which ensure there is no actual or perceived conflict of interest or bias.

The specific roles of each team member will be negotiated and agreed as part of the evaluation plan, with responsibility for data analysis, formulation of findings and recommendations, and the overall quality of all outputs resting with the Team Leader. The team will also include a representative from CAN DO who will be involved in the collection of data relevant to the inception deliverables for the Phase III response.

Save the Children, Plan International Australia, World Vision, CAN DO and DFAT will confirm their team members before the start of the evaluation process (i.e. before the preparation of the Evaluation Plan), so that the entire team can substantively contribute.

Evaluation Utilisation

The evaluation is intended to provide important background analysis and evidence to support the development of the Bangladesh Response Phase III Inception Deliverables and broader programming under the Phase III response. The evaluation is also intended to support shared learning and accountability amongst the involved organisations, including DFAT, Save the Children, Plan International Australia, and World Vision and their partners in Bangladesh. The evaluation process, and the report produced as a result, must be suitable for circulation and should also provide the basis for partners to share findings with the communities supported by the response.

The evaluation will also contribute to wider learning through the Australian Humanitarian Partnership through the Evaluation Steering Committee and the overall partnership, supported by the AHP Support Unit. The AHP Support Unit may utilise the evaluation report to prepare brief summary documents for wider circulation within the AHP NGOs as a basis for the intended learning. DFAT intends to publish the evaluation report.

Key Documents

The NGOs, the AHP Support Unit and DFAT will make available to the Team Leader information and documents relating to the project and relevant background information on the AHP. These will include, but not be confined to, the documents outlined below. The NGOs shall make available to the evaluation team any other reasonable requests for information and documentation relating to the evaluation. The evaluation team is also expected to independently source other relevant material and literature.

Page | 10

• NGO project documents: MEAL plan, proposal, logframe, needs assessment reports and baselines, project implementation workplans, progress reports etc

• Summary of the history of the AHP activation and project, including agreed changes

• DFAT Monitoring and Evaluation Standards

• DFAT Aid Evaluation Policy

• DFAT COVID-19 Aid Strategy: ‘Partnerships for Recovery: Australia’s COVID-19 Development Response’

• Other reports examining the needs of affected populations, and gaps of current humanitarian

assistance in Cox’s Bazar, to ensure a broad evidence base for the evaluation.

Evaluation Timeline

The evaluation is expected to take place from the start of July, and to be completed by September 2020, following the contracting of the Team Leader and the confirmation of all other team members. The key scheduling requirements are for the evaluation to capture data from NGO partners and stakeholders completing their activities in the June-August period and for the final report to be available by late September to inform the development of the inception deliverables for the Phase III response. Several NGOs (e.g. CARE and World Vision) are scheduled to complete their activities by the end of June 2020, so assessment of their projects would be prioritised in order to interview staff while they are still available. The indicative workplan for the Team Leader below sets out the number of days expected for each part of the evaluation process. This will be further developed in the evaluation plan.

Action Approximate Days

Initial briefing from the AHPSU and DFAT and high-level document review

2

Draft the evaluation plan, including data collection tools, interview guides, a framework for data analysis, and timeline. Update and finalise based on feedback from the AHP NGOs, DFAT and the AHPSU.

3

Develop a draft and final rubric for measurement of each of the evaluation questions

2

Initial key informant interviews with stakeholders in Australia (remote) 2

Evaluation data collection and consultations in Bangladesh (remote) (up to 5 days per NGO) including document analysis for each NGO

25

Data analysis 5

Presentation of preliminary findings in a validation workshop (remote) 1

Development of first draft of the report 6

Page | 11

Submission of the second, final draft following potentially two rounds of feedback

2

Communication of the key findings through a verbal report to the Evaluation Steering Committee members

1

Regular communication with the AHPSU and Steering committee 1

TOTAL 50

Additional time will also be allocated for engagement with any technical specialists contracted to support the evaluation, and engagement with the SenseMaker trial if this proceeds. The level of engagement by the evaluator with the CAN DO project in Bangladesh is still to be determined. While the team will not evaluate the work of CAN DO in relation to this response, it could be beneficial to conduct some data collection of their existing strengths and weaknesses in relation to the areas that inform the development of milestones for Phase III of the response.

Team Leader Selection Criteria

Required skills, qualifications and experience

• Academic degree in International Development Studies, Humanitarian Action, Evaluation, or a related field;

• Demonstrated experience in humanitarian response and knowledge of humanitarian standards (CHS, Sphere, Code of Conduct).

• Demonstrated experience in evaluations in the humanitarian sector, particularly involving people marginalised by age (especially children), ethnicity, disability and gender

• Strong understanding of humanitarian and evaluation ethics and a commitment to ethical working practices

• Demonstrated high level skills with quantitative and qualitative research and analysis

• Excellent analytical/problem-solving skills and detail-orientation

• Proven record of communicating with beneficiaries, including through interpreters

• Highly developed self-management, and communication skills, including advanced English writing skills;

• Relevant subject matter knowledge and experience regarding the key sectors of intervention

• Experience in working with international organisations or NGOs, including abiding by their child protection and prevention of sexual harassment, exploitation and abuse policies

Desirable skills, qualifications and experience

Page | 12

• Demonstrated experience of working in Bangladesh, particularly with the Rohingya refugee community

• Experience evaluating complex humanitarian crises responses in Bangladesh

• Knowledge of DFAT funding mechanisms, including for humanitarian responses

• Relevant language skills

• Familiarity with SenseMaker

• Expertise in one or more of the following areas; Gender equality; Disability inclusion; Monitoring and Evaluation, and ideally Localisation.

Page | 13

Annex 1 – DFAT Monitoring and Evaluation Standards – Independent Evaluations

STANDARD 5 INDEPENDENT EVALUATION PLANS

Page | 14

DETAILED DESCRIPTION OF STANDARDS FOR EVALUATION PLANS

Note: The Evaluation plan is developed by the evaluator based on the ToR. It is a negotiated document between

the client and the evaluator and should provide more detail and reflect final agreements after that negotiation.

The evaluation plan should be submitted as early as possible, to enable scheduling of site visits, interviews and

Page | 15

other data collection activities. The agreed Evaluation Plan ought to provide the basis by which evaluator

performance is assessed.

5.1. The evaluation plan is based on a collaborative approach The evaluator has consulted DFAT, and the stakeholders identified as important by DFAT, to develop

the evaluation plan. Consultation may have been in-person, by phone or by email. Important

stakeholders have been given the opportunity to comment on the evaluation plan before the

evaluation commences. Note: This ensures that additional information will not be requested after

the data collection phase is complete.

5.2. Primary intended users of the evaluation are clearly identified

An evaluation cannot meet the needs of all interested stakeholders. Individuals (by title) in named organizations should be identified as the primary users of the evaluation findings. These are the people who will be using the information to make judgments and decisions. Audience is a different concept and often refers to a broader group of people that may be interested in, or may be affected by any decisions that result from the evaluation.

5.3. The purpose and/or objectives of the evaluation are stated.

These would normally be taken from the terms of reference. The evaluation plan restates these so that it is a stand-alone document.

5.4. A summary is provided to orient the reader

This is an introductory orientation of the overall design of the evaluation. It is short, about one paragraph in length. For example, it could highlight whether the evaluation is predominantly exploratory or descriptive, or whether a cause and effect design is proposed, or whether or not any case studies would feature in the overall design. It would highlight the major methods for data collection and analysis. This is called the investigatory framework in research and evaluation terms. The evaluation plan does not go straight into detailed descriptions of methods without this general orientation.

5.5. Limitations or constraints for the evaluation are described

The time available for the evaluation has implications for the scope of the evaluation. If a large number of questions are posed, but DFAT only wants a cursory look at many of these, then a shorter time frame may be appropriate. The evaluator highlights any important limitations in terms of time available, resources applied, or the expertise of the evaluation team to deliver a credible, defensible evaluation product. Consideration is also given to any potential limitations in addressing gender equality issues and/or reaching representatives of special beneficiary sub-groups. Political sensitivities are highlighted where appropriate. The implications of these limitations are discussed. Note: A long list of limitations is not considered a substitute for a poorly negotiated TOR.

5.6. The Key Evaluation Questions are supplemented by detailed descriptions and/or sub questions

Although the terms of reference is where DFAT communicates what the evaluation is to address, the

evaluator will still need to give careful consideration to how these larger questions will be addressed.

This means that more detailed information requirements and/or sub-questions are generated.

Commonly, questions presented in a terms of reference are broad, therefore this more detailed

information allows information users to know how the evaluator has interpreted the broader

questions, and whether or not the evaluation will generate sufficient information to meet these

broader questions. It also allows the DFAT evaluation manager to see the implications of the

scope of the evaluation described in the terms of reference. This breakdown of information

Page | 16

requirements or questions allows the reader to assess whether or not the original scope was

realistic. Where relevant, the evaluator demonstrates how gender equality and social inclusion

issues are investigated as a stand-alone area of enquiry or as integral to other questions of

effectiveness, efficiency, sustainability and so on. The evaluation manager needs to pay careful

attention to this aspect of the evaluation plan.

5.7. High priority questions are identified

DFAT evaluations often have a very large number of evaluation questions that cover a very wide number of aspects of the investment to be evaluated. Some of these questions will be more important than others. The evaluation plan reflects where the emphasis will be placed, and it is clear that DFAT’s information needs will be met. The evaluation team will not usually be able to answer all the questions listed for all respondents and so will need to make decisions during interviews about what will be dropped and what is essential. The evaluation manager needs to be confident that the evaluator will, at a minimum, deliver information on the priority questions.

5.8. There is sufficient flexibility to be able to address important unexpected issues as they emerge

This flexibility may be built in to the questioning technique employed during an interview. It may be built into the schedule as a whole to allow new issues to emerge and be responded to through additional data collection if they are important. Where new issues cannot be adequately addressed within the schedule, there are processes to review possible trade-offs to allow them to be addressed.

5.9. Methods for each evaluation question are described

The evaluation plan shows how each of the evaluation questions will be answered by describing the methods that will be used to collect the information. For most DFAT evaluations this is likely to include in-depth interviews, focus group discussions/interviews, document reviews and in some cases observations of activities. Large workshops are not usually a suitable method to gather substantive, reliable and valid information – however, they may have other important political purposes. For several questions there may be a number of data collection methods proposed to strengthen confidence in the findings.

Consideration is given to the design of data collection methods that are responsive to the needs, rights and security of respondents, with special consideration given to the needs of any special sub-groups. This includes the identification and recruitment of investment participants as respondents, the time of day and location of data collection activities, and the techniques employed during data collection.

The design of major evaluation activities/studies are annexed and include tools such as interview guides or questionnaires. In some cases the evaluator will need to develop these later, or adjust them as the evaluation proceeds, but there is an absolute expectation that the evaluator uses tools to guide each evaluation activity, and does not rely on memory of all the evaluation questions identified in the evaluation plan. Where team members are working in different locations then tools are available ahead of time so that data is collected systematically. If flexibility on this is required, then a compelling rationale is provided. Summary statements of methods that are not linked with specific evaluation questions are not considered adequate.

5.10. Methods are appropriate for the evaluation questions posed

Although this takes evaluation expertise, it is still worth reviewing the questions posed and consider if the methods described could reasonably answer the questions. For example, a focus group discussion would be most unlikely to answer a sensitive question; a review of a program

Page | 17

strategy document (such as gender) would be unlikely to tell you if the investment’s actual gender activities were of a high quality. It would need to be supported by information from other sources.

5.11. Triangulation of methods is proposed

Triangulation is the use of a range of methods and/or sources of information to come to a conclusion or result. It can develop greater confidence in a finding. Given the short time frame of most DFAT evaluations or reviews, it is difficult to employ a wide range of methods. To deal with this, the evaluation has planned to discuss similar questions across a range of different respondents within and across different organizations or target beneficiary groups (particularly special sub-groups), or use a number of methods to examine the same issue. It is not sufficient to state that triangulation will be used if this is not demonstrated in the evaluation design.

5.12. Sampling strategy is clear and appropriate

Most evaluations will require some sort of sampling strategy across individuals and special sub-groups, sites or time periods. Appropriate sampling strategies are chosen and justified. For short reviews that rely on analytical rather than statistical inference, purposeful sampling will be appropriate and could include maximum variation, a critical case, or a typical case. Efforts should be made to avoid relying on a convenience sample which is likely to be unrepresentative of the population of interest. Where statistical inference will be used to generalize from the sample, random sampling strategies are appropriate – especially stratified random sampling which reduces the sample size required.

5.13. The plan describes how data will be processed and analysed

The evaluation plan describes how the data will be processed, including measures to check and correct any errors in data, ensure security of storage and prepare for analysis. The plan also describes how the data will be analysed in order to answer the Key Evaluation Questions. This may not necessarily require advanced analytical methods, but users of the information can determine exactly what is to be done. Consideration is given to the analysis of disaggregated data for gender and other relevant sub-groups where data collection or secondary sources has allowed this.

5.14. The plan identifies ethical issues and how they will be addressed

For most of the evaluations and reviews conducted by DFAT, this will mostly be around privacy and confidentiality issues. The plan identifies how these will be addressed when data are collected, stored and reported. In particular, assurances about anonymity must be honoured and data stored and reported in ways that do not inadvertently identify informants, including when providing a database of the evidentiary basis to DFAT as part of the deliverables. Other relevant ethical issues are addressed including processes for reporting serious issues if identified during data collection.

5.15. The process for making judgments is clear The evaluation plan makes it clear that the evaluator will make professional judgements based on the evidence gathered and on the agreed criteria or standards. DFAT’s response to the evaluator’s judgement should be provided in the Management Response to an evaluation. In some exceptional cases, DFAT may require an evaluator to report neutrally on facts and leave DFAT to make the final judgements, in which case the plan should make it clear how evaluative judgements will be made and by whom, as this is an important distinction and can affect the way information is collected and presented.

5.16. Approaches to enhance utilization of findings are outlined

Page | 18

The importance of utilization of findings needs to be communicated to the evaluator. There are a variety of well-tested approaches to utilization that a professional evaluator will be familiar with (e.g. stakeholder engagement strategies for evaluation design or developing acceptance of recommendations before the report is published). Approaches to utilisation of findings are outlined in the evaluation plan. Utilization begins with the evaluation design stage. Consideration is given to providing feedback to representatives of women's groups and relevant special sub-groups.

5.17. Scheduling guidance is provided

The schedule is developed by DFAT after the evaluation plan is submitted, and reflects guidance from the evaluator. The most common problem is that the persons recruited for interview are not always the best respondents for the evaluation questions posed. Often there are many donor meetings where respondents cannot provide substantive comment on many of the evaluation questions. Also consider the time for each interview with the associated evaluation questions. Most 60 minute interviews with a respondent cover no more than four or five key topics; less if translation is required. Sufficient time is available to meet with the implementation team. As part of reviewing the methodology DFAT negotiates the proposed list of respondents before final scheduling. The evaluator scheduling guidance is realistic for the time frame. Sufficient time is allocated to other methods proposed. There is sufficient time allocated to evaluation team discussions and early data processing (not late at night). Sufficient time is given to the recruitment of respondents who represent special sub-groups, any time required to travel to the appropriate locations, and setting the tone for interactions that enable respondents to provide their views with confidence and in comfort.

5.18. Evaluation tasks are allocated to team members

It is very important that each team member knows before the evaluation begins what they will be expected to do. It is not appropriate for the team leader to allocate reporting responsibilities on the last day of the in-country mission. The evaluation plan shows what responsibilities each team member has so they can ensure that adequate data is collected, processed, and interpreted and they can meet a high standard during the reporting stage. It is often useful to show which evaluation questions each team member carries responsibility for.

5.19 The plan for publication of the final evaluation report is documented

The plan for publication of the final evaluation report should be documented as part of the Evaluation Plan in order to facilitate meeting the requirements of the DFAT Aid Evaluation Policy.

STANDARD 6 INDEPENDENT EVALUATION REPORTS

Page | 19

Page | 20

DETAILED DESCRIPTION OF STANDARDS FOR EVALUATION REPORTS

INTRODUCTIONS

6.1. The background provides adequate information for individuals not familiar with the investment

The background provides adequate information to enable individuals not fully familiar with the investment to interpret the report. It summarizes: the total value of the investment; the number of years of the investment; the stage of investment implementation; the delivery mechanism; key expected outcomes of the investment; and the key issues identified in the terms of reference.

6.2. A brief summary of the methods employed is provided

Although a fully elaborated methodology was developed before the evaluation, a summary of the significant details is included. Sufficient information is required to enable the reader to quickly understand the evidentiary basis of the evaluation. The evidentiary base must be convincing and in proportion to the resources invested in the evaluation. The full methodology is annexed. Important aspects of the strategy to ensure findings are utilised are summarised here.

6.3. Key limitations of the methods are described and any relevant guidance provided to enable appropriate interpretation of the findings

Key limitations are summarised in the evaluation report to enable the reader to make appropriate decisions. Where necessary the author has provided specific guidance of where the reader ought to be cautious about the findings.

6.4. The executive summary provides all the necessary information to enable primary users to make good quality decisions.

The executive summary provides all the necessary information to enable primary stakeholders, especially senior management, to make good quality decisions without reading the entire document. It is not a simple cut and paste of the main body of the report. It summarises the key findings, provides sufficient analyses and arguments, and presents final conclusions and recommendations. Important information about gender equality and social inclusion are included to allow the reader to appreciate important achievements and challenges. Resource implications of recommendations are summarised. The length of the executive summary is proportionate to the length of the report (e.g. two to three pages for short uncomplicated reports, and up to five or six pages for more lengthy reports with complex issues).

Page | 21

FINDINGS AND ANALYSES

6.5. The evaluation report clearly addresses all questions in the Terms of Reference/Evaluation Plan

Note: As the Evaluation Plan supersedes the Evaluation Terms of Reference, the Plan is the appropriate document to assess whether or not the evaluation has delivered on expectations. In the absence of an Evaluation Plan, the Terms of Reference should be used.

It is relatively easy to identify where each of the questions in the Evaluation Plan are addressed. The report does not need to be a mechanical presentation of these questions, but it should be relatively easy to negotiate the report and find relevant information about specific questions in the Evaluation Plan. Where there are gaps, these have been explained. DFAT’s information needs, as set out in the Terms of Reference and Evaluation Plan, have been met.

6.6. The relative importance of the issues communicated is clear to the reader

The report makes it clear what issues are priority issues to consider. Minor issues are not set out mechanically against the terms of reference and given the same depth of treatment as more important issues. The breadth of description, depth of analysis and attention in the recommendations can indicate the degree of priority. The author may simply state the relative importance of issues.

6.7. There is a good balance between operational and strategic issues

The report addresses the full range of issues identified in response to the TOR and other critical issues that have emerged. There will be technical, managerial or operational issues that are very important to consider and are often at the core of many important challenges. The strategic direction or any higher order issues of the investment have been given adequate space, and minor technical issues are treated in a more limited fashion. Flexibility is required where the TOR evaluation questions demonstrate that this balance was not required.

6.8. The report clearly explains the extent to which the evidence supports the conclusions and judgments made

For key findings, the basis of the findings and related conclusions is communicated clearly. This includes reporting the degree to which views are shared across respondents, and representatives of relevant special sub-groups if appropriate. The information is brought together from a range of sources, but communicated as a coherent whole. Evaluator opinions that are based on limited evidence are made transparent and proposed as suggestive only.

6.9. Alternative points of view are presented and considered where appropriate

Alternative views must be presented, especially for important, controversial or disappointing findings. They are not immediately dismissed, but are seriously considered. Key stakeholder views such as those of the implementation team must be given sufficient attention, and balanced by national partners, DFAT or other important stakeholder views.

6.10. Complicated and complex aspects of issues are adequately explored and not oversimplified

The report adequately acknowledges complicated aspects of issues, such as multiple contributing factors, or emergent challenges and opportunities. The report does not present simple solutions to these types of situations. The findings are presented fairly so that specific stakeholders are not held fully accountable for problems when multiple factors are involved. Human development is challenging, and the report recognises that implementation teams and national partners are often facing multiple challenges.

Page | 22

6.11. The role of context and emergent risks to investment performance are analysed The report identifies relevant aspects of the context within which activities are implemented. These

might include geographic, cultural, gender, political, economic or social context. Sufficient

information is presented to allow the reader to understand the relationship between the

investment and its context. The report addresses: a) how the context may have affected the

achievement of outcomes (both supportive and inhibiting); and b) the extent to which the

investment may have had any effect on the context. Important emergent risks are identified.

6.12. The text uses appropriate methods/language to convince the reader of the findings and conclusions

Arguments presented do not use emotive word choices in an effort to appeal to the emotions of the reader. The method used to convince readers is the presentation of evidence or a credible basis for the finding. Using the international literature to build the credibility of the report can be effective. The report handles political issues with sensitivity. A good report considers the expected positions of the important stakeholders – if findings are unexpected then this is carefully communicated and explained in the text.

6.13. There is an adequate exploration of the factors that have influenced the issues identified and conclusions drawn

It is not sufficient to simply describe a situation. A full analysis of the likely factors that have led to the situation is necessary. Factors that enable progress or achievement are just as important as factors that inhibit them. These factors should be generated from a range of data sources. A range of causes should be considered rather than regularly offering a single cause for major and/or complex issues.

6.14. The implications of key findings are fully explored

DFAT investment managers, senior management and other stakeholders need some direction on the implications of the findings if this is not immediately apparent. Implications for achieving investment objectives, and for meeting time frames, expenditure projections, or sustainability are often important considerations.

CONCLUSIONS AND RECOMMENDATIONS 6.15. The overall position of the author is clear and their professional judgments are unambiguous. The task of the evaluator is to evaluate. They must make their position clear (and as early as possible

in the report) unless the TORs have required the evaluator to report on findings with neutrality. The

report does not simply state findings and expect DFAT to interpret them and draw their own

conclusions. The report presents the authors view unambiguously. Has the investment made

adequate progress or not? Are the factors that have accounted for the limited achievements been

unavoidable or are they due to poor management. Unambiguous judgements also present findings

and conclusions sensitively and constructively.

6.16. The conclusions and recommendations logically flow from the presentation of findings and any associated analyses.

It is possible to trace issues through the text from description, to analysis, to conclusion and recommendation. No recommendation appears at the end that is not supported by descriptive and analytical work in the text. There are no important inferred recommendations buried in the text that have not been drawn into the conclusion or list of recommendations at the end.

Page | 23

The “chain of evidence” is evident. This is where all questions in the methodology have data that has been collected, analysis conducted, findings presented, interpretation carried out and reported. If questions in the methodology have not been addressed then an explanation has been given. 6.17. Individuals have been allocated responsibility for responding to recommendations. Where appropriate, job titles, rather than organisations, have been allocated responsibility for actions against all recommendations. If it is not appropriate or possible to identify the individual, then the relevant work group is identified. If some recommendations are for broader partner government, or DFAT sectoral or corporate learning then these are identified separately. 6.18. Significant cost implications of recommendations have been estimated

If recommendations imply human, financial or material costs, these are estimated. If recommendations for additional technical support are made, then the number of days input is estimated. For important technical assistance positions proposed, the key content to consider for the terms of reference is annexed.

6.19 The recommendations are feasible Recommendations, in the most part, are acceptable to relevant stakeholders

(recommendations that stakeholders do not agree with rarely get implemented – coming to

acceptability is dealt with by the utilisation strategy). Recommendations are feasible from a

resourcing and cost perspective. Recommendations are likely to be effective to rectify a

situation, or to achieve an expected outcome.

6.20 The circumstances under which any important lessons are transferable are described

Where there are important lessons to be learned, the report provides sufficient information to inform the reader about the circumstances under which these lessons can be transferred. This could be at the sector level, the Country Program Level, for the Department as a whole, or for the development sector more broadly.

6.21 The final evaluation report is published within the timeframes outlined in the DFAT Aid Evaluation Policy

The final evaluation report must be published within the timeframes outlined in the DFAT Aid Evaluation Policy.

Page | 24

Annex 2 – Evaluation Rubric used for the Mid-Term Evaluation of the AHP Building Peaceful Futures Program Iraq, 2020

Criteria and Definitions

Standards

Satisfactory Unsatisfactory

Excellent (4) Good (3) Less than Adequate (2) Poor (1)

Relevance: How well is the BPF Consortium responding to the changing context and needs assessment information (both in planning, and over the course of implementation)?

The Consortium almost always responds effectively to changes in the context based on information from: 1: Needs assessments 2: Monitoring and evaluation 3: Feedback mechanisms 4: Conflict analysis

The Consortium more often than not responds effectively to changes in the context based on information from: 1: Needs assessments 2: Monitoring and evaluation 3: Feedback mechanisms 4: Conflict analysis

The Consortium sometimes responds effectively to changes in the context based on information from: 1: Needs assessments 2: Monitoring and evaluation 3: Feedback mechanisms 4: Conflict analysis

The Consortium almost never responds effectively to changes in the context based on information from: 1: Needs assessments 2: Monitoring and evaluation 3: Feedback mechanisms 4: Conflict analysis

Relevance: How relevant and appropriate is the assistance provided by the BPF Consortium from the perspective of affected communities?

Almost all (81%-100%) male, female, disabled and child beneficiaries report that the BPF assistance was relevant and appropriate to their needs

More than half (66%-80%) of male, female, disabled and child beneficiaries report that the BPF assistance was relevant and appropriate to their needs

Less than half (41%-65%) of male, female, disabled and child beneficiaries report that the BPF assistance was relevant and appropriate to their needs

Almost none (0%-40%) of male, female, disabled and child beneficiaries report that the BPF assistance was relevant and appropriate to their needs

Effectiveness: To what extent is implementation on track to achieve the intended end of program outcomes

Almost all (91%-100%) of the program outputs are on track to achieve the intended end of program outcomes

More than half (76%-90%) of the program outputs are on track to achieve the intended end of program outcomes

Less than half (41%-75%) of the program outputs are on track to achieve the intended end of program outcomes

Almost none (0%-40%) of the program outputs are on track to achieve the intended end of program outcomes

Effectiveness: To what extent are funded activities promoting longer-

Almost all (81%-100%) male, female, disabled and child beneficiaries report

More than half (66%-80%) of male, female, disabled and child beneficiaries report

Less than half (41%-65%) of male, female, disabled and child beneficiaries report

Almost none (0%-40%) of male, female, disabled and child beneficiaries report

Page | 25

term resilience1 of affected communities?

improvements in their lives due to the intervention with regards to any of the following spheres: - Access to WASH, PHC,

SRMH, legal, and livelihood support and services

- Improvement of social cohesion

improvements in their lives due to the intervention with regards to any of the following spheres: - Access to WASH, PHC,

SRMH, legal, and livelihood support and services

- Improvement of social cohesion

improvements in their lives due to the intervention with regards to any of the following spheres: - Access to WASH, PHC,

SRMH, legal, and livelihood support and services

- Improvement of social cohesion

improvements in their lives due to the program with regards to any of the following spheres: - Access to WASH, PHC,

SRMH, legal, and livelihood support and services

- Improvement of social cohesion

Effectiveness: How adequate are the Consortium’s M&E practices to inform their management, and to enable them to assess the effectiveness and inclusion of their response?

Monitoring and Evaluation practices are of high quality and strong evidence that this data has informed program management

Monitoring and Evaluation practices are of good quality and strong evidence that this data has informed parts of program management

Monitoring and Evaluation practices are of average quality and limited evidence that this data has informed program management

Monitoring and Evaluation practices are of low quality and there is almost no evidence that this data has informed program management

Effectiveness: To what extent is the program conflict sensitive? To what extent did the program adjust its activities and approach following the completion of the Conflict Analysis Study?

Conflict sensitivity was a major consideration when designing all program activities Information from the conflict analysis was adequately used to adjust all program activities

Conflict sensitivity was a major consideration when designing most program activities Information from the conflict analysis was used to adjust major program activities

Conflict sensitivity was considered limitedly when designing program activities Information from the conflict analysis was limitedly used to adjust program activities

Conflict sensitivity was not considered when designing the program activities Information from the conflict analysis was not used to adjust program activities

Effectiveness: To what extent is the program logic or theory of change, including any inherent assumptions holding true?

The theory of change of the program and its underlying assumptions are verified by beneficiaries in relation to how the program affected their lives as well as social cohesion in their community.

The theory of change of the program and most of its underlying assumptions are partly verified by beneficiaries in relation to how the program affected their lives as well as social cohesion in their community.

The theory of change of the program and most of its underlying assumptions are limitedly verified by beneficiaries in relation to how the program affected their lives as well as social cohesion in their community.

The theory of change of the program and most of its underlying assumptions are not verified by beneficiaries in relation to how the program affected their lives as well as social cohesion in their community.

1 Based on program proposal: “The Theory of Change for this project emphasises that an inclusive community-based approach plays a crucial role in ensuring equal access to humanitarian assistance and promoting social cohesion in a volatile context such as Iraq. The project’s ultimate outcome will be to improve vulnerable people’s resilience and wellbeing through inclusive access to WASH, primary health care and SRMH care, legal and livelihoods support which will contribute to social cohesion in Ninewa and Kirkuk Governorates.”

Page | 26

Inclusion: To what extent is the investment protecting the safety, dignity and rights of affected people? Specifically, is the investment making a difference to gender equality and empowering women and girls?

Almost all (81%-100%) female beneficiaries report changes in their lives due to the program intervention directly related to gender equality and empowerment.

More than half (61%-80%) of the female beneficiaries report changes in their lives due to the program intervention directly related to gender equality and empowerment.

Less than half (41%-60%) of the female beneficiaries report changes in their lives due to the program intervention directly related to gender equality and empowerment.

Almost none (0%-40%) of the female beneficiaries report changes in their lives due to the program intervention directly related to gender equality and empowerment.

Inclusion: Do people with a disability have equal access to the program’s activities and are their unique needs being met? Did they feel safe accessing assistance?

Almost none (0%-5%) of the beneficiaries with a disability report feeling unsafe accessing assistance Almost all (76%-100%) of the disabled beneficiaries report that the invention was tailored to their unique needs

Few beneficiaries (6%-15%) with a disability report feeling unsafe accessing assistance More than half (51%-75%) of the disabled beneficiaries report that the invention was tailored to their unique needs

Some of the beneficiaries (16%-30%) with a disability report feeling unsafe accessing assistance Less than half (26%-50%) of the disabled beneficiaries report that the invention was tailored to their unique needs

A lot of beneficiaries (31%-100%) with a disability report feeling unsafe accessing assistance Almost none (0%-25%) of the disabled beneficiaries report that the invention was tailored to their unique needs

Efficiency: To what extent is the program being implemented according to agreed timelines and budgets?

Almost all (91%-100%) program activities are being implemented according to the agreed timelines and budgets.

More than half (76%-90%) of program activities are being implemented according to the agreed timelines and budgets.

About half (41%-75%) of program activities are being implemented according to the agreed timelines and budgets.

Almost none (0%-40%) of program activities are being implemented according to the agreed timelines and budgets.

Leadership: To what extent do local authorities and key local actors believe they were able to guide and influence the program?

Women organizations, disabled persons organizations and government stakeholders feel that they are able to constructively influence the program, and evidence is available as to where this has happened

Some women organizations, disabled persons organizations and government stakeholders feel that they are able to constructively influence the program, and evidence is available as to where this has happened

Women organizations, disabled persons organizations and government stakeholders feel that they are limitedly able to constructively influence the program, and limited evidence is available as to where this has happened

Women organizations, disabled persons organizations and government stakeholders feel that they are not able to influence the program

Leadership: To what extent is the program strengthening local partners, including civil society (e.g. local women’s organisations, DPOs) and

Women organizations, DPOs and government stakeholders’ internal capacity has been very strongly strengthened through the program

Women organizations, DPOs and government stakeholders‘ internal capacity has been considerably strengthened through the program

Women organizations, DPOs and government stakeholders’ internal capacity has been limitedly strengthened through the program

Women organizations, DPOs and government stakeholders’ internal capacity has not been strengthened through the program

Page | 27

local government, and include their participation in coordination fora?

Accountability: To what extent are implementing partners sufficiently accountable to affected communities?

Almost all (81%-100%) male, female, disabled, and child beneficiaries report being able to provide input on the services they received

More than half (66%-80%) of the male, female, disabled, and child beneficiaries report being able to provide input on the services they received

Less than half (41%-65%) of the male, female, disabled, and child beneficiaries report being able to provide input on the services they received

Almost none (0%-40%) of the male, female, disabled, and child beneficiaries report being able to provide input on the services they received

Accountability: What evidence exists of the program having been influenced by effective communication, participation and feedback from affected people and communities?

All program partners have effective feedback mechanisms in place and analyse data coming from these mechanisms regularly There is strong evidence that data from feedback mechanisms is being used to make programmatic changes

All program partners have feedback mechanisms in place and analyse data coming from these mechanisms There is some evidence that data from feedback mechanisms is being used to make programmatic changes

All program partners have feedback mechanisms in place, but data from these mechanisms is only limitedly analysed There is limited evidence that data from feedback mechanisms is being used to make programmatic changes

Not all program partners have feedback mechanisms in place and data is not analysed