African Development Bankidev.afdb.org/sites/default/files/documents/files/IDEV...Independent Development Evaluation African Development Bank An IDEV Impact Evaluation Impact Evaluation

From experience to knowledge... From knowledge to action... From action to impact

Independent Development EvaluationAfrican Development Bank

An ID

EV Im

pact

Eva

luat

ion

Impact Evaluation of the Rural Water Supply and

Sanitation Program in Ethiopia2006–2014

Summary Report

September 2016

An IDEV Im

pact EvaluationIm

pact Evaluation of the Rural Water Supply and Sanitation Program

in Ethiopia 2006–2014 Summ

ary ReportIndependent Developm

ent Evaluation

IDEV conducts different types of evaluations to achieve its

strategic objectives

Thematic Evaluations Project Cluster Evaluations

Regional Integration Stra

tegy

Evaluations

Project Perfo

rmance Evaluations

(Public Secto

r)Impact Evaluation

Project Performance Evaluations

(Private Sector)

Coun

try S

trate

gy E

valu

atio

ns

Evaluation Syntheses

Corporate Evaluations

Sect

or E

valu

atio

ns

Impact Evaluations

From experience to knowledge... From knowledge to action... From action to impact


An ID

EV Im

pact

Eva

luat

ion

Impact Evaluation of the Rural Water Supply and

Sanitation Program in Ethiopia2006–2014

Summary Report

September 2016

An IDEV Im

pact EvaluationIm




ent Evaluation

ACKNOWLEDGEMENTS

© 2016 African Development Bank Group All rights reserved – Published September 2016

Impact Evaluation of the Rural Water Supply and Sanitation Program in Ethiopia 2006–2014 – Summary Report IDEV Impact Evaluation, September 2016

DisclaimerUnless expressly stated otherwise, the findings, interpretations and conclusions expressed in this publication are those of the various authors of the publication and are not necessarily those of the Management of the African Development Bank (the “Bank”) and the African Development Fund (the “Fund”), Boards of Directors, Boards of Governors or the countries they represent.

Use of this publication is at the reader’s sole risk. The content of this publication is provided without warranty of any kind, either express or implied, including without limitation warranties of merchantability, fitness for a particular purpose, and non-infringement of third-party rights. The Bank specifically does not make any warranties or representations as to the accuracy, completeness, reliability or current validity of any information contained in the publication. Under no circumstances including, but not limited to, negligence, shall the Bank be liable for any loss, damage, liability or expense incurred or suffered which is claimed to result directly or indirectly from use of this publication or reliance on its content.

This publication may contain advice, opinions, and statements of various information and content providers. The Bank does not represent or endorse the accuracy, completeness, reliability or current validity of any advice, opinion, statement or other information provided by any information or content provider or other person or entity. Reliance upon any such opinion, advice, statement, or other information shall also be at the reader’s own risk.

About the AfDBThe overarching objective of the African Development Bank Group is to spur sustainable economic development and social progress in its regional member countries (RMCs), thus contributing to poverty reduction. The Bank Group achieves this objective by mobilizing and allocating resources for investment in RMCs and providing policy advice and technical assistance to support development efforts.

About Independent Development Evaluation (IDEV)The mission of Independent Development Evaluation at the AfDB is to enhance the development effectiveness of the institution in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge.

Independent Development Evaluation (IDEV)African Development Bank GroupAfDB HeadquartersAvenue Joseph Anoma, 01 BP 1387, Abidjan 01, Côte d’IvoirePhone: +225 20 26 20 41E-mail: [email protected]

Layout & production: Visual Identity – www.visualidentity.co.uk Original language: English – Translation: AfDB Language Services Department

Task managers Rita Tesselaar, Methods and Quality Expert, Girma Kumbi, Principal Evaluation Officer and Foday Turay, Chief Evaluation Officer

Consultant Dr. Degnet Abebaw, Impact Evaluation Expert BDS Centre for Development Research (CDR)

Reference Group Teferi Menkir, Senior Water and Sanitation Officer, AfDB Ethiopia Field Office, Tom Robert Mugoya, Principal Water and Sanitation Engineer, AfDB Nigeria Field Office, Jochen Rudolph, Chief Rural Water and Sanitation Specialist, AfDB Water and Sanitation Department, Jacob Oduor, Principal Research Economist, AfDB Development Research Department, Solomon Tesfasilassie, Senior expert, Ethiopia National Planning Commission

External peer reviewers Dr. Howard White, 3ie, Dr. Hugh Waddington, 3ie, Prof. Mayra H. Addison, Catholic university Skat Consulting Ltd.

Knowledge management officers Jacqueline Nyagahima, Consultant, Jerry Lemogo, Junior Consultant

Other assistance / contributions John Mbu, Consultant; IDEV administrative team: Ruby Adzobu-Agyare, Myrtha Diop, Henda Ayari, Mariem Dridi, Blandine Gomez

Special thanks to We are grateful for financial support to this evaluation from the Rural Water Supply and Sanitation Initiative Trust Fund.

Division manager Rafika Amira

Evaluator-General Rakesh Nangia

mailto:idevhelpdesk%40afdb.org?subject=

http://idev.afdb.org/

http://www.visualidentity.co.uk

Acknowledgements iiAbbreviations and Acronyms vExecutive Summary 1Management Response 7

Background Information 17Country Context 17Description of the Rural Water Supply and Sanitation Program (RWSSP) 17Theory of Change Underlying the RWSSP 19

Objectives, Questions and Scope of the Evaluation 20Objectives of the Impact Evaluation 20Evaluation Questions 20 Scope of the Impact Evaluation 21

Methodology 23Impact Evaluation Design 23Sample Size and Sampling Procedures 23Survey Instruments and Implementation of the Field Survey 24Limitations 24

RWSSP Implementation and Impacts 27RWSSP Implementation 27Water, Sanitation and Hygiene Associated Effects 29 Safe Water Access and Use 29 Household Sanitation and Hygiene 32 Incidence of Diarrhea among Children under Five 33Time Savings, School Enrolment and Women’s Participation Self-employment 34Heterogeneous Impacts of the RWSSP 36Sustainability of Benefits of the RWSSP 37

Conclusions and Recommendations 39

Annexes 43RWSSP Theory of Change and Definition of Outcome Variables 44Definition of Community Level Outcome Variables 45Sample Size and Sampling Procedures 46Descriptive Statistics 50RWSSP Implementation by the Selected Communities 60Impacts of RWSSP on Community Level Outcome Variables (Full Results) 63

Contents

Contents

List of Figures Figure 1: Predominance of public taps/standpipes in RWSSP communities 27Figure 2: Substantial household use of RWSSP drinking water points. 29Figure 3: Self-reported incidence of hand-washing at critical times. 33

List of Tables Table 1: Equal distribution of representative evaluation Sample Size 24Table 2: High water quality at source relative to point of use 30Table 3: Household water storage and handling 30Table 4: Significant impact of RWSSP on access to improved water source, but not on daily

water consumption 31Table 5: Significant impact of RWSSP on household latrine ownership and use 32Table 6: No significant impact of RWSSP on incidence of diarrhea among under-fives 33Table 7: Significant impact of RWSSP on time spent on fetching water 34Table 8: No significant impact of RWSSP on children’s current school enrolment 35Table 9: No significant impact of RWSSP on women’s participation in (self) employment 36

References 76

Endnotes 79

vAbbreviations and Acronyms

An ID

EV Im

pact

Eva

luat

ion

Abbreviations and Acronyms

AfDB African Development Bank Group

CFT Community-Facilitation Team

CSA Central Statistical Agency

DID Difference-in-Difference

ETB Ethiopian Birr

ETFO Ethiopia Field Office

FDRE Federal Democratic Republic of Ethiopia

GoE Government of Ethiopia

IDEV Independent Development Evaluation

JMP WHO / UNICEF Joint Monitoring Program

M&E Monitoring and Evaluation

MEL Monitoring, Evaluation and Learning

MoE Federal Ministry of Education

MoFED Federal Ministry of Finance and Economic Development

MoH Federal Ministry of Health

MoWE Federal Ministry of Water and Energy

NGO Non-governmental Organization

OWAS Water and Sanitation Department, AfDB

PSM Propensity Score Matching

RWSSP Rural Water Supply and Sanitation Program

SNNPR Southern Nations, Nationalities and People’s Region

WaSH Water supply and Sanitation and Hygiene

WaSHCOM Water, Sanitation and Hygiene Committee

WHO World Health Organization

WSST Woreda Water Supply and Sanitation Team

WWT Woreda Water Team

1Executive Summary

An ID

EV Im

pact

Eva

luat

ion

Executive Summary

The African Development Bank (AfDB) supported the Rural Water Supply and Sanitation Program (RWSSP), one of the major water supply and sanitation devel-opment programs in Ethiopia for the ten-year period 2005–2015. The RWSSP aimed to increase access to and the use of improved water supply and sani-tation services in 125 Woredas/Districts across Ethiopia’s nine regions, using a community demand-driven approach to (i) deliver improved water supply and sanitation services to selected local communi-ties, and (ii) strengthen local, district, regional and national capacity for effective water and sanita-tion development and management. The RWSSP’s beneficiaries include rural sedentary and nomadic populations, artisans, entrepreneurs, and local and federal governments.

The RWSSP had three components: (i) constructing new water schemes, rehabilitating malfunctioning schemes or expanding existing schemes; (ii) build-ing institutional latrines for schools and health faci-lities and communal latrines in areas with crowded settlements; (iii) community sensitization, aware-ness raising, and facilitation, and capacity building of different water and sanitation services providers at the community, woreda, regional and federal levels. The program also encouraged and supported the participation of private sector agents.

Approved in 2005 and implemented during 2006–2014, RWSSP cost a total of UA 54.24 million, of which the AfDB contributed an African Development Fund (ADF) grant of UA 43.61 million (80%), and the Government of Ethiopia (GoE) and communities contributed UA 8.12 million (15%) and UA 2.510 million (5%) respectively. In 2014, when the program was completed however, the actual disbursement was UA 60.203 million, an 11% increase over commitment costs. The additional implementation costs were borne by the GoE and the communities.

The program allocated most of its investment to water supply schemes and sanitation facilities. Its delivery of completed water schemes started in 2008 and ended in 2014.

The main objective of this evaluation study is to provide credible estimates of the impacts of the RWSSP on participating communities and house-holds, especially in terms of (i) access to and use of safe water, (ii) the incidence of diarrhea in chil-dren under five years of age, (iii) children’s school attendance, and (iv) women’s participation in self-employment, and on the sustainability of the results. This is done primarily to (i) account for the RWSSP development outcomes to the AfDB Board and to the GoE, and (ii) provide AfDB’s Water and Sanitation Department (OWAS) with relevant recommendations for sustaining the benefits of the RWSSP and for informing the design and imple-mentation of similar programs in the future. The evaluation study also helps IDEV in building an evidence base for the 2015 Ethiopia CSPE and for learning from impact evaluation.

The evaluation questions focused on the RWSSP impact areas as described above. Given that it was designed and implemented with no rigorous impact evaluation in mind, the study assessed impact by comparing relevant RWSSP community outcomes with those of an appropriately and carefully matched non-RWSSP community group (“counterfactual” group) 1. Data on both groups were collected from random samples of 228 communities, 2736 house-holds, 114 water points, 74 primary schools and 73 health centers in 38 woredas/districts from three regions – Oromia, Amhara and SNNPR – the regions accounting for 85% of the total RWSSP direct partic-ipants. The propensity score matching (PSM) and difference-in-difference (DID) techniques provided the basis for the data analysis.

2 Impact Evaluation of the Rural Water Supply and Sanitation Program in Ethiopia 2006–2014 – Summary Report

Key findings

The RWSSP was effective in delivering improved water and sanitation facilities but less effective in building community institutional capacity. The program delivered close to 6810 water supply and sanitation facilities, or approximately 80%) of the target. Most of these facilities were functional and were in use. However, facility breakdowns reduced the functionality of the RWSSP facilities. Although the RWSSP delivered all planned training for the WASH team and WaSHCOM members, its activities in community awareness raising and sensitization about improved sanitation and hygiene practices were limited. The WaSHCOM membership also lacked gender-balance.

The RWSSP had significant impact on access to and the use of an improved water source ² in selected communities, but was not effective in ensuring the water was completely safe for drink-ing at the source and point of use. The RWSSP improved household access and use of quality water: approximately 91% of households in RWSSP commu-nities had access to and use of an improved water source. This was approximately 69% higher than the non-RWSSP group outcome (22%). In RWSSP communities, households mainly used water from improved water sources whilst only 22% of the house-holds in non-RWSSP communities obtained their water from improved sources. At 59%, the percent-age of the sample of RWSSP water sources that were free of E. coli was 34% higher than that of the non-RSSWP water points, at 25%. At the point of use, 33% of the households in the RWSSP communities had drinking water that tested free of E. coli compared to 14% in the non-RWSSP communities. In 27%–30% of these sampled water sources, the contamination was as low as 1–10 total E. coli count per 100 ml of sample water. This level of contamination at the point of water use was indicative of contamination during water transport and storage. The prevalence of the E-coli in the improved sources was due primar-ily to infrequent quality testing and treatment of the water sources. Water quality tests were not regularly conducted in most communities. Only about 7% of the

sample households reported that they applied water treatment technologies. Notwithstanding the E. coli contamination, about 92% of the sample households in the RSWWP communities perceive the quality of the water from their primary sources as either good or very good.

The RWSSP had no impact on the daily per capita water consumption. The self-reported daily per capita water consumption for domestic purposes in both RWSSP communities (34.12 liters) and non-RWSSP communities (31.11 liters) already far exceeded the national standard consumption target of 15 liters/day per capita.

The RWSSP contributed significantly to the surge in household ownership and use of private latrines and to the decrease in open-defecation, but not on hand washing. The RWSSP increased household sanitation coverage from 73% to 81%. Almost all the latrines in both program and non-pro-gram communities were pit latrines with a mud/wooden cover: in other words, they were almost all unimproved pit latrines 3. There was a marginal increase in the hand washing facilities. Only about a quarter of the household latrines in RWSSP commu-nities had hand washing facilities, compared to 18% in non-RWSSP communities. Hand washing with-out soap at critical times (before eating or prepar-ing food and after using the toilet) was widespread in both RWSSP and non-RWSSP communities. Although households perceived that open-defeca-tion had declined, households without private latrines resorted more to open-defecation and less to the use of public toilets.

The RWSSP had significant impact on redu-cing the incidence of diarrhea among persons of all ages but not among children under five. The self-reported incidence of diarrhea of around 5% among children under five in both RWSSP and non-RWSSP communities was low and similar. The lack of any detectable impact on the diarrhea inci-dence of children under five was due in part to the similarity of sanitation and hygiene practices of chil-dren in both RWSSP and non-RWSSP communities.

3Executive Summary

An ID

EV Im

pact

Eva

luat

ion

That said, compared to the non-RWSSP communities, the RWSSP reduced the incidence of diarrhea among persons of all ages by 7 percentage points or 45%, which is close to the target of a 50% reduction in the incidence of water borne diseases 4.

The RWSSP reduced the time for fetching water and generated time savings although less than what was planned for at appraisal. The travel time spent by households (in RWSSP communities) on fetching water dropped by about 10 minutes per round trip, and by 4 minutes to reach the primary water source. With an average of 2.28 water fetching trips per day to fetch water, the RWSSP saved 23 minutes per day per household, which was substantially less than the appraisal target of 2 hours per day 5. All the same, the RWSSP improved daily time to fetch water significantly in the RWSSP communities compared to the non-RWSSP communities.

The RWSSP had no discernable impact on school enrolment rates, which were already very high (in excess of 95%) in both RWSSP and non-RWSSP communities.

The RWSSP had no significant impact on women’s participation in (self) employment activities, which was low in both RWSSP and non-RWSSP communities. While the time saved fetching water was limited, it was positively associated with women’s participation in employment activities, and was used mainly for domestic activities, including cooking, fetching firewood and other unpaid work. Employ-ment of women as workers in water supply schemes was also very limited. The lack of significant impact on self-employment might be explained by the high competition for the limited time saved, coupled with the lack of well-developed (local) markets for self-em-ployment activities and also the local culture with respect to the role of women.

Sustainability of the RWSSP facilities and services: The institutional, technical, financial, and moni-toring and evaluation capacity constraints of the community limit the sustainability of the RWSSP facilities and services. By enhancing community capacity, especially in creating and ensuring the

functionality of the WaSHCOMs, the RWSSP provided a basis for sustaining its results. The communities effectively participated in the delivery of the RWSSP, but their institutional, technical, financial, and moni-toring and evaluation capacity remained insuffi-cient for operating and maintaining the RWSSP outputs. Although the WaSHCOMs responsible for operating and managing the RWSSP facilities were functional, they were too organizationally, techni-cally and financially weak to carry out this function. The challenges of operating and maintaining the RWSSP facilities were not only technical and finan-cial capacity constraints, but also the shortage of essential spare parts and the limited participation of the private sector. The majority of the RWSSP water points failed to generate sufficient revenue for their operating and maintenance costs. As a result, GoE support was necessary for maintaining the RWSSP facilities. Furthermore, there was no monitoring and evaluation system in support of the RWSSP nor was one planned to contribute to the sustainability of the RWSSP results.

Conclusions

The RWSSP was effective in delivering the water and sanitation infrastructure capacity but less so in build-ing community institutional capacity. The functional-ity of the water and sanitation infrastructure capacity was reduced largely as a result of the breakdowns and idle capacity of some facilities. The program enhanced community institutional and management capacity, particularly that of the WaSH committee (WaSHCOM), whose performance was modest in terms of managing and sustaining the WaSH facili-ties and services.

The RWSSP was successful in producing desira-ble impacts in its key targeted domain of improving access to and the use of improved water sources in the target communities, but not in other areas. The program significantly improved households’ access to and use of improved water sources although E. coli contamination remained an issue at water sources and points of use. However, it had no detectable


impact on daily water consumption, as the self-re-ported daily per capita water consumption was already far in excess of the national standard.

The RWSSP was effective in promoting private latrine ownership and use, thus contributing to less open-defecation, but not in enhancing hand wash-ing at critical times and in reducing the incidence of diarrhea among children younger than five. However, the quality of the latrines and hand washing was far below the expected standards. The RWSSP also succeeded in reducing the incidence of diarrhea in persons of all ages.

The program was also effective in reducing the time spent fetching water; the time saved was mostly used to undertake domestic activities but did not increase children’s school enrolment or women’s participation in self-employment.

The sustainability of RWSSP benefits is unlikely because of institutional, technical, financial and devel-opment information inadequacies and also because the availability of public funds was critical for opera-tions and maintenance of RWSSP facilities.

Recommendations

The key emerging recommendations are for the Bank to do the following:

1. In collaboration with key stakeholders (including the Federal, Regional and Woreda administrations, and WaSHCOMs), develop a clear strategy for sustaining the benefits of the RWSSP. While reflecting the strengths and weaknesses of the woreda and commu-nity institutional arrangements for designing, implementing, and managing WaSH facilities and services, such a strategy should address the following issues:

a. Poor quality water and sanitation facilities and services, and irregular water quality testing and treatment. The main issues are the poor

quality of the facilities, the presence of E. coli in water at the source and point of use, the predominance of unimproved latrine types, the widespread practice of washing hands without soap, and other hygiene practices undermine positive health outcomes.

b. Lack of adequate financial resources for oper-ating and maintenance of the RWSSP water facilities. Current water facility user charges are insufficient for meeting the cost of oper-ation, maintenance and replacement, which contributes to delays in or lack of repairs. Moreover, there is currently no clear strat-egy for ensuring the financial viability of the management of the water points.

c. WaSHCOMs’ weak organizational and mana-gement capacity. Whereas WaSHCOMs are important for sustaining the RWSSP results, they are organizationally and technically weak. In addition, 89% of the WaSHCOM members reported needing (refresher) training. The gender imbalance issue in WaSHCOMs should be addressed.

d. Weak capacity of WSST to better support WaSH-COMs and to effectively participate in monitoring the sustainability of the RWSSP benefits.

2. Support the development and implementa-tion of an effective monitoring, evaluation and learning system to ensure regular, perti-nent data collection, analysis, repor ting and feed-back, especially on RWSSP community WASH results. With no baseline data or effective M&E, the RWSSP missed an opportunity to learn and support the program completion reporting and the impact evaluation study. The Bank could build a sound MEL system for the post-RWSSP through its support for strengthening coun-try systems to support the sustainability of the RWSSP results. The use of smart technologies (GPS-enabled devices, geo-referenced manage-ment tools and smart phones) for MEL should also be explored.

7Management Response

An ID

EV Im

pact

Eva

luat

ion

Introduction

Rural water supply and sanitation coverage in Ethiopia, as at 2005, was low at 24% and 8% respectively. Under the National Water Supply and Sanitation Program (NWSSP), the Government of Ethiopia (GoE) planned to increase the national coverage to 62% and 54% respectively by 2015, through application of the Demand Responsive Approach (DRA) to services deliv-ery. Consequently, the AfDB designed the Rural Water Supply and Sanitation program (RWSSP) to support the GoE to achieve this intended outcome with fund-ing under the Rural Water Supply and Sanitation Initi-ative (RWSSI). Ethiopia was among the five countries selected for initial implementation of this Initiative which sought to assist Regional Member Countries achieve the Millennium Development Goals (MDGs) and the African Water Vision 2025 targets for water supply and sanitation in rural areas.

The impact study falls within the long-term strategy of the Bank involving a commitment to obtain credible evidence on development effects of its interventions through quality evaluations based on sound method-ologies. As part of its 2013–2017 strategy, the Bank’s Independent Development Evaluation (IDEV) commit-ted to undertake a small number of impact evaluations, in collaboration with the Bank’s operations teams and

Regional Member Countries, to promote the evaluation culture. The impact study of the AfDB-supported Ethio-pia RWSSP is the first completed of these evaluations.

Program components

The RWSSP aimed at increasing access to, and use of, improved water supply and sanitation services across Ethiopia. It was approved with a total commit-ment cost of UA 54.24 million, of which the ADF, GoE and communities contributed 80% (UA 43.61 million) in grants, 15% (UA 8.12 million) and 5% (UA 2.510 million), respectively.

The RWSSP was implemented over eight years (2006–2014); three years more than the planned five years (2006–2010). At completion in 2014, the actual disbursement was UA 60.203 million, an 11% increase over commitment costs.

The four components of the program were:

1. Water supply: provision of new and rehabi-litation of existing water supply services and livestock watering troughs. Technology ranged from hand dug wells, drilled wells, protected springs, boreholes, gravity schemes, water

Management Response

Management acknowledges the Independent Development Evaluation (IDEV)’s Impact Evaluation report on the Bank’s Rural Water Supply and Sanitation Program in the Federal Democratic Republic of Ethiopia, implemented from 2006 to 2014. The Evaluation revealed the positive impact on the commu-nities in the targeted areas, however, management acknowledges that the intervention slightly fell short in some aspects of the anticipated impact. Lessons drawn from this evaluation are expected to inform future strategies, policies and programs of the Bank, to improve the Bank’s operations and learning agenda. Already, most of these lessons have been integrated into the newly launched One WASH * National Program (OWNP) in Ethiopia, under the leadership of the Government of Ethiopia (GoE).

* Water, Sanitation and Hygiene.


harvesting, subsurface dams, small piped systems, and pumping systems (hand pumps, motorized, solar or windmill pumps). By the end of the program, a number of improved water sources had been developed including 1,911 hand-dug wells, 2,427 on-spot springs, 1,133 shallow bore holes, 118 rural piped schemes for bore holes, 92 rural piped schemes for springs, and 69 others (AfDB PCR 2015).

2. Sanitation services: provision of latrine faci-lities for schools and health centres; commu-nal latrines for crowded settlement areas and demonstration latrines for associations, coope-ratives and communities. Hygiene and health education campaigns were also conducted. The RWSSI delivered 93% (1,057 latrines) of the expected public sanitation facilities: 171 latrines for schools, 201 for health centres and 685 models for communities; in addition to promo ting household construction of 66,534 private latrines, which was 15% higher than the target (PCR 2015).

3. Capacity building: These involved community sensitization, awareness raising, organization, and training in water supply and sanitation. Capacity building was also carried out at the Federal Ministry of Water and Energy (MOWE) Federal Ministry of Health (MOH), Regional Water Bureaus, Regional Health Bureaus, Woreda Water Desks and Woreda Health Desks and Woreda Support Groups. There was also capa city building for community facilitation teams, local service providers, artisans, health extension workers and spare parts suppliers for deve lopment of supply chains.

4. Program support: at the federal, regional and local government level institutions, and to communities; establishment of a database, annual financial and technical audits; monitoring

and evaluation; and technical assistance to MOWR and MOH.

Evaluation Design

The evaluation study sought to provide credible evidence of program effects on rural communities and households. Given that the program was designed and implemented with no rigorous impact evaluation in mind, no baseline dataset was collected prior to implementation.

As such the evaluation employed an end-of-program quasi-experimental method which is the next sound and rigorous design for impact evaluations. The study specifically applied a matched-DID, that is propensity score matching (PSM) method together with a diffe-rence-in-differences (DID) approach †.

The absence of a reliable baseline data however, limited the use of the matched-DID, which could have further enhanced the robustness of the impact eval-uation estimates. With the current long term stra tegy of the Bank involving a commitment to obtain credible evidence on development effects, program designs are expected to take into account some of these weak-nesses and plan for more robust, rigorous and sound impact evaluations based on randomized control trials (RCT) with adequate baseline data.

Implementation and Impact of the Program

Overall, the program was delivered with substantial delays and moderate cost overruns. Country coordina-tion, stakeholder mobilization and initial preparations fell behind schedule at inception. Management notes however, that the program allocated most of the invest-ment to water supply schemes and sanitation facilities as intended in the Project Appraisal Report (PAR).

† The PSM and the matched-DID methods are quasi-experimental methods commonly applied in observational studies to measure changes caused by an intervention.


An ID

EV Im

pact

Eva

luat

ion

The RWSSP Implementation:

Management commends the GoE and the implementing authorities of the WASH sector in Ethiopia, for their focus in the implementation of this program. Although the RWSSP water points failed to generate sufficient revenue from participat-ing communities to meet operating and maintenance costs, the GoE supported the maintenance of water point facilities in most parts of the communities. In spite of the fact that WASH committees were formed to manage water points and sensitize communities to pay adequate water user fees to cover their water facilities, community payments were very low (mean yearly and mean monthly household fees were about 13.75 and 3.58 ETB ‡ respectively) due to low income levels, and only covered minor repairs. Around 65% of the water point repairs were financed through local government support. Due to the time overrun which resulted in increased investment cost, the GoE and program communities also bore an additional imple-mentation cost of 5.96 million UA when program funds dwindled, demonstrating high sense of commitment in the implementation of the program. On the whole, the RWSSP was effective in delivering 1) Improved WASH services, although not all of it was functional all the time due in part to technical and quality issues, delays in acquiring essential spare parts and limited participation of the private sector; 2) close to 6,810 water supply and sanitation facilities, approximately 80% of target; and, 3) 93% (1057) of expected public sanitation facilities. They were also effective in promot-ing household construction of 66,534 private latrines, about 15% higher than the target.

Management, however, acknowledges the lack of adequate management of water facilities resulting in breakdowns of water facilities in some of the targeted communities. Coupled with shortage of spare parts for facility repairs and limited role of the private sector, the RWSSP was less effective in contributing to the institutional, management, finan-cial and technical development of community WASH committees (WaSHCOMs).

In order to ensure a more effective delivery of these, as well as other rural water supply and sanitation services, management models are to be factored into future program designs to ameliorate recurrence. Lack of spare parts, limited participation of the private sector, low payment of community water user charges, weak capacities of some WaSHCOMs to manage water facilities are issues of note that affected the intervention one way or the other.

Water, sanitation and hygiene associated effects:

The evaluation notes that RWSSP had significant impact on access to and use of improved water sources in targeted communities, and improved household access and use of quality water. Approx. 91% of households had access to and use of improved water sources, which was 69% higher than non-tar-geted groups.

The intervention, however, was not effective in ensuring, safe water for households at water source and point of use at all times partly due to water contamination during transportation and/or storage at home. A third of the time, water was of low quality relative to the recommended WHO stand-ard of having no E. coli bacteria in human drinking water. At point of use, only 33% of the tested sample drinking water in RWSSP households was free of E. coli bacteria.

The incidences of hand washing improved with the program but sadly, not with use of soap or soap substitutes. Individuals washed their hands most of the time before eating and after using toilets but they rarely used soap. This leaves risks for infec-tion and hygiene issues; and needs consideration of people-centered strategies in project designs to spearhead this change. Though the report was not emphatic on the reasons of non-use of soap; whether due to knowledge gap or economic constraints, management recognizes that it is important to

‡ Ethiopian Birr.


enhance measures for sustained community sensiti-zation and animation in future WASH program designs.

Incidence of diarrhea among children under five years old:

The evaluation revealed that the RWSSP had a significant impact on reducing the incidence of diarrhea among persons of all ages, but not among children under five. Compared to the non-RWSSP communities, the RWSSP reduced the incidence of diarrhea among persons of all ages by 7 percentage points or 45%, which is close to the target of 50% reduction in the incidence of water borne diseases in the RWSSP Appraisal Report.

The incidence of diarrhea of around 5% among chil-dren under five in both RWSSP and non-RWSSP commu nities was low and similar. It is understood that the lack of any detectable impact on the diar-rhea incidence of children under five was due in part to the similarity of sanitation and hygiene practices (including hand washing) amongst children in both RWSSP and non-RWSSP communities, indicating that more should have been done in educating the RWSSP Communities.

Time savings, school enrolment and women’s participation in self-employment:

The RWSSP was modestly effective in saving time spent in fetching water and most of the time saved was used for domestic activities including cooking and other unpaid work. The travel time spent by households in RWSSP communities on fetching water dropped by about 10 minutes per round trip. With an average of 2.28 trips per day to fetch water, the RWSSP saved 23 minutes per day per household but this was less than the appraisal target of 2 hours per day. The time savings of 2 hours anticipated short travel distances to water points but actual implementation covered distances within 1.5 km service radius. Population increases could also have led to more time spent in queues at water points.

There was no discernable impact on school enrol-ment rates, which were already very high. The high enrolment rates in both program and non-program communities for both girls and boys is noted to be due to the current rapid expansion of school infra-structure throughout rural Ethiopia coupled with the wide scale adoption of school infrastructure stan-dards that prescribe provision of water supply and gender-segregated sanitation facilities for new schools. Improvements in school sanitation and hygiene facilities in both locations also explain the high enrolment rates.

The RWSSP had no significant impact on women’s participation in self-employment activities. As time saved in fetching water was limited, it did not moti-vate self-employment, though it enhanced domestic activities, including cooking, fetching firewood and other unpaid work.

The key note has been that although the time savings from the RWSSP were limited and perhaps not very conducive for creating economic activities, other external conditions of limited access to credit, resource and output markets, and business training and skills prevented use of the time for economic ventures. This further strengthens the point that several factors come into play to stimulate economic activities and therefore the Bank program designs should take these into account in order to promote the creation of economic activities.

Conclusion

Management takes note of the low areas of impact in some aspects of the program and the associated causes. It is realized that some of these causes are external and behavioral, but efforts will be intensi-fied in the medium to long term to minimize and/or ameliorate such occurrences. Already, most of these factors have been considered in the OWNP currently under implementation in Ethiopia and project imple-menters are supported and monitored closely for sustained compliance.


An ID

EV Im

pact

Eva

luat

ion

For instance, creating access to and use of improved water sources by beneficiaries must be combined strongly with sustained quality provi-sion. Whilst the program had significant impact on access to and the use of improved water sources in selected communities it was not effective in deliv-ering totally safe water for households at water source and point of use. As this relates mostly to behavioral change factors, adoption takes time, resources and concerted effort for effective hygiene education at community levels. This is the work of the village health teams and health inspectors who have to be trained, motivated and monitored in their support of WaSHCOMS. The new water goal under the Sustainable Development Goals framework,

to which the Bank and RMCs subscribe, necessi-tates that water quality aspects are entrenched in the development and monitoring of the delivery of water services.

The level of contamination at the point of water use was indicative of contamination during water transport and storage. The knowledge gained from targeted training sessions are expected to cause changes over time to ensure maintenance of quality water at points of source and use. The key lesson learnt is to ensure that adequate water safety plans encompassing protection of sources, as well as hygienic abstraction, transportation and storage, are in place.


Management action record

Recommendation Management’s Response

RECOMMENDATION 1: Develop a clear strategy for sustaining the benefits of the RWSSP. Such a strategy should address among others the following issues:

Poor quality water and sanita-tion facilities and services, and irregular water quality testing and treatment. The main issues for the RWSSP facilities and services are the poor quality of the facilities, the presence of E. coli in water at the source and points of use, the predom-inance of unimproved types of latrines, widespread hand washing without soap, and other non-hygiene practices that undermine positive health outcomes.

AGREED

Actions:

For the new RWSSI strategy coming up in the year, the Bank will take these deficiencies into account and enshrine approaches to deepen stakeholder coordination and efforts at country-level on quality of water and sanitation facilities. (Q1, 2017)

Options will be identified in the new RWSSI strategy on how the Bank can support governments to entrench these in their service delivery approaches including how to deliver and finance capacity strengthening interventions through health inspectors, Village Health Teams, etc. (Q1, 2017)

Lack of adequate financial resources for the operation and maintenance of the RWSSP water facilities. Current water facility user charges are insuf-ficient to meet the cost of operation, maintenance, or replacement, which contrib-utes to delays or lack of repairs. Currently, there is no clear strategy for ensuring the finan-cial viability of water point management.

AGREED – OWAS is currently developing its Strategic Focus and Priorities in support of implementation of the Bank’s Strategy. One of activities under the three pillars of the strategy is “to Assist RMCs to Develop Sustainable Infrastructure and Inclusive Services for Water Security”. OWAS coordinates the Bank’s Water and Sanitation Sector activities and related aspects of the Bank’s Strategy, and administers three special initiatives, including the RWSSI for inclusive and sustainable services in rural areas.

Actions: – On this strategic pillar, OWAS plans to deepen support to countries to scale up the development of sustainable water and sanitation infrastructure and services. The support among others, will ensure:

Full cycle cost assessments and supporting countries to ensure that at design stage, full cycle costs and sources of financing are identified.

Identification and development of viable management systems for water & sanitation infrastructure at community level with clear water management requirements.

Support for increased private sector participation and innovative financing mechanisms to accelerate infrastructure development and financial viability of water point management. (Q4, 2016)

Integration of income generating activities in project designs to help targeted communities become financially-independent and economically-empowered to sustain their livelihoods in the medium to long terms.

That sustainable water and sanitation infrastructure this is factored into the upcoming RWSSI Strategic framework, and monitored during project preparation and implementation. Spill-over effects anticipated are immense ranging through improvements in household living standards, community development and growth, creation of rural employment and ultimately, increased household incomes. (OWAS, Ongoing)


An ID

EV Im

pact

Eva

luat

ion

WaSHCOMs weak organiza-tional and management capac-ity. WaSHCOMs are important for sustaining the RWSSP results, but they are organiza-tionally and technically weak. Moreover, 89% of the WaSH-COM members reported that they need (refresher) train-ing. The gender imbalance issue in WaSHCOMs should be addressed.

PARTIALLY AGREED: – The efforts of WASH Committees (WASHCOMs) alone cannot sustain the results of the RWSSP. The participation of multiple service providers and actors, including federal, regional and local administrations, health and extension workers, communities, local service providers, and the private sector need encouragement and sensitization. Management models for sustainability of RWSS services are still being tested and improved. WASHCOMs have their benefits but they need long-term government support especially with respect to training and more complex repairs. Other models are being continuously tried out and evaluated, but the role of government in ensuring long term sustainability is still key.

Actions:

The Bank will continue to work with countries and other partners to enhance national systems and processes; identify management models that lead to greater sustainability of services including the life-cycle cost assessments and development; ensure more professionalized service delivery models for rural areas including greater involvement of the private sector; and more institutionalized community capacity building for sustainability of facilities and services, whilst also recognising long term organizational maintenance in policy and procedures for rural water supply and sanitaation. (Ongoing)

Already, gender issues have been strongly integrated into the newly launched One WASH National Program that has started in Ethiopia. OWAS will continue to ensure that a deepened sensitization of women representation in community associations and key management positions. Given that women were represented in about 84% of the WaSHCOMs in the previous program with 35% having women in key management positions, it stands to substantiate that women have interest to participate in community activities and this new program will fully pursue it. (OWAS, Ongoing)

Weak capacity of Water Supply and Sanitation Team ( WSST) to better support WaSHCOMs and to effectively participate in monitoring the sustainability of the RWSSP benefits.

PARTIALLY AGREED: – The water sector in Ethiopia is cognizant of the need for enhanced capacity of their staff in the federal states. Thus, capacity building for the different tiers of service providers is a key component of the One WASH National Program in Ethiopia – the Government-led multi-donor supported water, sanitation and hygiene program. The Bank is one of the partners supporting this sector program.

Actions:

Already, this recommendation has been integrated by OWAS, the Office coordinating and managing the RWSSI, in the newly launched One WASH National program in Ethiopia, and expected to be a continuous focus in the course of implementation with tangible effects from Q4, 2016.

OWAS will scale-out lessons learnt from these practices in all future Bank programs on WASH.


RECOMMENDATION 2: Support the development and implementation of an effective Monitoring, Evaluation and Learning (MEL) system to ensure regular and pertinent data collection, analysis, reporting and feedback especially on RWSSP community WASH results.

Given the lack of baseline data and effective M&E, the RWSSP missed an opportunity to learn and support program completion reporting and the impact evaluation study. Such a missed opportunity should be avoided by having a sound MEL system for the post-RWSSP. The Bank could build such a system through support for strengthening country systems to support the sustainability of the RWSSP results. The use of smart technologies (such as GPS-enabled devices, geo-ref-erenced management tools and smart-phones) for MEL should also be explored.

Agreed – The Bank recently strengthened its focus on country programming to deliver effective operations by rolling out quality assurance and results management tools focusing at the country level. The Bank intends to further deepen its results culture by reinforcing sector M&E and corresponding country M&E through capacity building.

Actions:

OWAS will continue to provide needed guidance to RMCs by participating in country M&E thematic groups, donor coordination platforms and other related fora to support governments in the development of sustainable country-led sector M&E frameworks. (OWAS, Ongoing)

OWAS will maintain continuous dialogue with country counterparts on sector M&E issues. Country office (ETFO) and M&E task force of the department will work together to ensure this. To guide RMCs and project teams to strengthen WASH data availability and quality, and sector M&E framework in general, a three-step procedure is planned consisting of:

Step 1: Undertaking Sector M&E Needs and Gaps Assessment (NGA);Step 2: Selecting and designing actions for strengthening the country’s sector M&E framework;Step 3: Implementing, monitoring and sustaining improvements.

The newly launched One WASH National Program with co-funding from the Bank has a sub-component on M&E financed by the RWSSI-TF. It aims to strengthen the national capacity to monitor and evaluate WASH data more sustainably. Implementing authorities of the program will be required to execute this component to the fullest in order to ensure the strengthening of M&E system. (OWAS, Ongoing)

At global level, the RWSSI is in the process of renewing its strategy for 2016–2020 and will consider mechanisms to deepen and strengthen country and project level M&E systems across the board. (Q1,2017)

As mandated, the Bank’s Statistics Department (ESTA), regularly assesses the national statistical systems of all RMCs, identifies challenges, and proposes assistance packages. The M&E situation as raised, will be assessed and discussions held within appropriate departments in the Bank including ESTA, on approaches to provide support for a sound MEL system for the post-RWSSP in Ethiopia. (Q4, 2016)

Management will ensure that increased attention is given to design and implementation of project logframes in new project designs. Logframes will be results-focused to capture impacts in targeted areas, and also prepared in broad consultation with country partners to facilitate buy-in, ownership and effective coordination. (OWAS Q3, 2016 Onwards)

Bank supervision missions and internal field monitoring will deepen focus on strengthening country M&E systems and processes by ensuring that: M&E protocols are in place; M&E personnel are in place; M&E budgetary allocation is adequate; skilful and modernized data collection mechanisms are adopted; and database systems are developed and functioning to entrench MEL and result measurement and management. Where necessary, logframes of ongoing projects will be reviewed to align with the current results-orientation of the Bank and OWAS. (OWAS Q3 2016 Onwards)

17Background Information

An ID

EV Im

pact

Eva

luat

ion

Background Information

Country context

Ethiopia is a diverse country in many ways, including geography, culture, and ethnicity. In recent years, Ethi-opia had one of the fastest growth rates in the world. Between 2003/04 and 2011/12, the country’s real GDP grew by about 11% per annum. National head count poverty decreased from 45.5% in 1995/96 to 29.6% in 2010/11 and per capita GDP, in nominal terms, rose from USD 387 in 2010/11 to USD 513 in 2011/12. The country has also recorded encouraging results in the social sectors: primary school net enrol-ment rates went from 77.5% in 2005/06 to 85.9% in 2012/13. The under-five mortality rate decreased from 97 deaths per 1000 live births to 59 deaths per 1000 live births.

Over 83% of Ethiopia’s 90 million people live in rural areas. The country has made overall progress in expanding access to improved sources of water and sanitation facilities, however much of the country’s population, especially in rural areas, remains without access to or the use of these services. Access to safe drinking water and access to latrines in 2011 stood at 52% (rural 49% and urban 75%) and 63% (rural 60% and urban 80%) respectively. About 75% of chil-dren’s health problems are associated with poor water supply and lack of sanitation services. Around 17% of under-five mortality is attributed to diarrhea.

Cognizant of the need to address the critical chal-lenges of the water and sanitation sector, the Government of Ethiopia (GoE) has formulated and implemented several policy measures, strategies and programs. In 2001, it formulated and adopted the National Water Resources Management Policy; in 2002, the Water Sector Development Program (WSDP), and in 2003 the Water Supply and Sanitation Master Plan. In 2015, the GoE and its development partners adopted the National Water Supply and Sani-tation Program (NWSSP) with a large rural component,

the National Rural Water Supply and Sanitation Program (NRWSSP) 2005–2015. The NRWSSP was designed to ensure that national MDG targets of 62% and 54% of rural water supply and rural sanitation coverage respectively were met by 2015. With regard to urban and rural coverage, the GoE’s Growth and Transformation Plan (GTP) 2011–2015 set targets of 98% of the population having water supply and 84% of the population having improved sanitation by 2015.

In line with the GoE’s policy, water supply and sani-tation services are decentralized. The Federal Government focuses on formulating national poli-cies, strategies and standards, while the regional and district governments are responsible for managing and delivering water supply and sanitation services to their respective populations.

Description of the Rural Water Supply and Sanitation Program (RWSSP)

The Rural Water Supply and Sanitation program (RWSSP), supported by the AfDB, was one of the major water supply and sanitation development programs in Ethiopia in 2005–2014 6. It aimed at increasing access to and the use of improved water supply and sanitation services in 125 woredas/districts across Ethiopia’s nine regions 7 through a community demand-driven approach to deliver an improved water supply and sanitation services to the selected local communities and strengthening local, district, regional, and national capacity to effectively develop and manage water and sanitation. Program beneficiaries include rural sedentary and nomadic populations, artisans, entrepreneurs, and federal and local governments.

The RWSSP includes hard infrastructure and soft development components. The hardware component consisted of (i) the construction of new water schemes,


rehabilitation of malfunctioning schemes or expand-ing existing schemes, and (ii) building institutional latrines for schools and health facilities, and commu-nal latrines in areas with crowded settlements. The soft aspects included community sensitization, aware-ness raising and facilitation, and capacity building of different water and sanitation services providers at the community, woreda, regional and federal levels. The program also encouraged and supported private sector agents to provide appropriate supportive services to the program implementing agencies.

The delivery of the RWSSP was guided by its imple-mentation manual, which includes, among other things, the community selection process and crite-ria such as poverty level, water supply coverage, and proximity to one another water supply facility, willing-ness to establish a trained 7-member Water Supply, Sanitation and Hygiene Committee (WaSHCOM) which includes at least 30% women, an acceptable facil-ity and management plan, and contribution in cash and kind 8. The participation of the multiple service providers and actors, including federal, regional and woreda administrations, woreda groups, health and extension workers, communities, WaSHCOMs, local service providers, community facilitation teams, arti-sans and spare part dealers was also encouraged. The RWSSP implementation manual stipulates that the communities had to prepare their project proposals on the basis of their needs and interests and submit them to their respective woreda administrations who were responsible for receiving, processing, and accepting or rejecting the proposals.

The RWSSP implementation relied on existing appro-priate institutional structures at the federal, regional, woreda and community levels, mainly comprising

the Ministry of Water and Energy (MoWE), Ministry of Health (MoH) and Ministry of Education (MoE) and their respective regional and woreda Offices. The Woreda Water Supply and Sanitation Team (WSST) under the chairmanship of the woreda administrator was the RWSSP management and implementing agency at the woreda level, supported by the woreda support groups (WSGs).

The RWSSP was approved in 2005 with a total commitment cost of UA 54.24 million, of which the AfDB ADF grant, GoE and communities contributed 80% (UA 43.61 million), 15% (UA 8.12 million) and 5% (UA 2.510 million) respectively. At completion in 2014, the actual disbursement was UA 60.203 million, an 11% increase over commitment costs. This increase was supported by additional contributions from the GoE of UA 3.8 million and community contri-butions of UA 2.3 million, as 0.04% (UA 0.02 million) of the AfDB grant contribution was cancelled 9. The RWSSP was implemented over eight years (2006–2014); three years more than the planned five years (2006–2010).

The RWSSP started delivering completed facilities in 2008 and finished in 2014. By the beginning of July 2013, the program was reported to have constructed 5061 drinking water schemes, or about 85% of the expected total number of schemes (MoWE, 2013). These water schemes include 2048 on-spot springs, 1588 hand-dug wells, and 958 shallow boreholes. In terms of regional distribution, about 84% of the water schemes are in three of the nine program regions – Oromia, Amhara and Southern Nations, Nationalities and Peoples (SNNP) Regions – which accounted for approximately 87% of the number of people directly benefiting from the RWSSP (MoWE, 2013).

19Background Information

An ID

EV Im

pact

Eva

luat

ion

Theory of change underlying the RWSSP

From the RWSSP theory of change, depicted in Figure A1.1 (Annex 1) and detailed in the evalu-ation inception report, the program activities and processes would lead to positive health, educa-tion and economic benefits for the intended rural communities and households through multiple and interlinked change pathways. The development hypotheses and assumptions include the following:

Development hypotheses:

Health outcomes: the RWSSP interventions would create adequate barriers against the transmission of human excreta pathogens, the source of various WaSH-related diseases. In this regard, the RWSSP would deliver improved water sources that would provide accessible, adequate and safe water for human consumption to the intended communities and households who would also have adequate access to and use improved sanitation and hygiene practices including better toilet facilities and hand washing with soap or soap substitutes as per the required standards.

Work and education: By expanding access to and use of improved sources of drinking water supply and sanitation services, the RWSSP would raise women’s (self) employment, and reduce girls’ dropouts from school; women and children, particularly girls, bear the burden of fetching water in rural Ethiopia (CSA, 2012). The women would allocate at least 30% of their

saved time from fetching water to productive employ-ment. In addition, access to a private latrine would save time looking for a safe, alternative place to defe-cate. Moreover, better health would enhance women’s employability and productivity and the likelihood of increasing remuneration from paid work.

In the case of children, the time saved fetching water would increase the probability school enrolment and retention. The education benefits would also result from the health benefits of the program and/or from an improvement in the economic status of women (Miguel and Kremer, 2014; Waddington et al., 2009).

The assumptions include:

❙ Participating communities would actively participate in the selection of water supply facility, design for its construction, and sites for installing the facility.

❙ Participating communities would adequately contribute in cash and kind to the facility construc-tion in their localities.

❙ The water and sanitation and hygiene commit-tees (WASHCOs) would be effective and gender sensitive in managing improved water systems and in promoting the construction of better private latrines and hygienic practices in water handling and hand washing.

❙ The facilities provided by the RWSSP would be technically sound and fit well with the local context and community needs.


Objectives, Questions and Scope of the Evaluation

Objectives of the impact evaluation

The main objective of this evaluation was to provide credible estimates of the effects of the AfDB-sup-ported Rural Water Supply and Sanitation Program 2006–2014 (RWSSP) interventions on participating communities and households, and to validate the underlying theory of change. This was done primarily to (i) account for program development outcomes to the AfDB Board and the GoE, and (ii) provide relevant recommendations to AfDB Management (OWAS) for sustaining the RWSSP facilities and services and for informing the design and implementation of similar future programs. The evaluation study was also to help IDEV build the evidence base for the 2015 Ethiopia CSPE and to learn from impact evaluation.

The main audiences for the evaluation include the AfDB – CODE/Board, Department of Water and Sani-tation (OWAS), ETFO, and IDEV – the GoE (MoFED; MoE; MoH; MoWE), and other development partners.

Evaluation questions

The evaluation questions focus on RWSSP operational effectiveness, impacts, and sustainability:

RWSSP operational effectiveness

1. What types of water supply, sanitation and hygiene promotion interventions were used?

2. Have communities had the capacity training to provide safe water supply and basic sanitation and hygiene promotion services?

Water and sanitation-associated impacts

Safe water access and use

3. Is the drinking water safe at the source and point of use (compared to WHO norms)?

4. Do households apply principles of hygiene in handling their water at home?

5. What has been the RWSSP effect on access to and the use of improved water sources?

6. What has been the RWSSP effect on human water consumption?

Sanitation and hygiene

7. What has been the RWSSP effect on the use and private ownership of toilets? Is open defecation practiced in the village? If so, did the extent of this practice change over recent years?

8. What has been the RWSSP effect on hand wash-ing practices at critical times?

Incidence of diarrhea among children under five years old

9. What has been the effect of the RWSSP WaSH interventions on the incidence of diarrhea among children under five years?

Time savings, school enrolment and women’s participation in self-employment

10. Who is responsible for fetching water?

21Objectives, Questions and Scope of the Evaluation

An ID

EV Im

pact

Eva

luat

ion

Time savings and use

11. What has been the RWSSP effect on the total time spent fetching water and traveling to the primary water source?

12. What have been the daily savings in time and how has the time been otherwise used?

School enrolment

13. What has been the effect of the RWSSP WaSH interventions on current school enrolment rates, particularly for girls?

Women’s participation in alternative work

14. What effect has the RWSSP WaSH intervention had on women’s participation in (self) employment?

Heterogeneity of the RWSSP effects across communities

15. Have impacts been similar across target commu-nities that differ in economic status, agro-ecology, distance to town, population density, etc.?

Sustainability

16. Are the RWSSP results sustainable?

a. Do the communities have the capacity to perform the requisite functions?

b. Are the facilities that have been provided tech-nically sound?

c. Are the facilities and services that have been provided financially viable, including maintenance and longer-term replacement of infrastructure?

d. Do the institutional arrangements provide for adequate monitoring?

Scope of the impact evaluation

Program components and scale of interven-tion to be evaluated: the primary focus was on the outcomes/impacts realized among the final intended beneficiaries – the communities and households. This evaluation therefore examines the combined impact of all resources invested at the community level in the various program interventions namely, water supply, sanitation and hygiene interventions and commu-nity capacity building. By contrast, the study does not assess any institutional capacity-building support provided by the program to the implementing agen-cies at different levels.

Timeframe. Although the AfDB RWSSP completion date was extended to 2014, the impact evaluation covers only those schemes that were completed up to July 2013 in order to allow for stabilized impact. To give impacts a reasonable amount of time to materi-alize, this study covered communities that participated in the program until end of June 2013.

Geographic coverage. This study was conducted in Amhara, Oromia, and Southern Nations Nationalities Peoples (SNNP) regions, which account for 41, 29, and 24 for the total 125 RWSSP woredas, respectively. As these regions cover 75% of the total RWSSP wore-das, these three regions accounted 2013 for about 87% of the total number of persons benefiting from the RWSSP in June.

23Methodology

An ID

EV Im

pact

Eva

luat

ion

Methodology

Impact evaluation design

The fundamental challenge of the study was to iden-tify and quantify the counterfactual situation; “what would have happened to the outcome variables of interest in RWSSP communities had the program not been implemented”.

RWSSP had several components delivered through a demand-driven approach rather than being based on a random assignment of participants. Further-more, the RWSSP was neither designed nor imple-mented with impact evaluation in mind. No baseline, or complete and credible outcome dataset was collected during implementation. As a result, the study employed quasi-experimental methods, which are commonly applied in observational studies to measure changes caused by the program. In particu-lar, this study applied a propensity score matching (PSM) method (Rosenbaum and Rubin, 1983; Heck-man et al., 1998) together with a difference-in-dif-ferences (DID) approach, or matched-DID. The PSM and the matched-DID methods take into account the pre-RWSSP intervention observable community char-acteristics in estimating the impacts of the RWSSP to minimize the estimation bias stemming from observa-ble heterogeneity (Heckman et al., 1998; Lokshin and Yemtsov, 2005; Smith and Todd, 2005).

As recommended by Heckman et al. (1998), the following tasks were carried out to increase the comparability of sample observations made in RWSSP and non-RWSSP communities. First, both program and non-program sample observations were drawn from the same geographic locations sharing similar markets and socioeconomic envi-ronments. Second, similar survey instruments were used to collect relevant data from both groups. Third, data from both groups were collected during the same period by the same interviewers. As a further check on robustness, the study also estimated the

impacts of the RWSSP using a weighted-least-squares (WLS) regression, wherein the program and non-program observation units are weighted by the inverse of estimated propensity scores to estimate impact (Hirano et al., 2003).

Despite these precautions, the PSM method could still be biased in the presence of unobserved character-istics that influence the participation/or non-partici-pation in and outcomes of the RWSSP. In this regard, to address the vulnerability of the estimated PSM results to unobserved characteristics, the Rosen-baum’s sensitivity test was applied (Rosenbaum, 2010). This study also examined whether the impacts of the RWSSP varied with participating communities’ pre-existing and exogenous characteristics.

The study used both descriptive and econometric analyses.

The definitions and measurements of outcome varia-bles are summarized in Annex 2 (Table A 2.1).

Sample size and sampling procedures

The empirical data to estimate program impacts were collected from 228 communities and 2736 households in RWSSP and non-RWSSP woredas. The sample observations are distributed across 38 woredas of three administrative regions – Oromia, Amhara and SNNPR. Taken together, the three regions account for over 85% of the total number of RWSSP direct beneficiaries in the country. Within these regions the woredas were randomly selected based on a probability proportional to size sampling procedure whereby size refers to the number of people within regions benefiting from the RWSSP. Next, an equal number of RWSSP and non-RWSSP sample communities and households were randomly selected in the selected woredas. Table 1 presents the distribution of the sample observations across


the three regions. The descriptions of sample size determination and sampling procedures are provided in Annex 3 (Table A 3.2).

During the field survey, three woredas in SNNPR that had initially been included in the sample –Badewa-cho (Hadiya Zone), Gofa Zuria (Gamo Gofa Zone) and Damot Gale (Wolaita zone) – were found to have no RWSSP interventions 10 and were therefore replaced with three new woreda samples – Bulea (Gedio zone), Borecha (Sidama zone) and Sodo Zuria (Wolayta zone) – selected from the list of RWSSP woredas in the SNNPR region.

The study also drew random samples of 114 water points in the chosen community samples and collected facility level data from a random sample of 74 primary schools and 73 health centers in the study areas. The school and health center samples were also taken from within the same sampled woredas and nearly half the schools and health centers were chosen in non-program areas in the woreda.

Survey instruments and implementation of the field survey

The main data used for this study were collected using household, community and water-point ques-tionnaires. Supplementary data were collected from primary schools and health centers in the study areas. Household questionnaires generated data on basic demography and location, and on various

Region Treatment sample Comparison sample Total

Communities Households Communities Households Communities Households

Oromia 54 648 54 648 108 1296

Amhara 36 432 36 432 72 864

SNNPR 24 288 24 288 48 576

Total 114 1368 114 1368 228 2736

Table 1: Equal distribution of representative evaluation sample size

indicators of program impact on water supply, sani-tation and hygiene, health, education, and employ-ment. They also captured data on household assets and access to other public facilities. Community questionnaires were used to gather relevant community-level variables, including pre-program characteristics of the sampled communities. Water-point questionnaires were used to gather data on several features of primary water source used by the sampled RWSSP and non-RWSSP communities and data on a range of water-point management issues and the participation of the local popula-tion. The school questionnaire collected data on school enrolments trends for boys and girls and on the availability and use of water supply and sani-tation services in the selected primary schools. The health-facility questionnaire was used simi-larly, to collect administrative data on outpatient visits caused by diarrhea and on water supply and sanitation facilities at the health centers. Survey questionnaires were tested and administered by experienced, trained enumerators and supervisors. Primary data collection took place during November and December 2014.

Limitations

The main limitations of the study concern selection bias and data quality.

Selection bias from observed and unobserved heterogeneities can distort impact evaluation

25Methodology

An ID

EV Im

pact

Eva

luat

ion

estimates and thereby reduce their credibility and reliability. Any bias from the non-random assign-ment of communities to participate or not to partic-ipate in the RWSSP was minimized by using PSM appropriately and carefully and to a limited extent the matched-DID (see Impact evaluation design). The absence of a reliable baseline data set, however, limited the use of the matched-DID, which would have further enhanced the robustness of the impact evaluation estimates. Regarding the bias from the unobserved heterogeneity, the results of the Rosen-baum’s sensibility test show that the PSM estimates of RWSSP impacts were not vulnerable to such bias.

Data quality: The study used recall and self-reported data, among other things, which could be biased and distort impact estimates. To mitigate this, the study used structured questionnaires and checklists that were pilot-tested and administered by experienced, trained enumerators and supervisors. The use of multiple data sources also facilitated data triangula-tion. Regarding the incidence of diarrhea among chil-dren under five, the study had to rely on a two-week data collection that was complemented with adminis-trative support from health centers. The data collection period was too short to capture the seasonal variability of this outcome variable.

27RWSSP Implementation and Impacts

An ID

EV Im

pact

Eva

luat

ion

RWSSP Implementation and Impacts

RWSSP Implementation

This section describes the extent of the opera-tional effectiveness of the RWSSP, focusing on the responses to two evaluation questions concerning RWSSP products and services.

❙ What types of water supply, sanitation and hygiene promotion interventions were used?

❙ Have communities had the capacity training to provide safe water supply and basic sanitation and hygiene promotion services?

The RWSSP was effective in delivering capac-ity output, especially in improved water supply and sanitation services, although not all of it was functional due in part to technical quality issues and breakdowns. It was also less effective in

contributing to the institutional, management, financial and technical development of the community. The overall program was delivered with substantial delays and moderate cost over-runs. Whereas the deliveries of the RWSSP were variable, they did cover the three core program compo-nents – water supply, sanitation services and commu-nity capacity building. The AfDB program completion report indicates that at closing, the RWSSP was able to deliver about 80% (6810) of the target number of water supply and sanitation facilities and all planned training for the WaSH team and WaSHCOM members (AfDB PCR 2015) .

On water supply, the program developed a number of improved water sources including 1911 hand-dug wells, 2427 on-spot springs, 1133 shallow bore holes, 118 rural piped schemes for bore holes, 92 rural piped schemes for springs, and 69 others (AfDB PCR

Figure 1: Predominance of public taps/standpipes in RWSSP communities

Source: Water Point Survey Data 2014.

Percent of sample RWSSP communities

Water sources/points

50

40

30

20

10

Public tap/standpipe Shallow wells Hand-dug wells Protected springs

0


2015) 11. The water point survey of a sample of RWSSP communities revealed a similar mix of improved water supply facilities, including public taps or standpipes, protected springs, hand-dug wells and shallow wells (see Figure 1).

From Figure 1, the public tap or standpipe was the most common source of drinking water in RWSSP communities, followed by protected springs and hand-dug wells. About 70% of all household inter-viewees in the RWSSP communities reported that their primary water sources were fully functional, whilst 25% of the household interviewees considered their primary water sources to be non-fully functional throughout the year because of technical problems. The remaining (5%) reported that their primary water sources were out of service for some time during the 12 months preceding the survey.

Approximately 38% of the RWSSP water facilities have experienced at least one breakdown since their construction. The primary causes of breakdown included the poor quality of construction and lack of proper management. Nearly 70% of the recent facility breakdowns have been adequately repaired and are now functional. At the time of the survey, however, about 11% of the water points remained unrepaired. A shortage of spare parts and other necessary materials for facility repairs was one of the main challenges reported by the WSST. About 60% of the time, the cost for repairing broken facil-ities came from external sources including the government. The WSST did most facility repairs, which were rarely done by the WaSHCOMs. The role of the private sector was also limited.

Facility breakdowns notwithstanding, the water point survey revealed that the households used more than two-thirds of the capacity of the RWSSP water points (see Figure 2 in following page). Figure 2 shows that household use of the RWSSP water points varied by water point type. Shallow wells serviced more house-holds than was planned, whereas the rest of the water points were used by fewer than the planned number of households on average.

The RWSSP also delivered 93% (1057) of the expected public sanitation facilities; 171, 201 and 685 for schools, health and model/communal facil-ities respectively, in addition to promoting the house-hold construction of 66,534 private latrines, which was 15% higher than the target (AfDB PCR 2015).

In tandem with the water supply and sanitation facil-ities, the RWSSP contributed to building the water supply, sanitation and hygiene-related software capacity of the target communities and WaSH team members. It promoted the creation by communities of WaSHCOMs for nearly all (96%) of the improved water supply facilities and also provided training for the WaSHCOM and WASH committee members, espe-cially on water facility operation and maintenance and improved sanitation and hygiene services. Approxi-mately 55% of all beneficiary communities reported that they received training on a range of topics includ-ing safe water transport, storage and handling, build-ing private latrine, personal hygiene.

Most WaSHCOMs had five to seven members elected by their communities to work with the Woreda Water Supply and Sanitation Team (WSST) on matters relat-ing to water supply, sanitation and hygiene, and to manage the water facilities. Women were represented in about 84% of the WaSHCOMs but only 35% of them had women in key management positions including chairpersons. In sum, only 45% of the WaSHCOMs had the minimum target (35%) of women representa-tion (see Annex 5). Furthermore, the staff employed at the water points was most often men (70%).

About 57% of the WaSHCOMs received capacity building training from the WSST and other agen-cies during the two years preceding the survey. The WaSHCOMs perceived the training to be useful (87%), and expressed a need for refresher training (81%). About 39% of the WaSHCOMs also reported that they had offered training to water users in their communities on community organization and plan-ning (44%), sanitation and hygiene practices (32%), and construction and service supervision (16%) (Annex 5, Table 5.2).


An ID

EV Im

pact

Eva

luat

ion

About 73% of the WaSHCOMs organized at least one meeting of its membership during the 12 months preceding the survey, and nearly 62% of the WaSH-COMs had general meetings with water users in their respective communities. Approximately 79% of the WaSHCOMs kept records of water users and of facility water sales revenues and expenses, and also minutes of meetings.

The WaSHCOMs were also helpful in pushing the targeted communities to pay their cash and/or kind contribution for facility construction, and in collecting water user fees. In this respect, about 96% of the treatment communities had contributed resources in kind and/or in cash to the construction of their water infrastructures. Less than half (45%) of the WaSHCOMs reported that households paid fees for water use at the time of the field survey (Annex, 5, Table 5.3).

Water, sanitation and hygiene associated effects

Safe water access and use

The section responds to four questions to bring out the effects of the RWSSP on water quality, access and use.

❙ Is drinking water at the source and point of use safe?

The RWSSP was not effective in delivering at all times safe water for households at the water source and point of use. Water quality tests detect-ing the presence of E. coli bacteria in water taken from a random sample of water sources and at points of use in both RWSSP and non-RWSSP areas reveal that at least one third of the time, the water was of low quality relative to the recommended

Figure 2: Substantial household use of RWSSP drinking water points

Source: Water point survey data 2014.

0

20

40

60

80

100

120

Public tap(N=51)

Shallow well(N=18)

Hand dug(N=21)

Protected spring(N=22)

Average number of household using the waterpoint

Water points

Africa

North Africa


E. coli count per 100 ml RWSSP communities Non-RWSSP communities

Number of water points at which water quality was tested

Number of households whose drinking water quality was tested at the point of use (N=227]

Number of water points at which water quality was tested

Number of households whose drinking water quality was tested at point of use (N=229]

0 35 [59.32%] 74 [32.6%] 13 [24.53%] 31 [13.54%]

1–10 16 [26.67%] 68 [29.96%] 14 [26.42%] 68 [29.69%]

11–100 6 [10%] 55 [24.23%] 13 [24.53%] 69 [30.13%]

101–300 1 [1.67%] 14 [6.17%] 2 [3.77%] 8 [3.49%]

>300 2 [3.33%] 16 [7.05] 10 [18.87%] 53 [23.14%]

Total 60 [100.00%] 227 [100.00%] 53 [100.00%] 229 [100.00%]

Table 2: High water quality at the source relative to point of use

Table 3: Household water storage and handling

Source: Computed using water quality test data from RWSSP woredas/districts.

Source: Computed Based on Data from the Household Survey 2014.

RWSSP Non-RWSSP Difference

Water handling practices

Number Percent Number Percent Percent

Water container used for fetching water is clean

1262 92.32% 1201 88.24% 4.08%

Water container used for storing water at home is clean

1228 91.17% 1130 84.01% 7.16%

Water container is kept above floor level at home

665 48.68% 632 46.57% 2.11%

Water container has a lid/cover

1185 86.69% 1192 87.71% –1.02%

How often is the water container cleaned

Daily 767 56.19% 781 57.34% –1.15%

Weekly 529 38.75% 51 38.25% 0.50%

Monthly 40 2.93% 40 2.94% –0.01%

Bi-annually 14 1.03% 8 0.59% 0.44%

Rarely 15 1.1% 12 0.88% 0.22%


An ID

EV Im

pact

Eva

luat

ion

Table 4: Significant impact of RWSSP on access to an improved water source, but not on daily water consumption

Source: Computed Based on Data from the Household Survey 2014.

*** Statistically significant at the 1% level. The magnitude of the estimated coefficient shows percentage point.

Outcome variable Estimated impact Standard error

Access to improved water source 0.687*** 0.050

Daily per-capita water use for all domestic purposes 3.003 2.977

Daily per-capita water use for human drinking and cooking 1.067 0.943

WHO standard of no E. coli bacteria in human drink-ing water (Table 2). However, overall water quality was better in the RWSSP communities than in the non-RWSSP communities at both the water source and point of use. Of the 60 water points tested for water quality in the RWSSP communities, only about 59% were free of E. coli (compared to 25% in non RWSSP communities), and therefore safe for human drinking. The implication was that not all RWSSP water points were providing safe water for human drinking due, in part, to the fact that the RWSSP water sources were rarely disinfected or chlorinated (Annex 5, Table 5.4). However, the household survey data showed that 91% of households that fetched water from the RWSSP sources reported that they were satisfied with the water quality (Table A4.2 in Annex 4).

At the point of use, the water was free of the total E. coli bacteria in only 33% of the sample RWSSP house-holds whose drinking water was tested (Table 2). This indicates water contamination during transportation and/or storage at home. However, water contamina-tion in most of the water sources and points of use in the program areas was marginally low. Drinking water in about 27% of water sources and in nearly 30% of point of use sites have bacterial contamination 1–10 total E. coli per 100 ml of water sample (Table 2).

❙ Do households apply principles of hygiene in handling their water at home?

The adoption of household water handling prac-tices and hygiene principles promoted by the

RWSSP was high in both RWSSP and non-RWSSP communities, although slightly better in the former. Table 3 shows the high uptake of house-hold water handling practices at home in RWSSP and non-RWSSP communities. Water containers used for fetching and storing water at home were observed by the field data collectors and reported as clean approxi-mately 92% of the time among the sample households in program communities, compared to approximately 86% in non-RWSSP communities. About 87% of the sample households reported storing water for human consumption in a container with a lid or cover, and over 95% of the sample households reported cleaning their water storage containers daily or weekly. Water stor-age containers were kept at home above floor level less than 50% of the time.

❙ What has the RWSSP effect been on access to and the use of improved water sources?

The RWSSP made a positive difference in enhanc-ing household access to and use of improved water sources. Table 4 shows the impact of the RWSSP to be statistically significant on access to improved water source, but not significant on per capita water consumption by households. The program has significantly increased household access to an improved water source by about 69% across RWSSP communities. At the time of the field survey, about 91% of the sample households in the program communities reported that they fetched water from improved water sources. In the absence of the RWSSP, this figure would have remained at about 22% (Table A6.5, Annex 6).


❙ What has been the RWSSP effect on human water consumption?

The RWSSP had no detectable impact on daily per capita water use. With respect to daily per capita domestic water use, the RWSSP had no statistically significant effect (Table 4). Although water consump-tion was similar for both groups, the treatment group consumed, on average, better quality water as a result of the program. For households in RWSSP communi-ties, per capita water use for all domestic purposes (i.e. drinking and cooking, hygiene, cleaning utensils, and washing clothes) and for human drinking and food preparation was around 34 liters and about 8 liters, respectively. The corresponding amounts of water for households in non-RWSSP areas were about 31 liters and 7 liters, respectively. The reported daily per capita water use far exceeded the RWSSP target of 15 liters for rural water supply. In general, the amount of self-reported human water consumption was high in both RWSSP and non-RWSSP communities.

Household sanitation and hygiene

❙ What has been the RWSSP effect on the use of toilets, private, communal, in schools and health facilities? Is open defecation practiced in the communities and if so, did the extent of this prac-tice change since the program?

The RWSSP was successful in promoting house-hold ownership and use of latrines, but less so in totally eliminating open-defecation. The proportion of household accessing private latrines increased significantly because of the RWSSP. The

program completion report indicated that 66,534 household latrines had been built in the RWSSP communities in addition to the 1057 public sani-tation facilities directly delivered by the program (AfDB PCR 2015). Furthermore, the proportion of households using latrines was significantly larger in RWSSP communities relative to those in the non-RWSSP communities. Table 5 shows the statistically and positive significant impact of the RWSSP on household ownership and use of latrines. The program increased household sanitation cover-age from 73% (comparison outcome) to 81%, and unconditional private latrine use from 64% to 74%, or from 85% to 90% conditional on having a private toilet (Table A6.8, Annex 6) in the RWSSP communi-ties. About 99% of all latrines in both program and non-program communities were pit latrines with a mud/wooden cover (91% and 92% respectively) or slabs of concrete (8% and 7% respectively). More-over, 25% of the household latrines in the program communities had a hand washing facility compared to 18% in the non-program communities. Among households without private latrines, only about 5% accessed communal or public toilets and about 91% use bush/open air for defecation (Table A4.6, Annex 4). When asked to report what has happened to the number of people practicing open defeca-tion in their community, about 79% of household respondents reported a decrease in recent years (Table A4.6, Annex 4).

❙ What has been the RWSSP effect on hand washing practices at critical times?

The incidence of washing hands at critical times (i.e. before eating and preparing food and after

Table 5: Significant impact of RWSSP on household latrine ownership and use

** Statistically significant at the 5% level.


Latrine ownership 0.082** 0.041

Use of private latrine among the total households 0.098** 0.043

Use of private latrine among households conditional on having it 0.052 0.034


An ID

EV Im

pact

Eva

luat

ion

Table 6: No significant impact of RWSSP on the incidence of diarrhea in children under five


Estimated impact Standard error

All persons –0.07*** 0.023

Children under five years of age 0.004 0.028

using the toilet) was widespread despite the fact that soap or soap substitutes were not used. Figure 3 in the following page, shows the high incidence of reported hand washing among household members at critical times in both RWSSP and non-RWSSP communities, which were similar in both communities. Figure 3 shows that individuals washed their hands most of the time before eating and after using toilets. Hand washing behavior before eating food did not vary much in adult men and women or young children, but did differ while preparing food and after using the toilet. In addition, the individuals rarely used soap when wash-ing their hands (see Annex 4 Table 4.7).

Incidence of diarrhea among children under five

❙ What has been the effect of the RWSSP WaSH interventions on the incidence of diarrhea among children under five?

The RWSSP had a noticeable impact on the inci-dence of diarrhea among people of all ages, but not on children under five. The study compared the incidence of diarrhea using a two-week period preceding the survey for reporting. The impact of the RWSSP on child diarrhea was not statistically signifi-cant (Table 6). In addition, the self-reported under-five

Figure 3: Self-reported incidence of hand washing at critical times

0%

20%

40%

60%

80%

100%

120%

Adul

t men

(1

5 an

d ab

ove)

Adul

t men

(1

5 an

d ab

ove)

Adul

t men

(1

5 an

d ab

ove)

Adul

t wom

en

(15

and

abov

e)

Adul

t wom

en

(15

and

abov

e)

Adul

t wom

en

(15

and

abov

e)

Child

ren

(6–1

4)

Child

ren

(6–1

4)

Child

ren

(6–1

4)

Before eating Before food preparation After using toilet

Source: Household Survey Data 2014.

RWSSP Non-RWSSP


diarrhea incidence was actually and remarkably low and similar in both program (5.3%) and non-program (4.8%) communities. Moreover, the analysis of the secondary data collected from health centers in the study areas indicates that the share of under-five chil-dren visiting a health center because of diarrhea does not show a significant difference between program and non-program communities (Table A4.9, Annex 4). Consequently, the failure of the program to reduce the incidence of diarrhea incidence among children under five could be linked to the similarity of sanita-tion and hygiene conditions to which both the treat-ment and comparison groups were exposed (Figure 3). Nonetheless, the program resulted in a significant reduction of 7%, equal to 45% of diarrhea incidence among persons of all ages in the RWSSP communi-ties with (matched) 0.087 in program communities and 0.157 in non-program communities (Table A6.9 in Annex 6).

Time savings, school enrolment and women’s participation self-employment

❙ Who is responsible for fetching water?

Water fetching and transportation. In the RWSSP communities, women and girls are primar-ily responsible for fetching water, and typically transported water on their heads. When asked to indicate the main person(s) in their house-holds responsible for fetching water, respondents

mentioned adult women (18 and above) and young girls (younger than 18) nearly 49% and 23% of the time respectively. Adult men and young boys were only mentioned about 14% and 16% of the time respectively (Annex 4, Table A4.5). As expected, the main means of water transportation from the primary water source to home was carrying it on the head. The photo below provides an example of women transporting water from source to home. However, a significant proportion of households (82.84%) in the program communities also used pack animals to transport water.

Time Savings and Use:

❙ What has been the RWSSP effect on the total time spent fetching water and traveling to the primary water source?

The RWSSP was modestly effective in reduc-ing the time spent fetching water. Households in RWSSP communities spent less time fetching water than those in non-RWSSP communities, includ-ing travel to and from the primary water sources. The average household time for fetching water decreased by ten minutes per round-trip, and by four minutes for travel to reach the primary water source in RWSSP communities compared to non-RWSSP communities 12.

❙ What have been the daily savings in time and how has the time been otherwise used?

Table 7: Significant impact of RWSSP on time spent on fetching water (in minutes)

*** Statistically significant at the 1% level. ** Statistically significant at the 5% level.


Matched single-difference estimate

Travel time to reach water source –5.586** 2.139

Total time spent on per round-trip for fetching water –12.510** 5.410

Matched double-difference estimate

Change in travel time to reach water source –4.423*** 1.330

Change in total time spent on per-round trip for water fetching –9.984*** 2.615


An ID

EV Im

pact

Eva

luat

ion

Table 8: No significant impact of RWSSP on current school enrolments


All children (6–14 years of age) 0.005 0.009

Boys (6–14 years of age) 0.002 0.013

Girls (6–14 years of age) 0.005 0.011

The RWSSP was modestly effective in saving time spent fetching water and most of the time saved was used for domestic activities. Table 7 above, shows the statistically significant impact of the RWSSP on time spent to fetch water. Estimated results from the single-difference and double-dif-ference approaches indicate that the RWSSP has significantly reduced the time needed to collect water. Compared with households in non-RWSSP communities, those in RWSSP areas needed less time not only to reach their water source, but also per round trip for fetching water. More specifically, the program reduced travel time to reach the water source by about 4 minutes (double difference) to 6 minutes (single difference) minutes. Likewise, households in program communities gained time from reducing the total time spent per round trip for fetching water by 10–13 minutes. As a result of the RWSSP, households in the program communi-ties were able to save around 10 minutes on every trip to fetch water. Given an average of 2.28 trips to fetch water per day, a household would save around 23 minutes per day fetching water.

The amount of time saved proved to be considera-bly less than the 2 hours per day that were expected during the AfDB program appraisal. The time spent on travel to primary water sources (10 minutes) indi-cates that the distance was within the target of the 1.5 km service radius. Nonetheless, the total time spent per round-trip including time spent waiting and filling containers at the source (41 minutes) was substantial (Table A6.6, Annex 6). According to households inter-viewed, the time that was saved fetching water was mainly used for domestic activities, including cooking and other unpaid work.

School enrolment

❙ What has been the effect of the RWSSP WaSH interventions on current school enrolment, particu-larly for girls?

The RWSSP had no detectable impact on girls’ current school enrolment. Its estimated impact on children’s current school enrolment was not statisti-cally significant (Table 8) 13. The very high enrolment rates (above 95%) in both the program and non-pro-gram communities for both girls and boys (Table A6.10, Annex 6) could be due to the rapid expansion of school infrastructure throughout rural Ethiopia in recent years (MoE, 2012). Improvements in school sanitation and hygiene facilities in both locations could also explain the high enrolment rates. The school survey data indi-cated that latrine facilities for students in program and non-program areas were available in 95% of the schools (Table A4.9), and that over 70% of the facili-ties were functioning.

Women’s participation in alternative work

❙ What effect has RWSSP WaSH interventions had on women’s participation in (self) employment?

The WaSH interventions had no detectable impact on women’s participation in self-employment. The RWSSP’s effect was not statistically significant even though its sign was in the expected direction (Table 9). Although the proportion of women work-ing in (self) employment activities was higher in program (13.6%) than non-program (9.2%) commu-nities (Table A6.7, Annex 6), the RWSSP’s impact on employment was not statistically significant. The


implication was that time saved fetching water was a key determinant of women’s self-employment. It could also mean that the time saved was too small to meaningfully contribute to women’s self-employ-ment, and/or that conditions were unfavorable such as the lack of or limited access to credit, resource and output markets, and business training. There was also very limited employment of women as workers in the RWSSP water supply schemes.

Heterogeneous impacts of the RWSSP

❙ Have impacts been similar across communi-ties that differ in economic status, agro-ecology, distance to town, and population density?

The heterogeneity of the RWSSP impact on economic status, region, agro-ecology and distance to town showed mixed results. Tables A6.12–A6.14 (Annex 6) show some noticea-ble evidence of heterogeneity in RWSSP impacts. Controlling for the number of households, commu-nities with larger population densities appear to have a significantly larger proportion of households with access to improved water sources. The effect was greater in poorer communities; the proxy for this was participation of the community in the productive safe-ty-nets program, which was a GoE social protection program targeting poorer and food-insecure locali-ties. Communities with health posts or development office agents tended to have a larger proportion of households that had access to improved water source. Moreover, compared with SNNPR, communi-ties in Oromia had a significantly larger proportion of

households with access to improved water sources. There was, however, no statistically significant vari-ation in the proportion of households with improved water sources over distance to district capitals and across different agro-ecologies.

The impact of the RWSSP on reducing the time spent fetching water varied with the number of households living in the communities, the location and percent-age of agricultural land. Reduced time needed to reach water source was greater in communities with larger numbers of households, controlling for popu-lation density. However, in communities with a larger percentage of agricultural land, reductions in travel time to reach water source were smaller. The program had a significantly larger impact on reducing the total time spent per round trip for fetching water in middle altitude/woinadega than in low altitude/kola areas.

As expected, the proportion of households owning private latrines was lower in relatively poor commu-nities; the proxy for this is that these households were targeted by productive safety nets. Since the households were expected to supply the latrines themselves, poor households might lack adequate financial and material resources to invest in latrine construction. The impact of the program on private latrine ownership was larger in middle- and high-altitude areas. Compared with SNNPR, communities in Amhara and Oromia regions had a significantly lower proportion of households that owned a private latrine. In program communities with a larger proportion of land in use for agricul-ture, a significantly smaller proportion of house-holds built their own latrines.

Table 9: No significant impact of RWSSP on women’s participation in (self) employment

Note: The magnitude of the estimated coefficient shows percentage points.


Both adult men and women (18–65 years) 0.040 0.0274

Adult men (18–65 years) 0.041 0.027

Adult women (18–65 years) 0.044 0.038


An ID

EV Im

pact

Eva

luat

ion

Sustainability of benefits of the RWSSP

❙ Are the RWSSP results sustainable?

In responding to the sustainability questions, the study assessed WaSHCOM performance (institutional aspects), the technical and financial viability of water facilities (technical and financial aspects), and the adequacy of the monitoring, evaluation and learning (MEL) system. The assessment shows that the results of the RWSSP are unlikely to be sustained in the absence of continued public (district/regional/federal) funding.

Institutional aspects: Modest performing WaSH-COMs due to weak technical capacity and incentive. About 96% of the RWSSP participating communities contributed to the construction of their water schemes in cash and kind. WaSHCOMs cover almost all RWSSP water-points; water users elect WaSHCOM members. WaSHCOMS were responsible for providing and managing water points/supply, sanitation and hygiene promotion services. In most WaSHCOMs, the propor-tion of women fell below the official target (30–50%). Although the RWSSP enhanced the capacity of the WaSHCOMs, they remain organizationally, technically and financially too weak to effectively carry out their responsibilities. Some WaSHCOM members hardly allocated sufficient time for WaSHCOM activities. As indicated in Table A5.4 (Annex 5), lack of local skills was the major reason reported for the failure to repair about 46% of the non-functioning water facilities.

Technical aspects: Limited technical viability with-out government support. Most of the RWSSP water and sanitation facilities were technically sound, but challenging to operate and maintain. The analysis of self-reported data of the water point survey shows that about 89% of the water points are functional. About 3.5% of the program communities with a broken water facility had alternative improved water sources within the communities. Around 70% of the broken water facilities were successfully repaired. Nearly 60% of these non-functional water points were repaired by the WSST. Beneficiary communities and WaSHCOMs were responsible for repairing only 20% of all water points in need of repairs. The participation

of the private sector in repairing water points was very low. Interviews with water point respondents and discussions with WSSTs reveal that the shortage of necessary spare parts was a major challenge to timely repairs. Water quality tests were only rarely conducted.

Financial aspects: Limited financial viability of opera-tions and maintenance of RWSSP water supply facilities. Although communities were expected to pay water user fees to cover the operation and maintenance of their water facilities, community payments were very low. From the water point survey, most RWSSP communities reported that they raised 5% or more of the construction costs of the water points. However, nearly 55% of the program communities surveyed in this study were not charging for water use (Annex 5, Table A5.3). In commu-nities with water user charges, users pay lump sum fees on a monthly (47.06%) or yearly basis 15.69% wherein the mean yearly and mean monthly house-hold fees were about 13.75 (range=1–40) and 3.58 (range=1–17) ETB respectively 14. The majority of water users consider that these water charges were either cheap (59.07%) or reasonable (34.81%). As a result, the majority of the water points surveyed appeared not to be generating enough revenue to cover their opera-tion and maintenance costs. For example, around 65% of the recent water point repairs were financed through local government support.

Institutional arrangements for adequate monitor-ing, evaluation and learning system (MEL): The MEL was minimal in design, implementation, and use. The RWSSP implementation manual included provisions for a complete and effective monitoring and evaluation (M&E) system so as to generate cred-ible information for informing program implemen-tation, evaluation and sustainability. Although the required M&E system was not put in place (AfDB PCR 2015), some monitoring data were generated especially during program supervision and comple-tion reporting. Though useful to some extent, the data was incomplete for post-program evaluation and monitoring especially of the sustainability of the program effects because it had only limited cover-age of program development outcomes, costs, and critical context variables.

39Conclusions and Recommendations

An ID

EV Im

pact

Eva

luat

ion

Conclusions and Recommendations

Conclusions

The RWSSP was effective in delivering the water and sanitation infrastructure capacity but less so in build-ing community institutional capacity. The functionality of the water and sanitation infrastructure capacity was reduced mainly because of breakdowns and the idle capacity of some facilities. The program enhanced the community institutional and management capacity particularly the WaSH committee (WaSHCOM), which had modest performance in managing and sustaining the WaSH facilities and services.

The RWSSP was successful in producing desirable impacts in its key objective domain of improving access to and the use of improved water sources in the target communities, but not in other areas. The program significantly improved households’ access to and use of improved water sources although contam-ination by E. coli remained an issue at water sources and points of use. But this had no detectable impact on daily water consumption, as the self-reported daily per capita water consumption already far exceeded the national standard.

The RWSSP was effective in promoting private latrine ownership and use, thereby contributing to the decrease in open-defecation, but not in enhanc-ing hand washing at critical times, or reducing the incidence of diarrhea among children under five. The quality of the latrines and hand washing was far below the expected standards. The program also succeeded in reducing the incidence of diarrhea among persons of all ages.

The program was also modestly effective in reduc-ing the time spent fetching water. The modest time

savings was mostly used for domestic activities, but did not increase school enrolment or women’s partic-ipation in self-employment.

The sustainability of the RWSSP benefits is unlikely because of institutional, technical, financial and devel-opment information inadequacies. Furthermore, the availability of public funds was critical for the opera-tions and maintenance of RWSSP facilities.

Recommendations

The key emerging recommendations are that the Bank should:

1. Develop, in collaboration with the key stake-holders (including the Federal, Regional and Woreda administrations, and WaSHCOMs), a clear strategy for sustaining the benefits of the RWSSP. While reflecting the strengths and weaknesses of the Woreda and community insti-tutional arrangements for designing, implement-ing and managing WaSH facilities and services, such a strategy should address among others the following issues:

a. Poor quality water and sanitation facilities and services, and irregular water quality test-ing and treatment. The main issues for the RWSSP facilities and services are the poor quality of the facilities, the presence of E. coli in water at the source and points of use, the predominance of unimproved types of latrines, widespread hand washing without soap, and other hygiene practices that undermine posi-tive health outcomes.


b. Lack of adequate financial resources for the operation and maintenance of the RWSSP water facilities. Current water facility user charges are insufficient to meet the cost of operation, maintenance, or replace-ment, which contributes to delays or lack of repairs. Currently, there is no clear strategy for ensuring the financial viability of water point management.

c. WaSHCOMs weak organizational and management capacity. WaSHCOMs are important for sustaining the RWSSP results, but they are organizationally and technically weak. Moreover, 89% of the WaSHCOM members reported that they need (refresher) training. The gender imbalance issue in WaSHCOMs should be addressed.

d. Weak capacity of WSST to better support WaSH-COMs and to effectively participate in monito-ring the sustainability of the RWSSP benefits.

2. Support the development and implementa-tion of an effective MEL system in order to ensure regular and pertinent data collection, analysis, reporting and feedback especially on RWSSP community WASH results. Given the lack of baseline data and effective M&E, the RWSSP missed an opportunity to learn and support program completion reporting and the impact evaluation study. Such a missed oppor-tunity should be avoided by having a sound MEL system for the post-RWSSP. The Bank could build such a system through support for strengthening country systems to support the sustainability of the RWSSP results. The use of smart technolo-gies (such as GPS-enabled devices, geo-refer-enced management tools and smart-phones) for MEL should also be explored.

Annexes


Annex 1: RWSSP Theory of Change and Definition of Outcome Variables

Communities know about the RWSSP and as a result households jointly apply for support for community-managed improved WASH service provision.

Woreda WASH team correctly identifies eligible communities.

Targeting Inputs Outputs Final OutcomesIntermediate Outcomes

AfDB and GoE provide assistance to woreda for delivery of WASH services to communities as planned.

Woreda WASH team provides inputs for construction and/ or rehab of improved water scheme and institutional latrines; and for community facilitation team (CFT).

Applying households contribute in cash or kind to construction/ rehab of water scheme as per the approved application.

Improved water source is constructed or rehabilitated as per the application and RWSSP requirements.

Institutional latrines are constructed as per (access, sanitary) RWSSP requirements.

Woreda Community facilitation Team (CFT) provides training to community to establish WASH committee, for scheme O&M, and for improved sanitation and hygiene practices.

In response to CFT training community establishes gender balanced WASH committee; committee takes care of continued functionality of the scheme; of scheme administration (incl. list of member HHs, income and expenditure); of regular collection of water fees from users this way providing sufficient revenues for O&M and scheme repairs.

In response to CFT training WASH committee has acquired S&H knowledge and uses this to provide continued sanitation and hygiene awareness raising and training to community members.

All eligible HHs use the new/ rehabilitated water scheme all the time as source for drinking and hygiene purposes.

Target beneficiaries use institutional latrines at times as expected.

Exposure to S&H training results in households building and using private latrines; hand washing at critical times; in safe water handling during water transport and storage in the homes.

Improved water scheme and use of (own) latrine reduces time needed for fetching water and human defecation (part. women and girls).

Water scheme results in increased daily per capita safe water consumption for drinking and hygiene purposes.

Improved WASH practices result in reduced incidence of water-related diseases (diarrhea incidence of children <5 as proxy).

Time saved results in increased children’s school attendance (part girls).

Time savings results in increased time spent on productive employment, including women’s income generating activities.

Figure A1.1: Theory of Change Underlying RWSSP: Causal Chain Outcomes, Means and Assumptions

45Annexes

An ID

EV Im

pact

Eva

luat

ion

Annex 2: Definition Of Community Level Outcome Variables

Table A2.1 Definition and Measurement of Community Level Outcome/Impact Variables

Definition Community-level outcome indicators

Access and use of improved sources of water supply and sanitation facility

Using an improved source of water Does the household obtain its drinking water supply from an improved water source?

Proportion of households obtaining their water from an improved water source

Travel time to a water source How much time does it take to reach the primary water source?

Average travel time to the primary water source (one way)

Total time spent to fetch water How much time per round trip (including waiting, filling time) does it take for the household to fetch water?

Average time spent per round trip to fetch water

Per capita daily water consumption How much water does a household use for domestic purposes (i.e. drinking, food preparation, cleaning utensils, and basic personal hygiene)

Average daily per capita water consumption

Using a toilet facility Do household members use a toilet/latrine all the time? Does the facility meet the criterion of hygienic separation of human excreta from human contact?

Proportion of households using a toilet facility; toilets meeting sanitation requirement of having a washable slab

Diarrhea incidence outcome

Diarrhea incidence in children under five Was the child sick from diarrhea in the last two weeks?

Proportion of children under five who were sick from diarrhea in the last two weeks?

Education outcomes

School enrolment for children Is a school-age child currently enrolled in school?

Proportion of primary school-age children currently attending school (i.e. total, boys, and girls)

Employment outcome

Women’s participation in productive (self) employment

Was a woman in the household employed in productive (self) employment?

Proportion of women working in productive (self) employment


Annex 3: Sample Size and Sampling Procedures

A3.1 Sample Size Determination

As stated earlier, the AfDB-supported RWSSP was implemented to generate desirable changes in a number of outcomes/impact variables. Our sample size estimation here focuses only on key outcome variables for which the necessary information are available to compute the statistical sample. Our sample size estimation has followed a number of steps and has made several important assumptions. First, we anticipated that we could minimize the probability of type-I and type II errors and decided to limit these at common standards namely, 10% (for type I error) and 20% (for type II error). A type I error is the probability of finding a significant program effect when the true effect is zero. On the other hand, a type II error is the probability of not detecting a program impact when it actually occurs. One minus type II error is the statistical power of the test, which is 80% for this evaluation. This means that our design has the power to detect an impact 80% of the time when one has occurred. Second, in the estimations of the minimum number of samples required to measure impact corresponding to each outcome vari-able of interest, we used information from program documents and secondary data sources, especially Welfare Monitoring Surveys, Demographic and Health Surveys, Annual Education Abstracts of the Ministry of Education. Third, the unit of assignment of the AfDB-supported RWSSP is the community, but most of the outcomes are primarily measured at the level of individuals or households living in those communities. Thus, our sample size estimation followed a two-step procedure and was implemented with STATA Software. The computation of how many sample communities and households/individuals per sample community need to be surveyed depends on two main factors: (1) the degree of homogeneity/intra-cluster correlation of households/individuals within the community, and (2) the survey cost of the evaluation. For instance, when the intra-cluster correlation is high, adding a new household/individual to the evaluation sample gives greater statistical power than adding a new household/individual from an already selected community but this comes with a higher survey cost because it is more expensive to survey an additional household/individual from a new community than to survey an additional household/individual within an already selected community (Gertler et al., 2011).

Fourth, we applied a balanced design whereby we allocated an equal number of evaluation sample units between treatment and control groups to maximize the statistical power. Fifth, the study used a two-tailed test in eval-uating the program effects to make it possible to detect both intended and unintended effects of the program. Finally, we took into consideration the sample size requirement of the proposed impact evaluation method. In particular, a 15% trimming of sample communities during the process of matching program or treatment and non-program or comparison communities was anticipated a priori. Therefore, the target estimated sample size was adjusted by a 15% inflation factor to obtain the final evaluation sample size.

Applying relevant values and important assumptions (as shown in Annex 2), the minimum required sample size for each outcome variable was estimated to be as indicated in Table A2.1.

47Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A3.1 Estimation of required minimum sample sizes

Outcome/impact indicator

Unit of observation for data collection

Minimum required sample size

# of communities # of households per community

# of children per community

Use of improved water source

Household 12 8

Travel time to water source (one way)

Household 66 4

Total time spent on fetching water

Household 46 7

Use of latrine Household 10 8

Incidence of diarrhea Children aged less than five years

198 12 7

School enrolment Children aged 6–14 40 3 4

Girls aged 6–14 38 5 3

Boys aged 6–14 46 7 5

Women (self) employment in productive economic activities

Adult females aged 18–65

208 5

The minimum sample sizes required for the evaluation vary across outcome variables. However, the minimum sample size required for the “school dropout for girls” variable satisfies the sample size requirements of all the remaining outcome variables. Therefore, our impact evaluation study required a sample of 228 communities and 12 households per sample community. This implies a minimum sample of 114 communities and 1368 households per comparison group.


A3.2 Sampling Procedures

As stated earlier, this evaluation study was conducted in Amhara, Oromia, and SNNP regions. Within these regions, we propose to choose 38 woredas in which the program is implemented. The sample woredas were allocated to these regions in proportion to the total number of beneficiaries found in them until our evaluation timeframe. Consequently, the evaluation sample included 18, 12 and 8 woredas from Oromia, Amhara and SNNP regions respectively. In each woreda, there were equal numbers of treatment and control community and household samples. This created the same workload for each survey team, and facilitated easy field monitor-ing and supervision. On the basis of this allocation, the minimum required sample size was distributed across regions as indicated in Table 1A.

Table A3.2 Minimum required evaluation sample size

Regions Treatment sample Control sample Total

Communities Households Communities Households Communities Households

Oromia 54 648 54 648 108 1296

Amhara 36 432 36 432 72 864

SNNPR 24 288 24 288 48 576

Total 114 1368 114 1368 228 2736

The next major task was to draw a random sample of treatment and control communities and households within the selected woredas. To do this, the study followed a two-stage selection process in which the target number of sample communities was selected first, and the target number of households per selected community was selected in the second stage. Sampling of communities and households were performed separately for the treat-ment and control groups. To this end, the survey teams, in consultation with the Woreda Water Supply, Sanitation and Hygiene (WaSH) team, carried out the following specific tasks:

(i) Listing and Selection of Treatment Communities and Households

Within each sample woreda, the selection of treatment communities and households was performed in four steps:

First, all communities having implemented the AfDB-supported RWSSP during 2006–2014 were listed. This list was carefully prepared in consultation with the Woreda WaSH team.

Second, treatment communities that began to receive Water Supply and Sanitation services after June 2013 were excluded from the list 16 to give time for impact to materialize.

Third, a random sample of 3 treatment communities was drawn randomly from the list.

49Annexes

An ID

EV Im

pact

Eva

luat

ion

Fourth, within each selected treatment community, 12 households were selected using a systematic random sampling procedure.

(ii) Listing and Selection of Control Communities and Households

The following steps were implemented to list and select control communities in the sample woredas:

First, all non-AfDB communities having similar characteristics before the program intervention and similar features fixed over time to the treatment communities were listed in consultation with the woreda WASH Team. Non-AfDB communities were listed as potential control communities based on their similarities to treatment communities in terms of:

❙ Population density before the intervention began in the treatment communities

❙ Water and sanitation coverage before the intervention began

❙ Community motivation and commitment to participate in the program had the program been extended to them

❙ Agro-ecological zone

❙ Distance to district capital

Second, communities that were adjacent 15 to an AfDB-intervention/treatment community or have received improved water sources through external support (e.g. NGOs, other development partners etc.), if any, were excluded from the list.

Third, a random sample of 3 control communities was made randomly from the list prepared in the second step.

Fourth, within each selected control community 12 households were selected using a systematic random sampling procedure.


Annex 4: Descriptive Statistics

Table A4.1 Summary statistics on selected pre-program community characteristics

Variables Means Difference in means

Non-RWSSP RWSSP Combined

(1) (2) (3) (1)–(2)

Population density (# of persons/ha) (ln) 1.322 1.472 1.397 –0.149

Access to improved water source 0.096 0.157 0.127 –0.061

Number of households (ln) 3.681 3.784 3.733 –0.103

Distance to district capital town 15.172 13.566 16.779 3.21**

Poverty status (=1 if participated in productive safety-nets program)

0.325 0.333 0.316 –0.018

A development agent or health extension worker lived inside community

0.202 0.246 0.158 –0.088*

Agricultural land (%) 73.2 71.4 75.0 3.538*

Community location

High-altitude/dega 0.268 0.298 0.237 –0.061

Middle-altitude/woindadega 0.614 0.596 0.632 0.035

Low-altitude/kola 0.118 0.105 0.132 0.026

Main topographic feature is:

Flat 0.526 0.509 0.544 0.035

Mountainous 0.408 0.430 0.386 –0.044

Other 0.066 0.061 0.070 0.009

Leadership experience (=1 if anyone within community assumed leadership role)

0.614 0.623 0.605 –0.018

Regions:

Amhara 0.316 0.316 0.316 0.00

Oromia 0.474 0.474 0.474 0.00

SNNPR 0.211 0.211 0.211 0.00

Number of sample communities 114 114 228

Source: Community Survey Data 2014.

51Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A4.2 Descriptive statistics of selected community level outcome variables (before matching)

Variables Means Difference in means

Non-RWSSP RWSSP Combined

(1) (2) (3) (1)–(2)

Water

Access to improved water source 0.221 0.910 0.566 –0.689***

Daily per capita water use (in liters) (for drinking and cooking, hygiene, cleaning and washing)

30.395 34.280 32.36441 3.884

Daily per capita water use (in liters) (for drinking and cooking)

7.011 8.397 7.707 1.386

Travel time to reach water source (single difference)

17.066 9.883 13.474 7.183***

Total time spent on per roundtrip for fetching water (single difference)

56.509 40.321 48.415 16,189***

Change in travel time to reach water source

–5.184 –0.929 –3.057 –4.256***

Change in total time spent on per round-trip for fetching water

–12.001 –2.321 –7.161 –9.68***

Sanitation

Latrine ownership 0.746 0.820 0.783 –0.074**

Latrine use 0.659 0.752 0.706 –0.093**

Health outcomes: diarrhea incidence

All persons 0.158 0.088 0.123 0.07***

Children less than five years of age 0.055 0.056 0.056 –0.002

School enrolment

All children (6–14 years of age) 0.972 0.964 0.968 0.007

Boys (6–14 years of age) 0.968 0.962 0.965 0.007

Girls (6–14 years of age) 0.972 0.973 0.973 –0.002

Self-employment

All persons (18–65 years of age) 0.105 0.14 0.123 –0.035

Adult men (18–65 years of age) 0.109 0.146 0.127 –0.036

Adult women (18–65 years of age) 0.099 0.134 0.117 –0.035

Note: difference is tested based on a two-tailed t-test statistic.

*** Statistically significant at the 1% level.


* Statistically significant at the 10% level.



Table A4.3 Primary water source used by the sample households

Primary water source Non-RWSSP RWSSP Combined

No. Percent No. Percent No. Percent

Piped into dwelling 6 0.44% 8 0.59% 14 0.51%

Piped to yard/plot 64 4.70% 6 0.44% 6 0.22%

Public tap/standpipe 0 0.00% 525 38.49% 589 21.61%

Shallow well 3 0.22% 197 14.44% 200 7.34%

Hand-dug well 75 5.51% 214 15.69% 289 10.6%

Protected spring 130 9.54% 300 22.00% 430 15.77%

Unprotected well 104 7.64% 29 2.13% 133 4.88%

Unprotected spring 609 44.71% 62 4.55% 671 24.61%

Tanker Truck 0 0.00% 1 0.07% 1 0.04%

River/stream 312 22.91% 21 1.54% 333 12.22%

Lake/pond 44 3.23% 0 0 44 1.61%

Rainwater 14 1.03% 1 0.07% 15 0.55%

Other 1 0.07% 0 0 1 0.04%

Total 1362 100% 1364 100% 2726 100%


53Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A4.4 Water charges and self-reported household water treatment behavior

Variables Non-RWSSP RWSSP Total


Household pays fees for water use

135 10.48% 545 39.9% 680 25.62%

Household perceives water from its primary source is good or very good

564 40.25% 1242 90.78% 1806 66.03%

Household applies water treatment before drinking it

Yes 96 7.023% 48 3.51% 144 5.27%

No 1218 89.1% 1313 95.98% 2531 92.54%

Missing 53 3.88% 7 0.512% 60 2.20%

Total 1367 100% 1368 100% 2735 100%

If yes, types of water treatments applied:

Boiling the water 14 14.58% 4 8.33% 18 12.50%

Adding bleach or chlorine to the water

35 36.46% 30 62.50% 65 45.14%

Using a water-filtering device or ceramic

32 33.33% 5 10.42% 37 25.69%

Let it stand and settle 8 8.33% 3 6.25% 11 7.64%

Other 4 4.17% 0 0.00% 4 2.78%

Missing 3 3.13% 6 12.50% 8 5.56%



Table A4.5 Time spent fetching water, persons responsible for fetching water and means of transportation used

RWSSP Non-RWSSP Combined

% of households reporting that time needed to water was reduced/not reduced in last few years

Reduced 60.97% 12.29% 36.62%

Not reduced 39.03% 87.71% 63.38%

If yes, on which activity is the time saved from fetching water mainly spent?*(% mentioning the below alternative)

Cooking and other domestic activities 82.59% 74.23% 81.21%

Unpaid work 38.94% 40.49% 38.79%

Paid work 2.42% 3.68% 2.62%

Firewood collection 13.18% 26.99% 15.45%

Other income generating activities 6.41% 1.23% 5.45%

% of households who reported person(s) responsible for water fetching is/are:

Females (18 years and above) 48.62% 48.76% 48.69%

Female children (less than 18 years) 22.44% 23.26% 22.85%

Males (18 years and above) 14.41% 12.62% 13.51%

Male children (less than 18 years) 14.53% 15.36% 15.92%

% of households who reported primary means of water transportation from primary water source to home is:

Animal cart 1.17% 1.11% 1.14%

Pack animals 15.98% 16.01% 16.0%

Human load 82.84% 82.66% 82.75%

Other 0.0% 0.22% 0.11%

Average number of times a household normally fetches water per day

2.28 2.46 2.37

Note: * due to multiple responses the percentages do not add up to 100%.


55Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A4.6 Household Construction of Own Latrine and Use

Note: * about 99% of all the latrines in both program and non-program communities are pit latrines with a mud/wooden cover (91% and 92%, respectively, in program and non-program communities) or slabs made of concrete (8% and 7%, respectively in program and non-program communities).


Non-RWSSP RWSSP Total


Constructed their own latrines* 1,018 74.58% 1,116 81.94% 2,727 78.25%

Source of initial idea for constructing own latrine

Self 178 17.49% 199 17.85% 377 17.67%

Neighbors/friends 18 1.77% 11 0.99% 29 1.36%

Health extension workers 813 79.86% 890 79.82% 1,703 79.84%

WaSHCOM 0 0.00% 11 0.99% 11 0.52%

Other (including missing values)

8 0.89% 4 0.36% 12 0.62%

Latrine has a hand washing facility

Within 3 meters radius 91 8.94% 148 13.26% 239 11.2%

Further than 3 meters radius 93 9.14% 126 11.29% 219 10.26%

No hand washing facility 808 79.37% 832 74.55% 1,640 76.85%

Missing values 26 2.55% 10 0.9% 36 1.69%

Main reasons for not constructing own latrine

Do not have enough land 40 11.53% 15 6.1% 55 9.27%

Lack of skill for construction 52 14.99% 49 19.92% 101 17.03%

Do not consider it important 82 23.63% 46 18.7% 128 21.59%

Shortage of money 64 18.44% 52 21.14% 116 19.56%

Shortage of construction material

70 20.17% 48 19.51 118 19.9%

Other 39 11.12% 36 14.63% 75 12.65%

Do not have a private toilet but use

Latrine shared with other households

13 3.82% 10 4.15% 23 7.95%

Communal or public latrine 17 4.98% 14 5.8% 31 10.33%

Bush/open-air 310 90.91% 216 89.63% 526 90.9%

Other 1 0.29% 1 0.41% 2 0.34%

Household perception of number of people practicing open-defecation in their community

Increased 66 4.96% 59 4.42% 125 4.69%

Remained same 213 16.0% 321 24.04% 534 20.03%

Decreased 1048 78.74% 945 70.79% 1993 74.76%

Do not know 4 0.3% 10 0.75% 14 0.53%


Table A4.7 Incidence of self-reported hand washing at critical times

Outcome variableReference person

Material used for hand washing Number


With soap Without soap



a) Hand washing before eating

Adult men (15 years and older)

9.82% 89.79% 8.10% 91.74% 8.96% 90.77% 2589

Adult women (15 years and older)

11.24% 88.01% 8.7% 91.15% 9.98% 89.68% 2665

Children (6–14 years)

8.31% 90.65% 6.66% 92.64% 7.48% 99.14% 1969

b) Hand washing before food preparation

Adult men (15 years and older)

11.61% 71.07% 10.68% 69.88% 11.14% 70.47% 1131


14.15% 82.84% 11.5% 85.78% 12.83% 84.50% 2651

Children (6–14 years) 10.54% 78.78% 8.19% 81.9% 9.34% 80.38% 1478

C) Hand washing after using toilet

Adult men (15 years and above)

19.33% 73.55% 16.69% 73.51% 18.01% 74.04% 2554


21.6% 73.05% 18.05% 73.68% 19.84% 73.81% 2677

Children (6–14 years)

18.11% 66.74% 14.95% 69.57% 16.52% 68.53% 1907


57Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A4.8 Participation of adult men and women (18–65 years of age) in self-employment


Variables Non-RWSSP RWSSP Total


Did he/she participate in self-employment activities in the last 12 months?

Yes 330 10.29% 449 13.92% 779 12.11%

No 2833 88.34% 2727 84.53% 5560 86.43%

Missing 44 1.37% 50 1.55% 94 1.46%

Total 3207 100.00% 3226 100.00% 6433 100.00%

If yes, types of self-employment activities:

Selling firewood/charcoal 23 6.97% 23 5.12% 46 5.91%

Trading in livestock/grain/or other agricultural product

190 57.58% 258 57.46% 448 57.51%

Retail/wholesale trade/shop 15 4.55% 36 8.02% 51 6.55%

Selling local drinks/brewery 44 13.33% 50 11.14% 94 12.07%

Restaurant/food selling 16 4.85% 12 2.67% 28 3.59%

Pottery 8 2.42% 10 2.23% 18 2.31%

Carpentry 8 2.42% 11 2.45% 19 2.44%

Weaving/Spinning 3 0.91% 17 3.79% 20 2.57%

Transporting 7 2.12% 8 1.78% 15 1.93%

Other 9 2.73% 13 2.90% 22 2.82%

Missing 7 2.12% 11 2.45% 18 2.31%


Table A4.9 Availability of water supply and sanitation facilities at the sample primary schools and health centers

RWSSP Areas Non-RWSSP Area Combined

No. Percent/Average

No. Percent No. Percent

School has a water facility

Yes 13 35.14% 8 21.62% 21 28.38%

No 24 64.86% 29 78.38% 53 71.62%

Total 37 100.00% 37 100.00% 74 100.00%

School has a toilet facility

Yes 35 94.59% 36 94.74% 71 94.67%

No 2 5.41% 2 5.26% 4 5.33%

Total 37 100.00% 38 100.00% 75 100.00%

School toilet facility is physically separate for boys and girls: yes

25 71.43% 27 75% 52 73.74%

Health Center has a water facility

Yes 35 100% 37 97.37% 72 98.63%

No 0 0% 1 2.63% 1 1.37%

Total 35 100% 38 100.00% 73 100%

Health Center’s toilet facility is currently functional

Yes 32 96.97% 35 100% 67 98.53%

No 1 3.03% 0 0% 1 1.47%

Total 33 100.00% 35 100.00% 68 100>00%

Average number of diarrhea outpatient visitors per Health Center in 2013/2014: children under five

29 214.69 34 315.15 63 268.9

Note: No. stands for number of sample primary schools.

Source: Primary School Survey Data and Health Center Survey Data 2014.

59Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A4.10 Descriptive summary of primary school (Grades 1–4) enrolment and school dropout rates in 2013/14.

Note: No. stands for number of sample primary schools.

Source: Primary School Survey Data 2014.


No. Mean No. Mean No. Mean

Average number of students enrolled per school in 2013/14

Boys 36 297.9 38 258.1 74 277.5

Girls 36 270.1 38 234.5 74 251.8

Both 36 568.02 38 492.6 74 529.3

Students who dropped-out school in 2013/14

Boys 32 15.5 [5.0% ]

38 13.8 5.8%]

68 14.6[5.4%]

Girls 32 13.7 [5.2%]

36 11.7 [5.4%]

68 12.7[5.3%]

Both 32 29.2 [5.1%] 36 25.5[5.6%]

68 27.3[5.8%]


Annex 5: RWSSP Implementation by the Selected Communities

Table A5.1 Management of water facilities constructed through the RWSSP

No. Percent

Water facility has a WaSHCOM 110 96.49%

WaSHCOM has women as its member 92 83.63%

Percentage of women within WaSHCOM members

Less than 35% 51 55.43%

35%–49% 23 25.00%

50% and above 18 19.57%

WaSHCOM has women members in key positions including chairperson, secretary, and treasurer

39 35.45%

WaSHCOM keeps records of water users, expenses of the facility and so forth

87 79.09%

WaSHCOM held its own meeting at least once last year 80 72.72%

WaSHCOM had a general meeting with water users last year 68 61.82%

WaSHCOM received capacity building trainings from WSST and others

63 55.26%

WaSHCOM perceives the trainings were useful 55 87.3%

WaSHCOM needs (re-fresher) trainings 89 80.91%

Source: Computed based on Data from the RWSSP Water-Points Survey 2014.

61Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A5.2 Technical Capacity Building Trainings Provided to Water Users by WaSHCOM

Table A5.3 Water users involvement in water facility construction and incidence of water fees

No. Percent

Provided trainings to water users 43 39.09%

Topics of trainings given to water users:

Community organization and planning 19 44.19%

Construction and service supervision 7 16.28%

Basic accounting 1 2.33%

Sanitation and hygiene 14 32.56%

Other 2 4.66%

No. Percent

Community contributed for the construction of the water facility 74 64.91%

Households pay fees for water use 51 44.74%

Total 114 100.00%

Source: RWSSP Water-Points Survey Data 2014.



Table A5.4 Functionality of water facility and quality of water at source

No. Percent

Water facility has ever broken down 43 37.72%

Main causes for recent break down of the facility

Lack of proper management 7 16.28%

Poor quality of facility construction 22 51.16%

Conflict among users 2 4.65%

Other 12 27.9%

The broken facility has been repaired 30 69.76%

For broken-water facilities not repaired yet at time of the survey, the major reasons were:

Lack of local skills to do the repair 6 46.15%

Lack of money 3 23.08%

Other 4 30.77%

Another water source available inside community whose main water source had not been repaired

Hand-dug-well/protected spring 4 30.77%

Unprotected well/unprotected spring 4 30.77%

River/stream/lake/pond 2 15.38%

Other 3 23.087%

Water facility repaired by:

WaSHCOM 5 16.67%

WWT 18 60.00%

User community 2 6.67%

Local Artisans 2 6.67%

Other 3 10.00%

Community-raised direct contributions or used savings to finance the recent repair of the facility

12 40.00%

Water quality tested at source last year* 23 20.17%

The water at the source has been disinfected with chemicals to improve its quality

37 32.46%

The water facility is fenced 73 64.04%


* Calculated based on data collected from community-level focus-group discussions.

63Annexes

An ID

EV Im

pact

Eva

luat

ion

Annex 6: Impacts of RWSSP on Community Level Outcome Variables (Full Results)

Probit analysis of community participation in RWSSP

The main unit of analysis in this study is the community because the RWSSP targeted the community level. A probit model is used to explain community participation into the program. The dependent variable in the probit model is coded as one for RWSSP communities and zero for non-RWSSP communities. The explanatory varia-bles included a set of time-invariant community characteristics or covariates measured before the start of the program. More specifically, these variables are chosen to capture stated official program targeting criteria, such as population density, remoteness, and water supply coverage, and other socioeconomic factors. These varia-bles are expected to determine not only the decision to participate into the program but they also can affect the outcome variables. As already noted, the main dataset on the explanatory variables were obtained from both treatment and comparison communities. During the sampling process, non-RWSSP communities were chosen from lists of comparison communities with similar characteristics as with the treatment communities before the RWSSP intervention. Therefore, the probit model was used to explain participation in the program of treatment and comparison communities that had already been made similar by the sampling process.

The probit results presented in Table A6.1 (Annex 6) indicate that the distance to district capital town decreased participation in the program. By contrast, the availability of a health post or office of a development agent within the community also increased the probability that the program selected the community. This suggests that communities with stronger ties with local administration are more likely to participate in the program. Commu-nities located in Oromia region were more likely to participate into the RWSSP compared to their counterparts in SNNPR. Mountainous or flat topography did not affect community participation in the program. However, controlling for population density, communities in high-altitude/dega areas were more likely to participate in the program whereas communities with a larger percentage of their land in agriculture had lower participation rates. The predicted propensity score or probability of community participation in the RWSSP varied from 0.204 to 0.925 (mean= 0.537) for the treatment sample and 0.133 to 0.778 (mean= 0.461) for the comparison sample communities. This implies a strong common support region for finding matches between the two samples (see Figure A6.1, Annex 6).

Analysis of mean impacts of the RWSSP

This evaluation has quantitatively measured the impacts of the RWSSP on a range of outcome variables including water supply, sanitation and hygiene, education, health and employment. Consistent with the RWSSP’s approach to measuring and evaluating its success (FDRE, 2004), these outcome variables were compared at the commu-nity level using data provided by sample households in treatment and comparison communities.

The impact estimates reported below are obtained by using nearest five neighbors matching estimator. After matching, all explanatory variables and the predicted propensity scores have similar mean values for program and non-program communities indicating that the balancing property of PSM is satisfied (Smith and Todd, 2005). Additionally, the pseudo-R2 dropped from 0.0582 (before matching) to 0.007 (after matching) implying that there are no systematic differences in the distribution of the explanatory variables between RWSSP and


non-RWSSP communities. Before matching, the likelihood ratio test indicated that the explanatory variables were jointly significant in explaining community participation in the program. However, after matching, this test result (LR chi2=2.23) is not statistically significant, strengthening the other test results that matching has been successful in creating observationally similar program and non-program communities (Tables A6.1–A6.2 for details on these tests and others).

As a robustness test, the study also estimated program impacts using alternative methods namely, one nearest neighbor matching (with and without replacement), two nearest neighbors matching and weight-ed-least squares (WLS) using inverse probability of treatment to weight program and non-program commu-nities (Hirano et al., 2003) Table A6.3 (Annex 6) provides matching quality from alternative algorithms. As can be seen, the nearest five neighbors matching has become the best alternative since it resulted in the lowest pseudo-R2, highly insignificant values for LR Chi2, largest mean bias reduction and lowest median bias values after forming matched pairs of program and non-program communities (Smith and Todd, 2005; Lokshin and Yemstov, 2005). The study used a propensity score matching algorithm developed by Leuven and Sianesi (2003).

Sensitivity analysis of the PSM results

As already stated, PSM ensures that the observed heterogeneities between program and non-program communi-ties are controlled for. However, potential unobserved differences between program and non-program communi-ties cannot be ruled out. In this case, the PSM results could be biased because of unobserved covariates affecting both participation in and outcomes of the program. To check the vulnerability of the estimated PSM results to selection on unobserved variables, Rosenbaum’s sensitivity test was applied (Rosenbaum, 2010). The test quan-tifies how strong selection on unobserved characteristics would have to be to alter the results of the PSM (Rosen-baum, 2010; Gangle, 2004). Table A6.10 provides tests of sensitivity of significant treatment effects estimated using the PSM. The estimated impact on access to improved water source remains statistically significant until the unobserved differences between the matched communities increase by a factor of more than 14.85. What this means is that program and non-program communities with similar observed characteristics should differ in their unobserved characteristics by at least 1385% in order to turn the significant effect into an insignificant impact on access to improved water. On latrine ownership, the estimated impact remains statistically signifi-cant and robust to unobserved selection bias until the matched communities differ in unobserved covariates by a factor of 1.35 (35%) to 1.4 (40%). The impact on travel time to reach the primary drinking water source is significant, and this effect remains robust to hidden bias until unobserved characteristics between matched communities increase by a factor of 3.4 (240%) or more. Given that the study has already controlled for a range of pre-intervention characteristics, it is unlikely that unobserved selections of such magnitude can occur in the data. Accordingly, one can conclude that the PSM estimates of impacts reported earlier are not vulnerable to the problem of selection on unobserved or unmeasured variables bias.

65Annexes

An ID

EV Im

pact

Eva

luat

ion

Analysis of distributional impacts of the RWSSP

Thus far, the analyses of impacts have rested on the assumption that impacts are constant across all beneficiary communities. However, as is evident from several previous studies, program impacts may vary across treatment communities depending on their pre-existing and other exogenous characteristics (Rauniyar et al., 2011; Mu and van de Walle, 2011). To detect the presence of heterogeneous impacts in the present study, community-specific impacts are regressed on pre-existing characteristics and exogenous factors. This approach was used in Mu and van de Walle (2011), and Wagstaff et al. (2009). The community-specific impact for a program community is computed by taking observed and counterfactual outcomes. Tables A6.11–A6.13 present analyses of impact heterogeneity using program communities as the estimation sample.

Table A6.1: Probit analysis of community participation in RWSSP

Variables+ Coeff. Std. errors Z p>|Z|

Population density (ln) 0.152 0.138 1.100 0.270

Number of households (ln) 0.064 0.144 0.440 0.658

Access to improved water source (1=yes) 0.426 0.316 1.350 0.177

Distance to district capital town –0.018 0.008 –2.210 0.027

Agriculture extension agents or health extension works live inside community (1=yes)

0.417 0.176 2.360 0.018

Agriculture land (%) –0.012 0.006 –1.880 0.060

Participated in local leadership 0.038 0.174 0.220 0.828

Poverty status (1=if ever participated in a productive safety net program)

–0.003 0.163 –0.020 0.986

Region: SNNPR reference

Amhara 0.183 0.140 1.300 0.193

Oromia 0.307 0.175 1.760 0.079

Agro-ecology: Low altitude/kola reference

Middle altitude/woinadega 0.252 0.198 1.280 0.202

High altitude/dega 0.340 0.211 1.610 0.108

Topography: Flat reference

Mountainous 0.153 0.192 0.800 0.425

Other 0.135 0.200 0.670 0.500

Constant 0.005 0.642 0.010 0.993

Number of observations 228

Wald Chi2 (13) 23.13

Prob>Chi2 0.0582

Psuedo-R2 0.0582

Note: the dependent variable was coded as 1 for RWSSP communities and zero for non-RWSSP communities. +Data on explanatory variables refer to 2005 or earlier.

Source: Estimation of Result Based on Community Survey Data 2014.


Figure A1.1 Distribution of estimated propensity scores

Non-RWSSP: Off support Non-RWSSP: On support RWSSP: On support RWSSP: Off support

0.2 0.4 0.6 0.8 1

Propensity score

67Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A6.2 Tests of equality of covariates means after nearest-five neighbors matching

Variable Mean % bias t p>t

Treated(RWSSP)

Control (Non-RWSSP)

Propensity score 0.520 0.520 –0.2 –0.02 0.984

Population density (ln) 1.426 1.459 –4.0 –0.30 0.766

Number of households (ln) 3.753 3.802 –6.7 –0.49 0.624

Accessed improved water source (1=yes) 0.130 0.111 5.6 0.42 0.678

Distance to district capital town 13.843 13.331 4.2 0.35 0.725

A development agent or health extension worker lived inside community (1=yes)

0.213 0.178 8.8 0.65 0.517

Agriculture land (%) 72.673 72.631 0.3 0.02 0.983

Participated in local leadership 0.639 0.667 –5.7 –0.43 0.670

Poverty status (1=if ever participated in productive safety net program)

0.333 0.335 –0.4 –0.03 0.977

SNNPR 0.213 0.237 –5.9 –0.42 0.674

Amhara 0.306 0.276 6.3 0.48 0.633

Oromia 0.481 0.487 –1.1 –0.08 0.935

Low altitude/kola 0.102 0.093 2.9 0.23 0.819

Middle Altitude/woinadega 0.611 0.615 –0.8 –0.06 0.956

High altitude/dega 0.287 0.293 –1.3 –0.09 0.929

Flat 0.509 0.476 6.7 0.49 0.626

Mountainous 0.426 0.465 –7.9 –0.57 0.567

Other 0.065 0.059 2.2 0.17 0.866


Table A6.3 Summary statistics on different balancing tests after propensity score matching

Matching Estimators Sample Pseudo-R2 LR Chi2 P>Chi2 Mean Bias Median Bias

Nearest five neighbors Unmatched 0.06 19.05 0.212 13 8.5

Matched 0.007 2.23 1 3.9 4.1

Nearest two neighbors Unmatched 0.06 19.05 0.212 13 8.5

Matched 0.02 5.97 0.98 7.8 8.5

Nearest one neighbor, with no replacement

Unmatched 0.06 19.05 0.212 13 8.5

Matched 0.036 10.74 0.771 9.4 8.6

Nearest one neighbor, with replacement

Unmatched 0.06 19.05 0.212 13 8.5

Matched 0.04 12.1 0.671 9.2 7.7

69Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A6.4 Estimated impacts using alternative matching estimators and WLS

Outcome variable Estimated impacts

NN5 NN2 NN1, with replacement

NN1, with no replacement

WLS

Access to improved water source 0.687*** (0.05)

0.694*** (0.057)

0.733*** (0.062)

0.681*** (0.044)

0.66*** (0.056)

Daily per capita water use for all purposes (in liters)

3.003(2.977)

3.279 (3.259)

2.57 (3.698)

3.669 (2.715)

2.277 (2.265)

Daily per capita water use for drinking and cooking (in liters)

1.067(0.943)

1.234(1.015)

0.928 (1.13) 1.153 (0.867)

0.915 (0.71)

Travel time to reach water source (single difference)

–5.586**(2.139)

–5.524** (2.092)

–3.548** (1.665)

–7.307*** (1.851)

–6.701*** (1.986)

Total time spent per round trip on fetching water (single difference)

–12.51**(5.41)

–13.430** (5.683)

–7.318(4.943)

–16.120*** (4.772)

–15.891*** (4.703)

Travel time to reach water source (double-difference)

–4.423**(1.33)

–4.381***(1.198)

–4.25***(1.238)

–4.435***(1.218)

–4.629*** (0.994)

Total time spent per round trip on fetching water (double-difference)

–9.84***(2.615)

–10.36***(2.458)

–10.232***(2.647)

–9.974***(2.42)

–10.958***(2.152)

Latrine ownership 0.082**(0.041)

0.112** (0.045)

0.08*(0.047)

0.069 (0.036)

0.073* (0.036)

Use of private latrine among all households 0.098**(0.043)

0.131*** (0.047)

0.096*(0.049)

0.085** (0.038)

0.093** (0.041)

Use of private latrines among households that own them

.052 (0.034)

0.065*(0.038)

0.041 (0.036)

0.047* (0.028)

0.056*(0.032)

School enrolment rate, all children 0.006(0.009)

0.007(0.009)

0.007(0.011)

0.0089(0.0086)

0.004(0.008)

School enrolment rate, boys 0.0006(0.013)

0.001(0.012)

–0.0001(0.013)

0.0111(0.0115)

0.006 (0.011)

School enrolment rate, girls 0.004(0.011)

0.003(0.012)

0.005(0.013)

0.0000056(0.01)

–0.004(0.011)

Incidence of diarrhea, all persons –0.07***(0.023)

–0.071** (0.026)

–0.072** (0.026)

–0.068*** (0.02)

–0.074*** (0.017)

Incidence of diarrhea, children under five years of age

0.004(0.028)

–0.015(0.031)

–0.025(0.036)

0.003 (0.028)

–0.005 (0.026)

Self-employment, all 0.0404(0.0274)

0.030(0.027)

0.023(0.03)

0.034 (0.024)

0.032** (0.013)

Self-employment, men 0.041(0.027)

0.037(0.027)

0.025(0.03)

0.037 (0.024)

0.034** (0.014)

Self-employment, women 0.044(0.038)

–0.003(0.042)

–0.01(0.051)

0.052(0.035)

0.05(0.037)

Note: NN1, NN2 and NN5 stand for one, two and five nearest neighbors matching.



* Statistically significant at the 10% level.


Table A6.5 RWSSP impact on access to improved water source and daily per capita water consumption

Table A6.6 RWSSP impact on time spent fetching water

Sample RWSSP Non-RWSSP Difference in means

Standard error

Access to improved water source Unmatched 0.910 [114] 0.221 [114] 0.689 0.044

Matched 0.905 [108] 0.218 [110] 0.687 0.050

Daily per capita water use (in liters) for drinking and cooking hygiene, cleaning and washing

Unmatched 34.280 [113] 30.395 [110] 3.884 2.629

Matched 34.121[107]

31.117[107]

3.003 2.977

Daily per capita water consumption for drinking and cooking only (in liters)

Unmatched 8.397 [113] 7.011 [112] 1.3862 0.850

Matched 8.286 [107] 7.218 [108] 1.067 0.943



Note: Numbers within brackets stand for number of sample communities.



Outcome variable Sample RWSSP Non-RWSSP Difference in means

Standard error

Single difference estimates

Travel time to reach water source Unmatched 9.882 [114] 17.066 [114] –7.183* 1.768

Matched 9.988 [108] 15.575 [110] –5.586** 2.139

Total time spent per-round trip on fetching water

Unmatched 40.320 [114] 56.509 [114] –16.188*** 4.61

Matched 40.545 [108] 53.056 [110] –12.51*** 5.41

Double-difference estimates

Travel time to reach water source Unmatched –5.184[114]

–0.929[114]

–4.255*** 1.157

Matched –5.326[108]

–0.903[110]

–4.423*** 1.33

Total time spent per-round trip on fetching water

Unmatched –12.001 [114]

–2.321[114]

–9.68*** 2.326

Matched –12.053 [108]

–2.213 [110]

–9.84*** 2.615

71Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A6.7 Impact of the RWSSP on self-employment of adult persons (18–65 years of Age)

Table A6.8 RWSSP impact on household latrine ownership and use

Note: Figures within brackets stand for number of sample communities.



Standard error

Total Unmatched 0.14 [114 ] 0.105 [114] 0.035 0.024

Matched 0.141 [108] 0.099 [110] 0.041 0.027

Men Unmatched 0.146[114] 0.109 [114] 0.036 0.024

Matched 0.146 [108] 0.104 [110] 0.041 0.027

Women Unmatched 0.134 [114] 0.099 [114] 0.0345 0.026

Matched 0.136 [108] 0.092 [110] 0.044 0.038


Standard error

Latrine ownership Unmatched 0.820 [114] 0.746[114]

0.074** 0.035

Matched 0.811 [108] 0.730[110]

0.081** 0.04

Use of private latrine among all households

Unmatched 0.752 [114] 0.659[114]

0.093** 0.037

Matched 0.741[ ]

0.643[ ]

0.098** 0.043

Use of private latrine among households who own it

Unmatched 0.906 0.849 0.056 0.028

Matched 0.901 0.849 0.052 0.034



Table A6.9 RWSSP Impact on the incidence of diarrhea

Table A6.10 RWSSP impact on current school enrolment rates of children 6–14 years old

Table A6.11 Sensitivity of estimated impacts to unobserved characteristics





Standard error

All persons Unmatched 0.088 [114] 0.157 [114] –0.069*** 0.02

Matched 0.087 [110] 0.157 [110] –0.07*** 0.023

Children under-five years of age Unmatched 0.056 [71] 0.054 [46] 0.002 0.025

Matched 0.053 [65] 0.048 [43] 0.004 0.028


Standard error

All children Unmatched 0.972 [114] 0.964 [114] 0.007 0.008

Matched 0.973 [108] 0.967 [110] 0.005 0.009

Boys Unmatched 0.968 [112] 0.972 [113] 0.007 0.011

Matched 0.969 [106] 0.971 [109] –0.002 0.013

Girls Unmatched 0.971 [113] 0.973 [111] –0.002 0.01

Matched 0.973 [106] 0.968 [107] 0.005 0.011

Outcome variables Estimated values of Rosenbaum bounds (gamma values)

Access to improved water sources 14.85–15.01

Latrine ownership 1.35–1.4

Use of private use (unconditional) 1.6–1.7

Double difference

Change in travel time to reach the primary water source 3.4–3.7

Change in total time spent on per-round trip for fetching water 2.8–3.1

73Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A6.12 Heterogeneity of RWSSP impacts on access to improved water source

Explanatory variables Coeff. Standard error p-value

Population density (ln) –0.084 0.051 0.105

Number of households (ln) –0.023 0.051 0.66

Access to improved water source (1=yes)

–0.002 0.094 0.979

Distance to district capital town

0.001 0.003 0.632

Agricultural extension agents or health extension work lived inside community (1=yes)

–0.162 0.088 0.074

Agriculture land (%) 0.002 0.003 0.422

Leadership experience 0.070 0.069 0.318

Poverty status –0.168 0.085 0.056

Amhara –0.066 0.099 0.51

Oromia –0.233 0.109 0.04

Middle-altitude/woinadega –0.090 0.126 0.483

High altitude/dega –0.076 0.122 0.539

Mountainous 0.068 0.080 0.397

Other –0.070 0.131 0.596

Constant 0.941 0.229 0

Number of observations 108

F( 14, 37) 2.17

Prob > F 0.03

R-square 0.157


Table A6.13 Heterogeneity of RWSSP impacts on household latrine ownership and use

Explanatory variables Latrine ownership Latrine use

Coeff. Standard error

p-value Coeff. Standard error

p-value

Population density (ln) 0.048 0.046 0.30 0.048 0.049 0.33

Number of households (ln) –0.058 0.051 0.26 –0.053 0.051 0.30


–0.026 0.113 0.82 –0.039 0.108 0.72


0.001 0.002 0.46 0.002 0.002 0.32


0.111 0.069 0.12 0.114 0.073 0.13

Agriculture land (%) –0.003 0.001 0.07 –0.003 0.001 0.01

Leadership experience 0.059 0.054 0.28 0.047 0.060 0.44

Poverty status –0.128 0.068 0.07 –0.176 0.075 0.03

Amhara –0.207 0.073 0.01 –0.180 0.076 0.02

Oromia –0.287 0.081 0.00 –0.333 0.090 0.00

Middle-altitude/woinadega 0.209 0.094 0.03 0.300 0.100 0.01

High altitude/dega 0.226 0.102 0.03 0.305 0.103 0.01

Mountainous –0.085 0.053 0.12 –0.074 0.059 0.21

Other 0.110 0.092 0.24 0.022 0.100 0.83

Constant 0.428 0.189 0.03 0.430 0.180 0.02

Number of observations 108 108

F (14, 37) 4.39 8.21

Prob > F 0.000 0.000

R-square 0.275 0.313

75Annexes

An ID

EV Im

pact

Eva

luat

ion

Table A6.14 Heterogeneity of RWSSP impact on time needed to fetch water (matched DID)

Explanatory variables Time to reach water source Total time spent per round trip on fetching water

Coeff. Standard error

p-value Coeff. Standard error

p-value

Population density (ln) 1.314 1.153 0.26 0.883 2.881 0.76

Number of households (ln) –2.494 1.392 0.08 –3.819 3.003 0.21


–5.267 3.677 0.16 –9.757 9.107 0.29


0.025 0.079 0.76 –0.117 0.190 0.54


2.533 2.495 0.32 4.104 4.932 0.41

Agriculture land (%) 0.082 0.059 0.18 0.142 0.128 0.28

Leadership experience –1.123 2.281 0.63 –2.114 5.094 0.68

Poverty status –3.807 2.793 0.18 –0.666 6.951 0.92

Amhara 0.653 3.261 0.84 –0.749 7.105 0.92

Oromia –3.269 3.146 0.31 –0.574 7.444 0.94

Middle-altitude/woinadega –1.892 3.505 0.59 –9.999 8.078 0.22

High altitude/dega 1.967 3.825 0.61 3.923 8.320 0.64

Mountainous 0.662 2.248 0.77 0.810 5.332 0.88

Other 2.676 2.943 0.37 7.611 5.003 0.14

Constant 0.402 6.449 0.95 1.189 15.298 0.94

Number of observations 108 108

F (14, 37) 1.35 1.61

Prob > F 0.226 0.12

R-square 0.123 0.153


References

African Development Bank/African Development Fund (AfDB/ADF). 2005. Ethiopia: Rural Water Supply and Sanitation Program: Appraisal Report.

African Development Bank/African Development Fund (AfDB/ADF). 2015. Ethiopia: Rural Water Supply and Sani-tation Program: Programme Completion Report.

Cairncross, S., Hunt, C., Boisson, S., Bostoen, K., Curtis, V., Fung, I. and Schmidt, W. 2010. Water, Sanitation and Hygiene for the Prevention of Diarrhoea. The International Journal of Epidemiology, 39, i193–i205.

Crow, B., Swallow, B., and Asamba, I. 2012. Community Organized Household Water Increases Not Only Rural Incomes, But also Men’s Work. World Development, 40(3), 528–541.

Central Statistical Agency (CSA) and ICF International. 2012 Ethiopia Demographic and Health Survey 2011. Addis Ababa, Ethiopia and Calverton, Maryland, USA.

Esrey, S.A. 1996. Water, Waste, and Well-being: A Multi-country Study. American Journal of Epidemiology. 143(6), 608–623.

Federal Democratic Republic of Ethiopia (FDRE). 2004. Water Supply and Sanitation Implementation Manual.

Federal Democratic Republic of Ethiopia. 2013. Water Supply and Sanitation Program: WASH II. Final Document on Environmental and Social Management Framework, E4363. Addis Ababa, Ethiopia.

Fewtrell, L., Kaufmann, R.B. Kay, D., Enanoria, W., Haller, L. and Colford, J.M. 2005. Water, Sanitation, and Hygiene Interventions to Reduce Diarrhoea in Less Developed Countries: A Systematic Review and Meta-analysis. Lancet Infectious Diseases, 5, 42–52.

Gangle, M., 2004. RBOUNDS: Stata Module to Perform Rosenbaum Sensitivity Analysis for Average Treatment Effects on the Treated. Available at: www.wz-berlin.de/ars/ab/people/gangl_wkp.de.htm.

Gertler, P.G., Martinez, S., Premand, P., Rawlings, L.B. and Vermeersch, C. M. J. 2011. Impact Evaluation in Prac-tice. The World Bank, Washington DC.

Heckman, J., Ichimura, H. and Todd, P. 1998. Matching As An Econometric Evaluation Estimator. Review of Economic Studies, 65, 261–294.

Hirano, K., and Imbense G. and Ridder, G. 2003. Efficient Estimation of Average Treatment Effects Using the Esti-mated Propensity Score. Econometrica, 71, 1161–1189.

IOB and BMZ. 2011. Impact Evaluation of Drinking Water Supply and Sanitation Programmes in Rural Benin: The Risk of Vanishing Benefits.

77Annexes

An ID

EV Im

pact

Eva

luat

ion

IOB and UNICEF. NODATE. Impact Evaluation of Drinking Water Supply and Sanitation Interventions in Rural Mozambique: More Than Water.

IOB. 2008. Support to Rural Water Supply and Sanitation in Dhamar and Hodeidah, Republic of Yemen. Printing OBT bv, The Hague, The Netherlands.

Leuven, E. and Sianesi, B. 2003. PSMATCH2: Stata Module to Perform Full Mahalanobis and Propensity Score Matching, Common Support Graphing, and Covariate Imbalance Testing. Boston College Department of Econom-ics, Statistical Software Components. Downloadable from http://ideas.repec.org/c/boc/bocode/s432001.html.

Lokshin, M. and Yemtsov, R. 2005. Has Rural Infrastructure Rehabilitation in Georgia Helped the Poor? The World Bank Economic Review, 19(2), 311–333.

Miguel, E. and Kremer, M. 2004. Deworming: Identifying Impacts on Education and Health in the Presence of Treatment Externalities. Econometrica, 72(1), 159–217.

Ministry of Education (MoE). 2012/2013. Education Statistics Annual Abstract. Addis Ababa, Ethiopia.

Ministry of Health (MoH). 2012. National Monitoring and Reporting System for the Implementation of Commu-nity-Led Total Sanitation and Hygiene. Addis Ababa, Ethiopia.

Ministry of Water and Energy (MoWE). 2013. National Water Supply, Sanitation and Hygiene Program (WASH). Annual Report for 2005 EFY (July 8, 2012 to July 7,v2013).

Mu, R. and van de Walle, D. 2011. Rural Roads and Local Market Development in Vietnam. Journal of Develop-ment Studies, 47(5), 709–734.

Rauniyar, G., Orbeta Jr. A. and Sugiyarto, G. 2011. Impact of Water Supply and sanitation Assistance on Human Welfare in Rural Pakistan. Journal of Development Effectiveness, 3(1), 62–102.

Rosenbaum, P.R. 2010. Design of Observational Studies. Springer, Heidelberg, Germany.

Rosenbaum, P.R. and Rubin, D.B. 1983. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70, 41–55.

Sijbesma, C., Verhagen, J., Nanavaty, R. and James, A.J. 2009. Impacts of Domestic Water Supply on Gender and Income: Results from a Participatory Study in a Drought-prone Region in Gujarat, India. Water Policy, 11(1), 95–105.

Smith, J. and Todd, P. 2005. Does Matching Overcome LaLonde’s critique of Non-experimental Estimators? Journal of Econometrics, 125, 305–353.

The World Bank. 2013. Ethiopia: Water Supply and Sanitation Project. Report No: ICR00002933.

UNICEF and Government of the Netherlands. 2011. Impact Evaluation of Drinking Water Supply and Sanitation Interventions in Rural Mozambique. One Million Initiative: Mozambique.


Waddington, H. and Snilstveit, B. 2009. Effectiveness and Sustainability of Water, Sanitation and Hygiene Inter-ventions in Combatting Diarrhoea. Journal of Development Effectiveness, 1(3), 295–335.

Waddington, H., Snilstveit, B., While, H. and Fewtrell, L. 2009. Water, Sanitation and Hygiene Interventions to Combat Childhood Diarrhoea in Developing Countries. Synthetic Review. International Initiative for Impact Evaluation.

Wagstaff, A., Lindelow, M., Jun, G., Ling, X. and Juncheng, Q. 2009. Extending Health Insurance to the Rural Population: An Impact Evaluation of China’s New Cooperative Medical Scheme. Journal of Health Economics, 28(1), 1–19.

Winters, P., Salazar, L. and Maffioli, A. 2010. Designing Impact Evaluations for Agricultural Projects. Impact Eval-uation Guidelines. Inter-American Development Bank.

79Annexes

An ID

EV Im

pact

Eva

luat

ion

Endnotes

1. The non-RWSSP group was created using propensity score marching (PSM), and this was done at the community level.

2. In the framework of the WHO/UNICEF Joint Monitoring Program (JMP) of MDG improved water and sanitation targets, an improved

water source is defined as one that by the nature of its construction and when properly used, adequately protects from outside

contamination, particularly from fecal matter.

3. In the framework of the WHO/UNICEF Joint Monitoring Program of MDG improved water and sanitation facilities, an improved

sanitation facility is defined as one that hygienically separates human excreta from human contact. Such facilities include pit

latrines with a concrete or other type of slab that can be properly cleaned.

4. Reference AfDB Ethiopia RWSSP Appraisal Report 2005, page v.

5. Reference AfDB Ethiopia RWSSP Appraisal Report 2005, page v.

6. Other major water supply and sanitation programs were in non-AfDB woredas and supported by other development partners

including the World Bank, DFID, UNDP, UNICEF, Agence Française de Développement (AFD), Finnish Development Agency (FINIDA),

Japan International Cooperation Agency (JICA), GTZ, KfW, CIDA, Development Cooperation Ireland, Government of the Netherlands,

and European Union (African Development Bank, 2005).

7. Except the Addis Ababa and Dire Dawa Administrative areas. At appraisal, the program targeted 120 woredas but with the upward

rationalization of the number of woredas in the country during implementation, the RWSSP woredas increased to 125.

8. Programs funded by the World Bank and the Department for International Development for rural water supply and sanitation used

the same implementation manual.

9. By the beginning of July 2013, the disbursement rate of the AfDB grant amount was approximately 77%, of which approximately

76% was on goods and works and 23% on services (MoWE, 2013; aide memoire supervision mission RWSSP 2014).

10. The evaluation team requested that the AFDB Ethiopia Field Office contact the MoWE to ask for an explanation of the finding that in

these 3 woredas listed as AfDB RWSSP-supported woredas no interventions had taken place. No explanation has been provided as yet.

11. In effect, hand-dug and shallow wells should be classified as improved if they have internal linings and are protected from bird droppings.

12. Unless otherwise stated, time spent per round trip for fetching water is meant to include travelling time to reach the water source,

time spent queuing and filling water containers at the source, and travel time to return home from the water source.

13. Because school dropout rates were very low (about 1%) in program and non-program communities, no econometric estimation of

impact on dropout rates was undertaken as it does not provide much economic meaning.

14. During the field survey, 1 USD was worth about 20.18 ETB.

15. The main reason for excluding a control community that borders a treatment community is to reduce potential spillovers between

treatment and control communities.

16. We want to implement this to minimize any potential substitution bias that may arise if control communities have accessed

improved water supply and sanitation services because of interventions by another external source.


idev.afdb.org

Layo

ut &

pro

duct

ion:

Vis

ual I

dent

ity –

ww

w.v

isua

liden

tity.

co.u

k

About this Evaluation

The Ten-Year Strategy of African Development Bank (AfDB) includes a commitment to undertaking more impact evaluations so as to obtain credible evidence on the development effects of its supported programs and projects.

This evaluation examines the impact of the Rural Water Supply and Sanitation Program (RWSSP) of the African Development Bank Group (AFDB) on key aspects of the livelihoods of people in rural Ethiopia. The program was implemented from 2006 to 2014 at a cost of UA 60.2 million (USD 82.5M). It was financed by the AfDB ADF grant (72%) and the Government and communities in Ethiopia (28%).

The evaluation compiles credible evidence about the impacts of the program on (i) access to and use of safe water, (ii) diarrhea incidence among under-fives, (iii) children’s school attendance, and (v) women’s participation in self-employment; and draws lessons and makes recommendations to the AfDB for sustaining the benefits of the program; and implementing similar programs in the future.

The evaluation finds that the program had significant impact on increasing the rural population’s access to less contaminated water; Reducing diarrhea incidence, but negligibly for children under five; Saving the time spent by rural households on fetching water. The program didn’t have visible impact on school enrolment by children or on women’s participation in income generating activities. The evaluation recommends that the Bank develops clear strategies for sustaining the benefits of the program, and implement effective monitoring, evaluation and learning systems for similar future programs.

An IDEV Impact Evaluation


African Development Bank GroupAvenue Joseph Anoma, 01 BP 1387, Abidjan 01, Côte d’IvoirePhone: +225 20 26 20 41E-mail: [email protected]

An IDEV Im

pact EvaluationIm




ent Evaluation

http://idev.afdb.org/

http://www.visualidentity.co.uk

mailto:idevhelpdesk%40afdb.org?subject=

Documents

African Development Bankidev.afdb.org/sites/default/files/documents/files/IDEV...Independent Development Evaluation African Development Bank An IDEV Impact Evaluation Impact Evaluation