Impact Evaluation of the MCC-Funded Rural Water … › 2016 › 02 › mcc...2014/02/07 · 10 15 10 15 Baseline Study (May-July 2011) August 2011 – April 2013 Follow-up Study

Ralph Hall (Co-PI), Virginia TechJenna Davis (Co-PI), Stanford University

Eric Vance, Virginia TechEmily Van Houweling, Virginia Tech

February 7, 2014

Impact Evaluation of the MCC-Funded Rural Water Supply Activity (RWSA) in Nampula, Mozambique

Rural Water Supply Activity (RWSA) Installation of 600

handpumps in rural communities across the provinces of Nampula (358) and Cabo Delgado (242)

Installation of 8 small scale solar systems in Cabo Delgado

2

3

The objectives of the RWSA, as stated in the Compact, are to increase beneficiary productivity and income through:

4

Time savings Reducing water-related illnesses (diarrhea,

dysentery, etc.)

Impact Evaluation of Rural Water Points Installation Program (RWPIP)

in Nampula

5

6

Demand Response ApproachCommunities submitted an application + Contributed 2,500 MZN ($86 USD)

7

8

9

10

11

The water committee received training on: Handpump operation and

maintenance Hygiene and sanitation

(PHAST or CLTS)

PHAST = Participatory Hygiene And Sanitation Transformation

CLTS = Community Led Total Sanitation

Research Approach

12

Principle Objective of Impact Evaluation Impact evaluations seek to provide confident causal

inference about the link between an intervention and outcomes

Difficulty is determining what would have happened to the individuals/communities of interest in absence of the project

Our Task: Identify the impacts of the installation of handpumps in rural communities in Nampula from all other confounding factors

13

Research Approach Develop Panel Data: Compare observed changes in

the outcomes for a sample of participants and non-participants

Key Assumption: In the absence of the program, communities selected into the participant and non-participant groups would be changing at the same rate

14

Treatment Comparison

Baseline

Follow-up

t0

t1

t0

t1

HandpumpsInstalled

Research Design for Phase 2 Communities

15

Selection of Treatment Communities Treatment communities were randomly selected from

the communities included in Phase 1 and 2 of the Rural Water Points Installation Program (RWPIP) in Nampula

Phase 1 Districts: Meconta, Mogovolas, and Nampula-Rapale

Phase 2 Districts: Moma, Mogincual, and Murrupula

16

Selection of Comparison CommunitiesVisited District Office Informed District Government of impact evaluation study Obtained permission to undertake the studyVisited Localidade Informed Localidade Authority of impact evaluation study Obtained permission to undertake the study Developed list of potential comparison communities with

the Chefe de Localidade Dry communities were excluded

The Chefe de Localidade randomly selected the comparison communities (one for each treatment community in Localidade)

17

18

Phase 1 Treatment

100 handpumps in Meconta,

Mogovolas, and Nampula-Rapale

HandpumpsInstalled

1510

1510Baseline Study(May-July 2011)

August 2011 –April 2013

Follow-up Study (May-July 2013) 6

6

Phase 1 Comparison

Communities in Meconta,

Mogovolas, and Nampula-Rapale

without a handpump

Phase 2 Treatment

146 handpumps in Moma, Mogincual,

Murrupula, and Mogovolas

Phase 2 Comparison

Communities in Moma, Mogincual,

Murrupula, and Mogovolaswithout a

handpump

23

23

HandpumpsInstalled

Sample Frame at Follow-up(2013)

19

20

20

21

Final Sample Frame Community

Classification

Number of Communities in

Group

Number of Communities by

District

Phase 1

Treatment 104 Meconta3 Mogovolas3 Rapale

Comparison 62 Meconta1 Mogovolas3 Rapale

Phase 2

Treatment 15

8 Mogincual3 Murrupula2 Mogovolas2 Moma

Comparison 23

4 Mogincual8 Murrupula1 Mogovolas10 Moma

22

Sample Validity Following the baseline study:

9 treatment communities became comparison communities

8 comparison communities became treatment communities

ANOVA test comparing the overall difference in means between treatment and comparison communities at baseline showed that for 13 of 15 key variables, the differences were not statistically significant

23

Data Collection

24

Data Collection Activities (RWSA)Activity 2011 Baseline Study 2013 Follow-up Study

Household Surveys 1,579 (54 communities: 27

treatment; 27 comparison)

1,826 (62 communities: 32

treatment; 30 comparison)

Water Committee/ Leader Interviews 54 31

Water Sampling11 communities (39 community water

sources and 259 household containers)

11 communities (32 community water

sources and 873 household containers; water source

variability tested in 4 communities)

Handpump Observations NA 17(17 communities)

25

73% of the households interviewed during the baseline

study were surveyed again in the follow-up study

26

Fieldwork Preparation Household surveyors and water quality testing team members

were trained for 2 weeks A pilot study was undertaken to test instruments and fieldwork

protocols

27

Fieldwork Preparation Following pilot – surveyors were retrained and instruments/

protocols were revised Three household surveying teams consisted of:

• 1 team leader • 3 household surveyors• 1 driver

28

Fieldwork Preparation Stanford-VT-WE Consult team supported the water sampling

team (consisting primarily of Universidade Lúrio students) in the field and laboratory work

29

Household Survey Teams (in field)

30

Household Survey Household surveys undertaken using PDAs Data were cleaned during fieldwork Enumerators were provided with feedback on their data entry

errors and outliers were checked Feedback dramatically reduced the number of recurring

errors

Summary data were sent to the MCA/MCC every two weeks during fieldwork

31

Respondent/Household Characteristics

Baseline Follow-upMean Median Mean Median

Age of survey respondent 39.6 38 39.7 37% female 38% – 44% –% literate 32% – 32% –Number in household 4.2 4 4.2 4Number of children < 5 0.7 1 0.7 1

32

Water Sources Used by Households

33

78% of the households surveyed in the treatment communities reported using a handpump

% of HHs Using SourceBaseline Follow-Up

Handpump 9% 78%Unprotected Well 85% 21%River/Lake 16% 9%

Phase 2 Treatment – Percent of Households Using Source and Percent of Total Water Collected from Source

34

% of HHs Using SourceBaseline Follow-Up

Handpump 10% 2%Unprotected Well 78% 65%River/Lake 15% 35%

Phase 2 Comparison – Percent of Households Using Source and Percent of Total Water Collected from Source

The water sources used by households in the comparison communities remained relatively

unchanged from the baseline to follow-up study

35

Handpump UseWater Committee Estimate (n=28) ValueMedian number of people using the handpump in the dry season (Aug-Nov) 391

Median number of people using the handpump in the wet season 176

Handpump Observation (n=17) Value

Average number of water collectors per hour 6

Average number of water collectors per day (8 hours) 48

Average number of trips per household 2.3

Estimated number of people served by the handpump 88

36

Water Consumption

All Sources and Improved Sources

37

The installation of the MCA handpumps are associated with an insignificant 2.5 LPCD increase in median water consumption (from all sources) (p<0.1)

Phase 2 Median Total Liters per Capita per Day (LPCD) (All Sources)

Number of Communities

Baseline Follow-Up DifferenceMean of

Median LPCDMean of

Median LPCD LPCD

Treatment 15 17.2 19.5 2.3.

Comparison 23 18.5 18.3 -0.2

Difference in Differences 2.5.

Significance codes: *** p<0.001 ** 0.001>p<0.01 * 0.01>p<0.05 . 0.05>p<0.10

38

p < 0.001 = very strong evidence that there is a difference between treatment and comparison

0.001 < p < 0.01 = strong evidence that there is a difference …

0.01 < p < 0.05 = evidence that there is a difference …

0.05 < p < 0.1 = indication that there is a difference …

39

The installation of the MCA handpumps are associated with an 15.1 LPCD increase in median water consumption (from improved sources) (p<0.001)

Phase 2 Median Total Liters per Capita per Day (LPCD) from Improved Sources


Baseline Follow-Up DifferenceMean of

Median LPCDMean of

Median LPCD LPCD

Treatment 15 0.0 15.1 15.1***

Comparison 23 1.8 0.2 -1.6

Difference in Differences 16.7***


40

In treatment communities, 3 out of every 4 buckets of water collected are from an improved source

Baseline Follow-Up DifferencePhase/

CommunityNumber of

CommunitiesMean of Median

LPDMean of Median

LPD LPD

Treatment (all sources) 15 65.4 76.5 11.1*

Treatment (improved) 15 0.0 58.0 58.0***

Comparison (all sources) 23 75.6 68.5 -7.1

Comparison (improved) 23 7.5 1.3 -6.2


Phase 2 Median Total Liters per Households per Day (LPD)

41

Time Spent Collecting Water

42

Females account for three quarters (76%) of the total time spent collecting water

8%

9%

7%

10%

13%

53%

Males Age 5-11

Males Age 12-17

Males Age 18+

Females Age 5-11

Females Age 12-17

Females Age 18+

43

Following the installation of the MCA handpumps there was an 88-minute decline in

the time households spent collecting water from all sources, but this decline was

statistically insignificant

But…

44

The installation of the MCA handpumps can be associated with a 62-minute reduction in the median roundtrip time to the ‘primary’ source (p<0.05)

Phase 2 Median Roundtrip Time to Primary Source


Baseline Follow-Up DifferenceMean of Median Time (Minutes)

Mean of Median Time (Minutes) Minutes

Treatment 15 161 76 -85**

Comparison 23 137 114 -23

Difference in Differences -62*


45

The wait time at the primary source in treatment communities declined by 41 minutes relative to comparison communities (p<0.05)

No statistically significant change was found in the one-way walk times to the primary source

46

The installation of the MCA handpump can be associated with a 30% reduction in the total median time females (aged 12 and above) spend collecting water each day

There was no overall reduction in the time males spent collecting water

47

By comparing the time and water volume data by demographic groups, the installation of the MCA handpump can be associated with …

an increase in the quantity of water collected by girls and boys aged 12-17 and women aged 18 and above, …

but a decline in the time these groups spend collecting water

48

The installation of the MCA handpumps can be associated with a 55-minute reduction in the median time to collect 20 liters of water (p<0.001)

Phase 2 Median Time to Collect 20 Liters of Water


Baseline Follow-Up DifferenceMean of Median Time (Minutes)

Mean of Median Time (Minutes) Minutes

Treatment 15 104 62 -42*

Comparison 23 86 99 13

Difference in Differences -55***


49

Probability of Using the Installed Handpumps

50

As distance to the nearest handpump increases, the probability that a household will use the handpumpdecreases. The distance at which the probability of using a handpump drops below 0.5 is 1.2 km.

51

Reasons for Not Using a Handpump 22% of households in the treatment communities do

not use the handpump

Reason for Not Using Handpump

Percent of Households (n=170)

Distance 64.7%Too expensive 28.8%Don’t like taste 14.1%Closed or broken 7.1%Too crowded 6.5%Not permitted to use 5.9%Conflicts 1.2%

52

Impact on Schooling

53

MCA handpumps are associated with a 17.5% reduction in the mean percentage of households stating that water fetching negatively affects the school attendance of their children (p<0.01)

Phase 2 Mean Percentage of Households (HHs) Stating that Water Fetching Affects School Attendance


Baseline Follow-Up DifferenceMean % HHs Stating that Water Fetching

Affects School Attendance

Mean % HHs Stating that Water Fetching

Affects School Attendance

Change in Percentage

Treatment 15 26.7% 7.1% -19.6%**

Comparison 23 16.8% 14.7% -2.1%

Difference in Differences -17.5%**


Satisfaction with Water Supply

55

MCA handpumps are associated with a 63% increase in respondent satisfaction with their water supply relative to comparison communities (p<0.001)

Phase 2 Percentage of HH Indicating Satisfaction with Water Supply Situation


Baseline Follow-Up DifferenceMean Percent of HH Satisfied

Mean Percent of HH Satisfied

Change in Percentage

Treatment 15 22% 79% 57%***

Comparison 23 31% 26% -6%

Difference in Differences 63%***


56

Water Quality Testing

57

Water Sampling Methodology

58

Source and household stored water sampled in 11 communities during baseline and follow-up studies

Source water quality variability study undertaken in 4 communities (follow-up study only)

All samples processed tested for fecal indicator bacteria (E. coli results presented here)

IDEXX protocol used to determine most probable number (MPN) of colony forming units (CFU) of E. coli in each sample Results presented in terms of 0-10, 11-100, and >100

CFU/100mL (MPN) as per older WHO guidelines and current custom among WASH researchers

Water Quality at Point of Collection

59

Variability of Water Quality at Source

60

Quality of Stored Household Water by Study Phase: Pooled data from 7 treatment communities

61

Quality of Stored Household Water at Follow-up: Source-stratified data from 7 treatment communities

62

Health, Hygiene, and Wealth

63

The Installation of the MCA Handpump was not associated with significant changes in:

Health

Percentage of children with reported symptoms of gastrointestinal or respiratory illness in week prior to interview

Sanitation and Hygiene

Self-reported handwashing practices, latrine use, or satisfaction with household’s sanitation situation

Wealth

Household income and expenditure

64

However, the percentage of households using a latrine did increase by 10% in the treatment communities (p<0.01)

Phase 2 Percentage of Households Using Latrines


Baseline Follow-Up Difference% HHs Using

Latrine% HHs Using

LatrineChange in

Percentage

Treatment 15 23% 33% 10%**

Comparison 23 16% 18% 2%

Difference in Differences 8%.


65

Incomes and expenditures increased in both treatment and comparison communities along with household engagement in agriculture and consumption of meat and fish, pointing to a general trend of economic development in Nampula (or a productive farming season)

66

Are the Impacts Sustained?

67

An analysis of Phase 1 treatment and comparison communities revealed no significant changes in the key variable of interest

This suggests that the various impacts observed due to the installation of the MCA handpumps have been sustained for at least two years

It also indicates that there has been no significant increase in benefits over time

68

Policy Implications from RWSA

69

Policy Implications “Distance” was the number one reason households did not

use the handpumps

It may be necessary to consider alternative approaches to selecting sites for handpump installation, as well as criteria for installing multiple water points or small piped systems

Future demand responsive programs should include a broader range of water technologies and management models

Consider alternative approaches to sanitation and hygiene promotion that result in broader reach and better uptake of key messages

70

Policy Implications Local governments may require additional financial

support to cover major system repairs

Follow-up trainings with the water committee may be required in the areas of financial management, and operation and maintenance

A small stipend or incentive may also be necessary to ensure that the water committees continue to function at a high level over the life of the handpump

Explore the use of new handpump maintenance models

71

Cabo DelgadoJune 18-July 3, 2013

Andy Ismali Buanando and Valdimira Caetano

Small Scale Solar Systems (SSSS)

Research Questions1. How are the water systems functioning from a

financial, technical, and managerial perspective?

2. To what extent do the water systems satisfy the water users’ needs and preferences?

3. What are the prospects for the long-term sustainability of the water systems in Cabo Delgado?

73

SSSS Technology 7.5m tower Tank 10 cubic meter 3 standpipes, each with 2

taps and water meter 6-10, 250 W solar panels 72 to 96 V DC 96mm

submersible pumps Designed to serve 1,500

people

74

• Population greater than 2,000 people

• Borehole yield greater than 3,000 L/h

• Low water coverage rate and unmet demand

SSSS Communities in Cabo Delgado

75

Selection Criteria

Research Methods Interviews with SDPI

directors Interviews with Cowater

animators Interviews with water

committees Interviews with operators

Focus groups with water users

Interviews with community leaders

One day of observation at the SSSS or records of water use

76

Water Use Water used mostly for domestic purposes (some brick

making and food preparation) Wide variation of water use between systems (213-676

users/day and 2-22 cubic meters/day) Highly seasonal usage (highest demand Sept-December)

77

Average Number of People Served per Day by System

Data from observations (num. of people on observation day)Data from records (avg. num of users per day)

78

0100200300400500600700800

Number of Daily SSSS Users in Muiha

0

200

400

600

800

1000

1200

1400

1600

79

Number of Users

Finances All communities have positive account balances Savings vary between systems (300-55,000 Mt.) Average monthly expenses: 3,600 Mt. Revenue highly seasonal (example: 900-14,458 in 50

Casas)

80

Community-Led Management

SDPI

Water committeeOperator

Tap attendants Guard

Water users

District Comm. name

Type of mngt. model

Nagade Chitunda WC and OP

Nagade Muiha WC and OP

Nagade Ntanga WC and OP

Nagade 50 Casas WC and OP

Palma Manguna WC only

Palma Mute 3 WC and OP

Mocimboade Praia Diaca WC and OP*

Mocimba de Praia Malinde WC only

81

Management Challenges No supervision of the

operator Lack of financial

transparency Poor communication

between the users, operator, and SDPI

Water users are not well informed

Lack of technical competence

82

Technical Performance System supplies enough water

to meet water demand All systems functioning well,

except Chitunda and Muiha At least one breakdown in 6 of

8 systems Technical problems mostly

related to electrical issues Long duration of breakdowns

(45-90 days) Two solar panel robberies All breakdowns repaired by

Cowater / GM Todd Irrigation

83

User SatisfactionSatisfied Somewhat

satisfied Not satisfied

Type of technology 100%

Location of taps 87.5% 12.5%

Water quality 87.5% 12.5%

Hours system is open 75% 12.5% 12.5%

Price of water 63% 37%

Reliability 25% 62.5% 12.5%

Management 25% 37.5% 37.5%

84

Major Sustainability Challenges The majority of the

communities will have difficulty fixing major technical problems

Lack of financial transparency

Lack of communication among management groups

No supervision of operator

85

86

Questions?

Impact Evaluation Team Co-Principal Investigators: Dr. Jennifer Davis (Stanford University) Dr. Ralph Hall (Virginia Tech)

Core Team Members: Dr. Eric Vance (Virginia Tech) Dr. Emily Van Houweling (Virginia Tech) Marcos Carzolio (Virginia Tech) Mark Seiss (Virginia Tech) Kory Russel (Stanford University) Wouter Rhebergen (WE Consult)

87

Documents

Impact Evaluation of the MCC-Funded Rural Water … › 2016 › 02 › mcc...2014/02/07 · 10 15 10 15 Baseline Study (May-July 2011) August 2011 – April 2013 Follow-up Study