Validation Report December 2020

India: Rajasthan Urban Sector Development Investment Program (Tranche 3 and Multitranche Financing Facility)

Reference Number: PVR-668 MFF Number: M0015 Project Number: 40031-013 and 40031-053 Loan Number: 2725

ABBREVIATIONS

ADB – Asian Development Bank DMF – design and monitoring framework EIRR – economic internal rate of return ha – hectare IED – Independent Evaluation Department IPIU – investment program implementation unit IPMU – investment program management unit km – kilometer lpcd – liters per capita per day MFF – multitranche financing facility mld – million liters per day O&M – operation and maintenance OCR – ordinary capital resources PCR – project completion report RRP – report and recommendation of the President RUSDIP – Rajasthan Urban Sector Development Investment Program SARD – South Asia Department (ADB) SRP – short resettlement plans STP – sewage treatment plant SWM – solid waste management ULB – urban local body

NOTE

In this report, “$” refers to United States dollars.

Director General Deputy Director General Director Team Leader

Marvin Taylor-Dormond, Independent Evaluation Department (IED) Veronique Salze-Lozac’h, IED Nathan Subramaniam, Sector and Project Division (IESP) Tomoo Ueda, Principal Evaluation Specialist, Thematic and Country Division (IETC)

The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. The final ratings are the ratings of IED and may or may not coincide with those originally proposed by the consultants engaged for this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

PROJECT BASIC DATA Project number 40031-013 and

40031-053 PCR circulation date 30 July 2019

Loan number 2725 PCR validation date Dec 2020 Program name Rajasthan Urban Sector Development Investment Program

(Tranche 3 and Multitranche Financing Facility) Sector and subsector

Water and other urban infrastructure and services

Other urban services Urban policy, institutional and capacity development Urban sewerage Urban water supply

Strategic agenda Environmentally sustainable growth Inclusive economic growth

Safeguard categories

Environment B Involuntary resettlement B Indigenous peoples C

Country India Approved ($ million)

Actual ($ million)

ADB financing ($ million)

ADF: 0.00 Total project costs 390.00 323.74 OCR: 273.00 Loan

2366 2506 2725

60.00

150.00 63.00

50.02

117.26 52.60

Borrower 2366 2506 2725

15.00 69.00 33.00

13.52 66.93 23.41

Beneficiaries 0.00 0.00 Others 0.00 0.00

Cofinancier Total cofinancing 0.00 0.00 Approval date

2366 2506 2725

8 Nov 2007 19 Jan 2009 13 Dec 2010

Effectiveness date 2366 2506 2725

16 Apr 2008 19 May 2009 16 Jun 2011

28 Feb 2008 20 Apr 2009 16 Jun 2011

Signing date 2366 2506 2725

17 Jan 2008 18 Feb 2009 17 Mar 2011

Closing date 2366 2506 2725

30 Jun 2013 30 Jun 2014 30 Jun 2015

22 May 2015 29 Nov 2017 24 Nov 2017

Project officers

S. Penjor M. Sharma P. Srivastava

Location ADB Headquarters ADB Headquarters India Resident Mission

From Oct 2010 Jun 2012 Jan 2015

To Jun 2012 Jan 2015 Nov 2017

IED review Director Team Leader

N. Subramaniam, IESP Tomoo Ueda, Principal Evaluation Specialist, IETC*

ADB = Asian Development Bank, ADF = Asian Development Fund, IED = Independent Evaluation Department, IESP = Sector and Project Division, IETC = Thematic and Country Division, OCR = ordinary capital resources, PCR = project completion report. *Team members: A. Brubaker (Quality Reviewer), F. De Guzman (Senior Evaluation Officer), and D. Corderi and D. Gibson (Consultants).

2

I. PROJECT DESCRIPTION A. Rationale 1. Rajasthan is India’s largest state with an area of 350,000 square kilometers and a population of 56.5 million (2001 census). Its urban centers have been suffering from poor basic urban services. Access, availability, and quality of piped water, in-house sanitation, and solid waste management (SWM) services, as well as urban road infrastructure were inadequate. These were due to population pressures, increased influx of tourists, lack of adequate funding, poor public services management and operation, and the weak capacity of urban local bodies (ULBs). The Asian Development Bank (ADB) approved the Rajasthan Urban Sector Development Investment Program (RUSDIP) in October 2007 for $390.0 million.1 The RUSDIP was designed as a multitranche financing facility (MFF) aimed to deliver essential municipal infrastructure and services, including improvements in water supply, wastewater management, SWM, urban drainage, urban transport and roads, and social infrastructure. The investment program also aimed to provide capacity development and program management support to 15 medium-sized ULBs to foster a sustainable management of urban services. It was expected to increase economic growth, reduce poverty, and sustain improvements in the urban environment and quality of life for the population in the ULBs. The MFF consisted of three projects to be financed through three tranches: (i) tranche 1 was closed in May 2015, executing $63.5 million2 in three ULBs;3 (ii) tranche 2 was closed in November 2017, executing $117.3 million in 14 ULBs;4 and (iii) tranche 3 was closed in November 2017, executing $52.6 million in 15 ULBs.5 2. The use of the MFF modality was deemed appropriate due to its clear strategic long-term sector road map, creating the right conditions for a long-term urban sector partnership with ADB. The modality was to allow flexibility in making investment decisions according to what the ULBs required within the sector road map and to the needs and constraints of the investment program of the ULBs. The MFF was also to support the implementation of ongoing institutional and financial reforms, and to build the capacity of urban institutions and local bodies with individual loans—in tranches linked to reform targets over a realistic time frame. ULBs were expected to assume responsibility for providing urban services. The MFF’s main focus was placed on institutional change and financial reforms, particularly cost recovery and the long-term sustainability of the assets and services. This included various instruments, such as increasing tariffs and imposing service charges or property taxes to achieve cost recovery. ULBs were to be responsible for providing urban services in a phased manner as the assets were transferred, and the targeted ULBs gained the resources and revenues to maintain them. 3. ADB completed three project completion reports (PCRs): two tranche PCRs and one facility completion report that included the PCR for tranche 3.6 This validation pertains to tranche

1 ADB. 2007. Report and Recommendation of the President to the Board of Directors: Proposed Multitranche

Financing Facility to India for the Rajasthan Urban Sector Development Investment Program. Manila. 2 ADB. 2017. Completion Report: Rajasthan Urban Sector Development Investment Program–Tranche 1. Manila. 3 The ULBs in Alwar, Jaisalmer, and Jhalawar–Jhalarapatan. 4 ADB. 2019. Completion Report: Rajasthan Urban Sector Development Investment Program–Tranche 2. Manila. At

appraisal, tranche 2 was expected to benefit 12 ULBs and was later expanded to 14 ULBs. 5 Of the 15 ULBs selected, two pairs of ULBs were twin towns and were counted as one each: Jhalawar–Jhalarapatan,

and Baran–Chabra. No interventions were taken up in Chabra. 6 ADB. 2019. Completion Report: Rajasthan Urban Sector Development Investment Program–Tranche 3 and MFF.

Manila.

3

3 and the facility completion report.7 ADB approved the third periodic finance request in December 2010 for the third project under the MFF (tranche 3) in the amount of $63 million.8 The MFF, including project 3, comprised investments9 in (i) urban infrastructure improvements10 (water supply interventions, sewerage and sanitation, urban drainage, urban transport and roads, SWM, social infrastructure, and support infrastructure for cultural heritage); and (ii) capacity development and implementation support. The third tranche targeted 15 ULBs.11 As for the whole MFF, 15 ULBs were covered with three tranches combined. B. Expected Impacts, Outcomes, and Outputs 4. At appraisal, the MFF’s design and monitoring framework (DMF) indicated that the project’s expected impacts were increased economic growth and reduced poverty, and sustained improvement in the urban environment and quality of life in Rajasthan’s initially identified ULBs. There were two expected project outcomes: (i) increased access to sustainable urban infrastructure and services for 1.6 million people in the investment program ULBs by the end of the program; and (ii) improved capacity and sustainable management of urban services by the investment program ULBs by the end of the program. Expected project outputs were implemented investment programs for (i) water supply, (ii) wastewater management, (iii) SWM, (iv) urban drainage, (v) urban roads and transport, (vi), social infrastructure, (vii) support infrastructure for cultural heritage, (viii) capacity development, and that the (ix) investment program was implemented and well managed. 5. The DMF for project 3 established its expected impacts, outcomes, and outputs.12 The project’s expected impact was increased access to sustainable infrastructure and services for 2.16 million people in Rajasthan’s 15 ULBs.13 The project’s intended outcome was improved urban services in the 15 ULBs. There were eight expected outputs: (i) improved and increased metered water supply service area in Baran and Chittorgarh; (ii) increased sewerage treatment coverage in the five ULBs; (iii) reduced flooding through improved urban drainage in Sawal Madhopur; (iv) traffic flow improved as a result of construction of two minor bridges in Sawal Madhopur and one road in Dholpur; (v) increased solid waste collection and disposal in 10 ULBs; (vi) firefighting strengthened in at least five ULBs; (vii) heritage sites in seven ULBs conserved; and (viii) implementing agencies effectively manage project-financed assets.

7 The project validation reports for tranches 1 and 2: (i) ADB. 2019. Rajasthan Urban Sector Development Investment

Program–Tranche 1. Manila; and (ii) ADB. 2020. Rajasthan Urban Sector Development Investment Program–Tranche 2. Manila.

8 ADB. 2010. Periodic Financing Request Report. MFF 0015-IND: Rajasthan Urban Sector Development Investment Program—Proposed Tranche 3. Manila.

9 The heritage component was taken up only under project 3, the solid waste component was not taken up under project 2, and the firefighting component was included during appraisal of project 3.

10 Project 2 interventions did not include SWM and heritage components originally included as part of the MFF program. 11 Alwar, Baran, Barmer, Bharatpur, Bundi, Chittorgarh, Churu, Dholpur, Jaisalmer, Jhalawar, Karauli, Nagaur,

Rajsamand, Sawai Madhopur, and Sikar. Jaisalmer was added as the 15th ULB from tranche 2. 12 Included in Appendix 2 of the periodic financing request (PFR) report. 13 Although, at the time of appraisal, it was mentioned that the project 3 was expected to benefit 12 ULBs with (i)

increased average duration of drinking water in two ULBs; (ii) nonrevenue water reduced to 20% in two ULBs; (iii) increased population with access to piped sewerage provided in five ULBs; and (iv) 50% of households covered with house-to-house municipal waste collection in all 12 ULBs, with 25% coverage of slum and low-income or women-headed households. The difference of 12 and 15 are as follows: of the 15 ULBs selected, two pairs of ULBs were twin towns and were counted as one each—(i) Baran–Chabra and (ii) Jhalawar–Jhalarapatan. No interventions were taken up in Chabra.

4

C. Provision of Inputs 6. MFF. The MFF was approved in October 2007 and was initially planned to be completed by December 2014. It was extended twice by a total of 36 months from December 2014 until June 2017. According to the PCR, the extensions were to (i) use savings attributable to United States (US) dollar appreciation; (ii) use high contingency allocations by including additional subprojects and increasing coverage of ongoing subprojects; and (iii) complete the remaining portions of ongoing contracts under projects 2 and 3 as well as terminated contracts that could be re-awarded only in 2014 and 2015. There was no change in the MFF’s scope, and the extension of the MFF resource availability period did not affect the intended MFF outcomes and outputs. 7. At appraisal, the MFF’s total project cost was estimated at $390.0 million. ADB planned to finance $273.0 million from ordinary capital resources (OCR), and the Government of Rajasthan contributed $117.0 million. At completion, actual project costs were $323.7 million—ADB loan was $219.9 million and the state government’s contribution was $103.8 million. Of the total allocation of $273.0 million under the MFF, $219.9 million was utilized ($50.0 million under project 1, $117.3 million under project 2, and $52.6 million under project 3) and $53.1 million was canceled. 8. Consulting services amounted $21.0 million for the entire MFF. At investment program appraisal, it was proposed to contract national consulting firms to provide investment program management consultants for 443 person-months. National firms were also to provide design and construction supervision consultants for 1,464 person-months. The PCR did not provide the total number of person-months used by the end of the MFF. Actual consulting services costs were $14.9 million for the MFF.14 9. The MFF and all its tranches were classified category B for environment and involuntary resettlement. For indigenous peoples, the MFF, project 1, and project 2 were classified category B, and project 3 as category C. 15 The environmental assessment and review framework, resettlement framework, and indigenous peoples planning framework were prepared during respective project loan processing. ADB approved 45 resettlement plans during MFF implementation, of which 33 resettlement plans were actually implemented. Land acquisition was not involved in project 1, while 0.33 hectares (ha) of land was acquired in project 2. The PCR documented that under the MFF, four affected households were compensated with ₹7.15 million for 0.9 ha of land, and 719 affected households were compensated with ₹7.03 million for temporary loss of livelihood and moving costs. A gender action plan was prepared for project 3 only. 10. ADB provided a bridging technical assistance (TA) to mobilize consultants to assist the state government to prepare the RUSDIP.16 The TA contributed to the shortened investment program preparation period. It delivered its intended outcomes of (i) improving project implementation readiness, (ii) strengthening local capacity and ownership, and (iii) enhancing the efficiency of implementation.17

14 The project costs in Appendix 3 did not differentiate consulting cost expenditures for each tranche 3. 15 For projects 1 and 2, policies applicable were ADB’s environmental policy (2002), indigenous peoples’ policy (1998),

and involuntary resettlement (1995). For project 3, the applicable policy was safeguards policy statement (2009). 16 ADB. 2006. Technical Assistance Cluster to India for Project Processing and Capacity Development. Manila. 17 The activities included: (i) preparation of infrastructure base maps for all the investment program’s ULBs; (ii) conduct

of the necessary survey, data collection, and studies to complete the detailed design of the identified subprojects in the three sample ULBs; (iii) detailed design of the subprojects in the sample ULBs; (iv) assistance in the preparation

5

11. Tranche 1. The MFF’s tranche 1 covered three ULBs and was approved on 8 November 2007, and the loan was effective on 28 February 2008, earlier than the planned date of 16 April 2008. The loan closed on 22 May 2015 after two extensions, almost 2 years after the target date of 30 June 2013. The delayed closing date enabled project 1 to complete additional sewerage subprojects that optimized loan utilization. The actual project cost was $63.5 million, less than the estimated $75.0 million. Civil works were expected to cost $33.8 million, but the actual was $43.3 million, with $4.0 million undisbursed from the revised allocation of $47.3 million. Expenditure on capacity development and program management was $6.7 million, less than the planned $17.3 million, since most work were transferred to subsequent MFF tranches. The originally unallocated sum of $8.9 million for price and physical contingencies was canceled. Consulting services were planned to cost $13.2 million, of which $5.8 million was disbursed. ADB prepared short resettlement plans (SRPs) for wastewater and water subprojects. All 35 households affected by the project were either compensated or were being compensated in 2017. 12. Tranche 2. Tranche 2 covered 14 ULBs and the ADB’s Board approved it in January 2009 and became effective in April 2009. The project closed in November 2017 after four extensions,18 more than 3 years after the target date of June 2014. At appraisal, total project cost was estimated at $219.0 million. ADB planned to finance $150.0 million from OCRs, and the state government, $69.0 million. At completion, actual project costs were $184.2 million, $117.3 million from ADB loan and $66.9 million, from the state government. Consulting services were planned to cost $9.0 million and this was eventually disbursed. 13. ADB approved 20 SRPs during project implementation. The project acquired 0.33 ha of land belonging to two households and 565 households suffered temporary livelihood loss. The PCR documented payment of compensation of $9.6 million to 512 households, while 44 households were not present during the execution of works and 11 households were not interested in receiving compensation, despite being offered. 14. Tranche 3. Tranche 3 covered 15 ULBs and the ADB’s Board approved it in December 2010. The agreement was signed in March 2011 and became effective in June 2011. The project closed in November 2017 after three extensions, more than 2 years after the target date of June 2015. The PCR stated that extensions were granted to: (i) use the additional funds available as a result of the US dollar appreciation in additional subprojects; and (ii) enable the completion of additional subprojects. The project 3 cost at appraisal was estimated at $96.0 million. ADB planned to finance $63.0 million from OCRs, and state government, $23.4 million. At completion, actual project cost $76.0 million, ADB loan was $52.6 million, and state government, $23.41 million. The PCR stated that cost differences arose from: (i) decreased financing charges because of lower interest rates, with $6.7 million savings; (ii) US dollar appreciation and overestimation of contingencies resulted in an amount of $11.0 million canceled,19 and (iii) $2.4 million unutilized at loan closing. Both price and physical contingencies were estimated at $13.0 million at appraisal. Part of the originally allocated budget for contingencies was reallocated to the infrastructure and capacity development components and the rest was canceled or unused. Urban infrastructure costs amounted to $68.0 million, compared to the $62.9 million at appraisal. Capacity development and implementation support costs were $4.4 million, slightly higher than the original $2.9 million estimated at appraisal. ADB’s total disbursements amounted to $52.6 million,

of bid documents for the identified subprojects; and (v) support to the executing agency in updating the resettlement plans and environmental assessment documents for the sample subprojects.

18 The project extended twice by 1 year each, followed by another two extensions of 6 months each. 19 Partial loan cancellations of $3.0 million on 3 December 2015 and $5.0 million on 27 June 2016 were approved,

while the unutilized amount of $2.4 million was canceled at loan closure on 24 November 2017.

6

equivalent to 83% of the $63 million originally committed, or 95% of the revised loan amount of $55 million. The initial disbursement was in August 2011 and the final disbursement in November 2017. 15. ADB approved six sample specific sector-specific, town-specific initial environmental examinations and SRPs were prepared at approval in each project. During implementation a total of 55 initial environmental examinations and 45 SRPs were prepared for subprojects in the 15 ULBs. A total of ₹3.41 million was paid as land compensation to two households, and ₹1.15 million as compensation for livelihood and crop loss to 197 households. D. Implementation Arrangements 16. The implementation arrangements for the MFF and its tranches were carried out as planned. The state government’s Local Self Government Department was the executing agency while the department’s Investment Program Management Unit (IPMU) was the implementing agency, headed by a project director. An interministerial state-level committee, chaired by the minister of urban development and local self-governance, with the project director as member secretary, provided policy guidance and coordination.20 A work finalization committee, headed by the principal secretary of urban governance, reviewed and endorsed subproject appraisal reports prior to submission to ADB. Investment program implementation units (IPIUs) were set up in each project town, concerned agencies, and departments. City-level committees,21 chaired by district collectors, provided guidance to IPIUs on actions to improve project performance. The project assets, once developed, were handed over to line agencies and ULBs for operations and maintenance (O&M), including the management of extant O&M contracts. The Public Health Engineering Department managed water supply assets, while the Public Works Department oversaw major roads and transport assets. ULBs managed drainage, internal municipal roads and streetlights, SWM, and wastewater. 17. An investment program management consultant firm assisted the IPMU and IPIUs, and three design and supervision consultant firms prepared design documents, managed contract tendering, and supervised construction works. A community action and participation program consultant firm and an investment program performance and management system consultant firm took care of training and monitoring. 18. According to the PCR, all of the assurances and covenants for the MFF and its tranches were complied with except for two that were partially complied with. The covenant on the levy of sustainable user tariffs was partially complied with. Enhanced water and sewerage tariffs, the infrastructure development tax, or the urban development tax were levied. The PCR did not provide a full picture on the financial sustainability of solid waste collection and SWM treatment, and its impact on ULBs operating sustainability. The covenant on audited project financial statements was also partly complied with. Annual audits were conducted but submitted late for 3 fiscal years. At the same time, separate accounts were not created for each project to be reconciled with ADB’s loan financial information services.

20 Secretaries of art and culture, financing, infrastructure, local self-governance, planning, tourism, and urban

governance; and the Public Health Engineering Department were members. 21 The committees also functioned as grievance redress committees.

7

II. EVALUATION OF PERFORMANCE AND RATINGS A. Relevance of Design and Formulation 19. The PCR rated the MFF and project 3 relevant. The rating was based on the project’s alignment with government plans and development objectives, and with ADB’s country and sector strategies. At appraisal, the MFF and project 3 were aligned with ADB’s country strategy and program for 2003–2006, its update for 2006–2009, and the partnership strategy for 2009–2012,22 which prioritized urban infrastructure and the support of environmentally inclusive growth. The facility and project were also consistent with India’s and Rajasthan’s Eleventh Five-Year Plans (2007–2012) on augmenting economic and social infrastructure and providing improved municipal services to a larger number of people in urban areas.23 At completion, the MFF and project 3 were aligned with government’s plans that prioritized the supply of water to urban population, wastewater management, and urban poverty alleviation.24 Both were also aligned with ADB’s country partnership strategy (2018–2020) 25 in the areas of inclusive growth, infrastructure, municipal services for the urban poor, and environmental sustainability. Finally, the facility and project 3 were consistent with respect to ADB’s water for all policy (2001) and the water operational plan (2011–2020) in promoting access to services and developing sustainable mechanisms to deliver social services, particularly in the water sector.26 20. The PCR further considered the whole facility’s design as an MFF modality adequate since it provided the flexibility to sequence investments and facilitated an effective long-term approach to existing large-scale development challenges in the urban sector. However, the MFF and project 3 continued with the design flaws, such as it failed to incorporate a continuous (24/7) water supply or the volume of incremental wastewater collected and treated as a project outcome, which was mentioned in PCR validation of the previous two tranches. Mechanisms to encourage household sewerage connections were not appropriately considered at project design either. The PCR stated that no changes in scope occurred during the project 3 implementation. This validation notes that a minor change in scope should have been processed for project 3 in view of a significant change from the original output indicator target related to urban transport. 27 Furthermore, project 3 included investments in firefighting services (stations, trucks, and equipment) that were not originally proposed under the MFF. Insufficient justification was provided for the needs and rationale to include these investments under the project.28 21. The PCR also emphasized design weaknesses in the DMF, and the inconsistencies noted between the performance indicators across the facility and tranche DMFs.29 Firefighting outputs, for example, were not proposed under the MFF’s DMF. Several performance targets for indicators did not clearly specify quantitative targets and sources of validation, and baseline data was

22 ADB. 2003. Country Strategy and Program: India, 2003–2006. Manila; ADB. 2006. Country Strategy and Program

Update: India, 2006–2008. Manila; and ADB. 2009. Country Strategy and Program: India, 2009–2012. Manila. 23 Government of India, Planning Commission. 2008. Eleventh Five-Year Plan, 2007–12. New Delhi; and Government

of Rajasthan, Planning Department. 2008. Eleventh Five-Year Plan, 2007–12. Jaipur. 24 Government of India, NITI Aayog. 2017. India Three Year Action Agenda, 2017–18 to 2019–20. New Delhi; and

Government of Rajasthan, Planning Department. 2012. Twelfth Five-Year Plan, 2012–17. Jaipur. 25 ADB. 2017. Country Partnership Strategy: India, 2018–2022. Manila. 26 ADB. 2001. Water for All: The Water Policy of the Asian Development Bank. Manila; and ADB. 2011. Water

Operational Plan, 2011–2020. Manila. 27 Urban transport investment went from $1.9 million to $14.0 million at completion. The originally proposed

improvement of 3.97 kilometers (km) of existing roads went up to 23.1 km, extending the interventions from the original ULB of Dolphur to Barmer, Nagaur, and Rajsamand.

28 ADB. 2019. ADB’s Multitranche Financing Facility, 2005–2018: Performance and Results Delivered. Manila. 29 Project 1 did not even have a DMF.

8

unavailable. These DMFs’ weaknesses limited the ability to measure project’s achievements and properly assess the sector needs and justification for project’s interventions. With the similar analysis as the previous tranches, this validation assesses the MFF and project 3 relevant. There were some weaknesses in design. However, they were aligned with government and ADB development policy, the entire MFF focus, and strong purpose for the state. B. Effectiveness in Achieving Project Outcomes and Outputs 22. The PCR rated the MFF and project 3 effective. The MFF comprised 13 indicators associated with two outcomes that covered access to sustainable urban infrastructure and services, and capacity to sustainably manage urban services. According to the PCR, 12 of the 13 outcome targets were fully achieved and 1 was partially achieved. This validation notes that 7 out of the 13 targets were achieved. First, the target for water supply interventions was achieved as 2.23 million people (95% of the population) were provided with treated piped water supply with an average 122 liters per capita per day (lpcd) supply (target: 1.4 million people with at least 100 lpcd). Increased piped water supply was provided. However, household connections were below 95% and there were less than 2 hours of water supply per day. Second, outcome achievement of wastewater management interventions were: (i) 1.92 million people in 14 ULBs (100% of the population) were provided with sanitation infrastructure (target: 1.6 million); however, only 61,946 households were reported to connect to the sewerage networks; (ii) sewerage facilities, including sewage treatment plants (STPs), provided in 14 ULBs (target: 12 ULBs); and (iii) wastewater discharge to water bodies reduced by 41 million liters per day (mld), however, this was well below the installed treatment capacity of 109 mld. Third, outcomes on SWM were partly achieved. Over 90% of solid waste generated was collected and transported. However, sanitary landfills were not operational. Fourth, outcome indicators proposed for urban drainage constitute outputs were: (i) developed outfall drains in 5 ULBs (initial target: all program’s 15 ULBs) and the PCR claimed that these were ULBs’ priorities based on flood risk assessments; (ii) roadside drains developed in 6 ULBs (initial target: not clear);30 (iii) a more appropriate outcome indicator would have been incidence of flooding, while the PCR claimed that incidence was reduced, no quantitative evidence was provided. Fifth, outcomes related to urban transport were achieved. The PCR reported that traffic flow improved, reducing travel costs and traffic congestions in 12 ULBs. However, no quantitative data was provided to support this statement. Sixth, the number of people living in slums benefiting from improved basic urban services met the target 0.21 million. Seventh, outcome indicators proposed to measure achievements in institutional capacity were generically defined with no specific targets. According to the PCR, they were achieved, however, this validation notes that these were partially achieved at best based on the evidence presented: (i) existing evidence suggests that the capacity of program ULBs meets O&M costs of urban services and have sound financial management practices varied across ULBs;31 (ii) not all ULBs have adequate resources from their own sources, such as local taxes and levies to meet all of their expenses;32 and (iii) little evidence is provided to support the indicator of ULBs’ human resource capacity to maintain urban facilities and services.33

30 The DMF indicator is: “all core areas of investment program ULBs provided with roadside drains.” 31 As noted in validation reports for project 1 and 2. 32 There is still reliance on transfers from other government bodies, such as the central and state finance commissions. 33 The PCR only mentioned about trainings and the preparation of capacity development plans, but no information yet

was presented on the implementation.

9

Table 1: Summary of Achievement of Multitranche Financing Facility’s Outcome Indicators

Sub-sector/component

Achievement

Detail

1. Water supply 1 achieved (1) 2.23 million people (95% of the population) were provided with treated piped water supply with an average 122 lpcd supply (target: 1.4 million people with at least 100 lpcd). It must be noted that household connections were below 95% and there were less than 2 hours of water supply per day.

2. Wastewater management

3 achieved (2) 1.92 million people in 14 ULBs (100% of the population) were provided with sanitation infrastructure (target: 1.6 million); yet only 61,946 households were reported to connect to the sewerage networks.

(3) Sewerage facilities, including STPs, provided in 14 ULBs (target: 12 ULBs).

(4) Wastewater discharge to water bodies reduced by 41 mld. It must be noted that this is well below the installed treatment capacity of 109 mld.

3. Solid waste management

1 partly achieved (5) Over 90% of solid waste generated was collected and transported. Sanitary landfills were not operational.

4. Urban drainage 2 achieved (6) Outfall drains were developed in 5 ULBs (initial target: all 15 program ULBs. The PCR claimed that these were the ULBs priorities based on flood risk assessments).

(7) Roadside drains developed in 6 ULBs (initial target was not clear).

5. Urban transport and roads

1 achieved (8) Based on the PCR, traffic flow improved, reducing travel costs and traffic congestions in 12 ULBs, however, no quantitative data was provided to support this statement.

6. Social infrastructure

1 achieved (9) The number of people living in slums benefiting from improved basic urban services was 0.21 million (target: 0.21 million).

7. Improved institutional capacity

1 achieved (based on the PCR), 2 partially achieved

(10) The validation assesses this as partly achieved as existing evidence suggests that the capacity of program ULBs to meet O&M costs of urban services and have sound financial management practices varied and challenging, except in a few ULBs.

(11) The PCR indicated that ULBs have adequate revenues from their own sources and compensations from the central and state governments for abolished and subsumed taxes. The validation views that this is partly achieved given that not all ULBs have adequate resources from their own sources ,such as local taxes and levies to meet all of their expenses.

lpcd = liter per capita per day, mld = millions per day, O&M = operation and maintenance, PCR = project completion report, STP = sewage treatment plant, ULB = urban local body. Source: Independent Evaluation Department. 23. The PCR assessed that 25 outputs were achieved, 1 partially achieved, and 1 not achieved. Physical outputs were largely achieved, although there were some variations in outputs due to lack of quantification of the baseline or adjustments after detailed designs of the

10

subprojects were completed.34 The output not achieved was the construction of the sanitary landfills as the state government decided to withdraw the works under the MFF. The most relevant physical output achievements were: (i) laying of a total of 1,704 kilometers (km) of water distribution pipes; (ii) installation of 135,862 household water meters and 172 bulk water flow meters; (iii) rehabilitation of two water treatment plants and construction of 126 reservoirs; (iv) construction of 920 km of sewer network and 61,946 connections; (v) wastewater treatment capacity expanded by 109 mld; (vi) 41.8 km outfalls and stormwater drains constructed, and 27.7 km of roadside drains constructed; and (vii) over 98.22 km of roads constructed or strengthened, 2 lane roads over bridges, 2 major bridges, 2 minor bridges, 1 railway under bridge were constructed. 24. Project 3 was associated with one outcome—improved urban services in the 15 ULBs—and had 5 outcome indicators. This validation assesses that 2 outcome indicators were achieved, 2 partly achieved, and one was not achieved. This validation notes, however, that several DMF’s outcome performance indicators were not appropriate to assess project achievements. On the outcome indicators related to water supply interventions in Baran and Chittorgarh: (i) while drinking water quality supposedly met prescribed quality standards of India, 35 the average duration increased from 1 hour to 1.5 hours in Chittorgarh and stayed at 1 hour in Baran36 (target: 8 hours). Although not included in the DMF, water consumption rose from 60 net lpcd in 2008 to 100 net lpcd in 2017 in Baran, and from 65 net lpcd to net 100 net lpcd in Chittorgarh; (ii) nonrevenue water was reduced to meet the target of 20%. On sewerage and sanitation interventions, (iii) the total number of households connected to the network as reported by the investment program performance and management system (IPPMS) consultants’ end report was 22,096 or 14.7% of the total population in the selected ULBs. Also, while treatment capacity was expected to increase by 35 mld, only two STPs for a capacity of 17 mld were functional in 2018, and capacity utilization was below 50%. Based on the presented information, this validation notes that sewerage outcomes were partially achieved. On the outcome indicators for SWM, (iv) landfill sites were not available for proper disposal by the end of the project; and (v) almost 100% of households were being covered by door-to-door collection, including slum/low-income or women-headed households (targets: 50% and 25% respectively).

Table 2: Summary of Achievement of Tranche 3’s Outcome Indicators Sub-sector/component Achievement Detail Urban services 2 achieved, 2 partly

achieved; and 1 not achieved.

(1) Drinking water quality supposedly met India’s prescribed quality standards. The target increase in average duration of water supply to at least 8 hours was not achieved. Average duration of supply increased from 1 hour to 1.5 hours in Chittorgarh and stayed at 1 hour in Baran (partly achieved)

(2) Nonrevenue water was reduced to meet the target of 20% (outcome achieved)

(3) The validation notes that sewerage outcomes were only partially achieved due to the (a) total population targets were not met in some ULBs, (b) only two STPs were functional in 2018, and (c) capacity utilization was below 50%.

34 For example, wastewater treatment capacity and the length of roads improved were adjusted. 35 The report and recommendation of the President (RRP) and the PCR did not clearly specify nor provide additional

information on the quality standards. 36 Extracted from Annex 4 of IPPMS consultant’s end-term report.

11

(4) Landfill sites did not materialize (outcome not achieved)

(5) The PCR noted that almost 100% of households being covered by door-to-door collection, including slum/low-income or women-headed households (target was 50% and 25% respectively) (outcome achieved)

PCR = project completion report, STP = sewerage treatment plant, ULB = urban local body. Source: Independent Evaluation Department. 25. According to the PCR, 28 of 31 output targets were achieved. In terms of water supply outputs, 7 were achieved and 1 was not achieved, which was related to the development of a road map for 24/7 water supply. Sewerage outputs had 6 indicators achieved and 1 partially achieved, with 18,018 sewers connected against the target of 37,600. Reduced flooding through improved urban drainage, and urban transport and the improvement of the traffic flow achieved 3 output indicators each. SWM and the increase in solid waste collection and disposal achieved 3 output indicators and 1 not achieved, as sanitary landfill sites were not functional.37 Firefighting strengthening had 4 output indicators achieved. Heritage site conservation output indicator was achieved. Finally, the 5 output indicators related to implementing agency’s’ effective management of project-financed assets were considered achieved. However, this validation notes that most of the indicators do not properly reflect effective management, but rather capacity-building activities. The indicators were not described by the outcomes but counting inputs or activities. 26. Environmental assessments were undertaken and ADB approved reports for 101 subprojects under the MFF, of which 36 subprojects were approved under project 3. The PCR mentioned that requisite environmental permissions as per India’s environmental were obtained from the respective regulatory agencies. Under project 3, some environmentally sensitive areas were overlooked during appraisal and were addressed during implementation.38 The objectives of the environmental management plans were achieved through training and supervisions of contractors. Implementation of these plans was satisfactory. During project implementation, environmental and social monitoring reports were prepared and regularly disclosed, including to ADB, with a focus on procedural compliance rather than on implementation practices. Public consultations were carried out regularly to disseminate project benefits, identify issues, and address problems encountered during construction. Grievance redress committees were established at the city level, and routine grievances were mainly about removing debris, backfilling open pits, restoring roads and footpaths, and choking drains. However, no major environmental issues were reported during implementation of the subprojects. The continued renewal of the consent to operate after loan closure for some of the sewage and water treatment plants developed under the project was not being obtained from the state pollution control board at project completion. The PCR suggested that this was a procedural issue and that the plants were reported to be performing to the desired standards. 27. ADB approved 45 resettlement plans during MFF implementation (15 for project 3), of which 33 were implemented (11 for project 3) and the remaining 12 were not required as anticipated impacts were avoided, or subprojects were dropped. The implemented plans were compliant with both ADB’s policy and government legislation. Land acquisition was not involved in project 1, while 0.33 ha of land was acquired in project 2. Under the MFF, four affected

37 Due to Rajasthan government’s decision to withdraw landfill works under the MFF and handover to Rajasthan Awas

Vikas Nigam. 38 The intake works of the water supply project in Dholpur were located within the boundaries of National Chambal

Sanctuary after a complaint by a local conservation agency. The IEE reports were updated accordingly.

12

households were compensated with ₹7.15 million for 0.9 ha of land, and 719 affected households were compensated ₹7.03 million for temporary loss of livelihood and shifting assistance.39 This included 0.57 ha land acquired from two households for construction of approach road to the road under bridge subproject at Chittorgarh, and 205 households that suffered temporary livelihood loss during project 3 implementation. The project ULBs’ extensive consultations with affected persons and support enabled the successful implementation and completion of the SRPs. The external monitoring report confirmed that all the affected households were appropriately and timely compensated, without any significant temporary loss of livelihood. 28. Indigenous peoples were not impacted during the MFF implementation, including project 3. No gender action plan was prepared for projects 1 and 2, while project 3 had a gender action plan and results monitoring framework. Appendix 8 of the PCR documented the activities and actions planned and the expected results that were deemed as achieved. The key results were summarized as follows: (i) identified priority households for inclusion; (ii) reduced burden and time for women in managing household water requirements; (iii) reduced safety risks for women; (iv) reduced expenses on household health care; (v) resource allocated for implementation of gender activities; (vi) enhanced awareness and participation in decision making; (vii) conducted human capital development and capacity building on gender issues for implementation agencies. 29. On the whole, while there was progress in achieving MFF and project 3’s objectives, there was limited evidence available to support that outcomes were sufficiently available to the intended beneficiaries, particularly for sewerage, piped water supply, SWM, and institutional development of the ULBs. Other areas, such as urban transport, did show achievements. Substantial works and other outputs were completed, particularly infrastructure assets. However, it was unclear if expected outcomes were delivered to the beneficiaries. Performance indicators definitions and targets made it also difficult to assess effectiveness. For these reasons, this validation assesses both the MFF and project 3 less than effective.40 C. Efficiency of Resource Use 30. The PCR rated project 3 efficient. The economic internal rates of return (EIRRs) of the subprojects identified at appraisal were reevaluated at completion using the same approach as at appraisal.41 These were the results: (i) water supply—Chittorgarh, 29.6% (appraisal 17.8%); (ii) wastewater—Bharatpur, 15.9% (appraisal 13.2%); (iii) urban drainage—Sawai Madhopur, 14.5% (appraisal 16.4%); and (iv) urban transport—Sawai Madhopur, 14.2% (appraisal 14.0%).

39 IPMU records indicated payments of ₹3.41 million as land compensation to two households and ₹1.15 million as

compensation for livelihood and crop loss to 197 households. Six households were not present during the execution of works. and two households were not interested in receiving compensation.

40 IED and South Asian Department (SARD) have differing views on the outcome and outputs to be used on assessing effectiveness. SARD considered project changes that were recorded in the project aide-mémoires, and the records of annual tripartite project review meetings among ADB, the Government of India, and project executing agencies, which are treated as official documents between ADB and the government. IED is of the view that all operational departments should follow the same set of procedures on change of scope, as the ADB Board approved the RRP’s original scope.

41 For calculating population growth, the compounded annual growth rate of the population between 2001 and 2011 was used to project the population until 2046. The economic viability of the sector is evaluated over a period of 25 years for water supply, 30 years for urban drainage and urban transport, and 35 years for wastewater, with no salvage value assumed thereafter. For economic analysis, the financial cost was converted into economic costs by applying prescribed conversion rates, adjusting for contingencies, taxes, and duties, but excluding the financing costs (interest during construction and commitment charges). The local benefits and costs were converted at an average exchange rate of Rs69 = $1.

13

The PCR provided further justification on the differences among estimates from appraisal to completion, most of the deviations stemmed from the use of actual numbers.42 31. In total, the PCR evaluated 18 subprojects at completion and the EIRRs ranged from 12.6% to 39.4%, all being above the 12% threshold. The estimated EIRRs of the project components at completion were (i) water supply,43 26.4%; (ii) wastewater and urban drainage,44 14.3%; and (iii) urban transport, 16.7%.45 The estimated EIRR for the total project combining all subprojects was 18.3%. Sensitivity analysis was conducted for a 20% reduction in benefits showing that EIRR remained above the 12% threshold.46 The EIRR was estimated at appraisal. However, it was not assessed for the Bundi SWM subproject at completion as the sanitary landfill was dropped. The EIRRs for firefighting and heritage subprojects were not estimated at appraisal and not assessed at completion. 32. The PCR rated the MFF efficient, with EIRRs of the major components for all tranches estimated at 24.5% for water supply, 15.8% for wastewater and drainage, and 17.6% for urban transport. The estimated EIRR of the facility was 19.6%. 33. The adopted approach to estimate the EIRR had these limitations for water supply and sanitation projects: (i) operations and maintenance costs were mostly constant over the project’s lifetime and did not vary with the increasing customer connections; and (ii) no replacement costs were considered. Loan extensions were approved to accommodate additional subprojects and complete the implementation of delays in executing some contracts. Program costs were substantially lower than estimated at appraisal due to overestimated contingencies, lower interest rates, and depreciated local currency. This validation assesses project 3 and the MFF efficient as the estimated EIRR is above 12%. It is noted, however, that there were limitations in the approach that biased the estimate upwards. Process efficiency was also mixed, intended outcomes achieved were below planned, costs were overestimated, and substantial implementation delays occurred both for project 3 and the MFF. D. Preliminary Assessment of Sustainability 34. The PCR rated the MFF and project 3 likely sustainable. The projects’ financial viability evaluation was considered for the entire MFF and financial sustainability analysis for ULBs under project 3 were carried out during loan appraisal. The weighted average cost of capital for project 3 was computed as 3.17%. The financial internal rate of return (FIRR) was calculated for water supply, sewerage, and SWM projects at four towns.47 FIRR estimates were positive for water supply projects ranging from 13.4% to 23.0%. Sewerage projects had returns below the weighted

42 EIRR at completion for water supply in Chittorgarh was higher than appraisal estimates as beneficiaries’ welfare

increased significantly beyond appraisal estimates and considered health benefits during reevaluation. In case of wastewater components, EIRR was higher than appraisal estimates at Bharatpur, although no explanation was provided by the PCR. For urban drainage, EIRR was lower than appraisal estimates as benefits covered a smaller part of the population. EIRR for urban transport in Sawai Madhopur was higher than appraisal estimates as the number of vehicles and people using the bridges marginally increased during the period between appraisal and completion.

43 Benefits were estimated based on non-incremental resource cost savings and health costs avoided. 44 Benefits were estimated based on health cost savings and avoidance of flood losses. 45 Benefits were estimated based on avoided income loss due to time savings. 46 At the subproject level, some projects displayed an EIRR below the 12%. 47 Estimates were as follows: (i) 13.4% for water supply and –4.7% for sewerage in Chittorgarh; (ii) 23.0% for water

supply and 2.6% for sewerage in Bharatpur; and (iii) 14.9% and 18.3% for solid waste management in Bundi and Sawai Madhopur, respectively. The evaluation also considered sensitivity analysis under situations of (i) capital cost, +10%; (ii) O&M cost, +10%; (iii) benefit, –10%; and (iv) the worst case.

14

average cost of capital. SWM projects had FIRRs above 14%. The analysis concluded that O&M costs for water supply and sewerage projects would be fully covered with the proposed tariffs. At completion, no FIRR was calculated for revenue-generating projects. The PCR presented the budgets and financial statements for 5 out of the 14 ULBs for the period 2014–2017. The data revealed that the operating ratio had been below 1.0. However, ULBs remained dependent on transfers and grants from the state government to meet their expenditures. 48 Caution was warranted as to the financial capacity to cover O&M expenditures of the assets developed through the project. There was a heavy reliance on transfers and, although tariffs were officially increased, evidence on increased revenue collection was mixed. Sampled ULBs seemed to have the capacity to consistently cover operating expenses in the past years. 35. Regular public consultations were carried out to disseminate project benefits, identify issues, and address problems encountered by the public during construction. However, no evidence was provided to show that these activities led to a sense of ownership of project investments with benefits for sustainability. Similarly, no evidence was provided of the project’s effects on biodiversity, greenhouse gas emissions, natural resource management, and pollution. 36. From an institutional perspective, while training was provided to build capacity for sustainable management of the assets under the MFF, little evidence was provided on the existing human resource capacity to fulfill these objectives, particularly at ULB level. The PCR stated that MFF supported ULBs in introducing improved financial management practices. However, limited evidence was presented about the financial and institutional capacity of the ULBs to manage the infrastructure assets and financially cover at least the daily running cost. This validation assesses the MFF and project 3 less than likely sustainable.49

III. OTHER PERFORMANCE ASSESSMENTS A. Preliminary Assessment of Development Impact 37. The PCR rated the development impact of the MFF and project 3 satisfactory. The PCR provided information on the achievement of MFF’s impact performance targets, although most of the DMF’s impacts could be directly attributed to the program interventions, and several indicators were not measured at ULB level. First, the increase in real income per capita in program’s ULBs could not be directly measured (target increase: 10%). The PCR provided an estimated increase of 206% in per capita real income at the state level from 2007 to 2016. Second, the target of 20% more of the population in program’s ULBs reached by urban infrastructure services was considered achieved according to the PCR as improved water supply supposedly served 30% more population. Third, tourists increased by an average of 31% between 2007 and 2018 in Rajasthan. However, it was unclear what the increase was at program’s ULBs (target increase: 10%). Fourth, the reduction in sanitation-related diseases was estimated at 38% when compared

48 Mainly from compensation grants and transfers from the state and central finance commissions. 49 On financial sustainability of the ULBs, SARD acknowledged (i) a risk of not achieving the intended tariff increase to

full cost recovery of operation and maintenance at risks section of the RRP, and (ii) the government’s commitment to make the project sustainable through other means at the “assurances” indicated in the MFF RRP, tranche 2 PFR report, and tranche 3 PFR, which stated that “(ii) The government of Rajasthan will ensure adequate funds toward the operation and maintenance of the facilities created under the Facility through budgetary allocations or other means, to be provided to the Executing Agency, the appropriate ULB, or line agencies during and after subproject completion.” Thus, SARD considered the project sustainable. However, IED is of the view that the validation has to abide by what was the original intent of the project design. IED does not treat assurances to override the project construct on financial recovery.

15

against 2008 (target: 20%).50 Fifth, a 21% increase in the population living below the poverty line (BPL) had access to improved urban infrastructure services (375,547) in the program ULBs (target: 20%).51 Sixth, the PCR stated that 13 out of 15 ULBs had shown a declining trend on the percentage BPL population (target: either constant or declining trend).52 38. The PCR provided information on the achievement of project 3’s impact performance targets. First, the target of 60% increase of beneficiary population was partially achieved as the actual increase was 42%.53 Second, the reduction in waterborne diseases was estimated at 26%, exceeding the 15% target. Given all the factors that affect waterborne diseases and the nature of the program’s interventions, this result could not be directly be attributable. 39. The PCR acknowledged other possible impacts from the MFF and project 3’s interventions, such as: (i) the decrease in medical treatment costs and productivity losses; (ii) reduced flooding in low-lying areas; (iii) reduced traffic congestions, travel time, vehicle operation costs, and increase road safety; and (iv) improved environmental quality. No quantitative evidence was provided to assess the level of impact. 40. The PCR also highlighted the impacts on institutional capacity. The IPMU has learned from implementing the first ADB loan for the Rajasthan Urban Infrastructure Development Project and the first tranche of this MFF. The IPMU has significantly strengthened its capabilities to plan and prepare the feasibility documents of subprojects, procurement, contract management, safeguards, and monitoring. The IPMU is now a part of the state-level nodal agency for implementing urban infrastructure projects and the leading organization of the state in managing large-scale, foreign-assisted, and government-funded projects. However, the capacities of ULBs remained weak because of a lack of qualified staff and inadequate revenues. Reforms were also introduced for the corporatization of water supply and wastewater. The state-level institution, Rajasthan Urban Drinking Water Sewerage Infrastructure Corporation Limited, was created to build capacity of municipal staff and ensure a coherent approach to urban service provision. 41. Based on all of these aspects, this validation assesses development impact of the MFF and project 3 satisfactory. The impacts listed above are positive improvements for the project towns. However, there were shortcomings in the outcome achievements and difficulty stemming from the lack of data and monitoring reports to provide substantive trace of the whole facility. B. Performance of the Borrower and Executing Agency 42. The PCR rated the performance of the borrower and the executing agency under the MFF and project 3 satisfactory. The borrower, represented by the Government of India’s Department of Economic Affairs, provided timely guidance and decisions to the Rajasthan government of Rajasthan on the project and undertook regular tripartite review meetings with ADB, the state government, and the IPMU. This helped ensure adequate project preparation, implementation, and sustainability. The PCR indicated that the state government provided strong support to the IPMU, including timely counterpart funding and adequate human resources throughout the project

50 The number was based on the number of households that reported incidences of water-borne diseases based on

the IPPMS consultants-conducted survey in the 15 ULBs. 51 The PCR did not clarify how this indicator was measured and referred to IPPMS consultant’s end-term report as a

source of information. 52 The PCR mentioned that BPL population data was only available at state level and that IPMU collected data for 13

ULBs. 53 It was unclear how beneficiary population was defined. The source for the information was IPMU data.

16

period.54 The PCR also noted that the IPMU established implementation procedures for planning and implementation early in the project cycle. The IPIUs also established effective monitoring and implementing mechanisms in each project ULB. The PCR identified some weaknesses, such as: (i) financial management had issues with auditing due to the IPMU’s failure to create separate project accounts; and (ii) periods for some water supply and sewerage contracts were underestimated, leading to contractors exceeding implementation milestones. This validation assesses the performance of the borrower and the executing agency satisfactory for project 3 and MFF, although some shortcomings in terms of clear and detailed monitoring and associated date between the planned and achieved results were missing. The organization capacity enhancement activities’ results were not clearly checked and assessed with traceable evidence on how much achievement were realized in terms of: (i) double-entry accounting system; (ii) computerized financial management systems; and (iii) asset management systems improved in the targeted ULBs. This is partly because the baseline was never clearly set. C. Performance of the Asian Development Bank and Cofinanciers 43. The PCR rated the performance of ADB under project 3 and the MFF satisfactory. ADB undertook regular review missions and a disbursement review mission. It also prepared a midterm review to assess progress and provide advice on outstanding issues. ADB responded promptly when changes of scope were required and provided effective support to the IPMU in resolving project management issues. ADB also provided training and supported exposure visits of IPMU and IPIU personnel to other ADB-funded projects in India and abroad. ADB monitoring ensured adherence to due processes and transparency in procurement, disbursements, and safeguards. The PCR noted that the audit shortcomings were identified, but not resolved. This validation also notes that ADB could have improved its work on the project’s DMF design, the estimation of project cost contingencies, and the estimation of project implementation timing. This validation is of the same view with the rating of satisfactory for the project 3 and MFF. D. Others 44. The PCR provided information on project procurement. The IPMU was familiar with ADB procedures for the procurement of civil works and goods and conformed to ADB’s procurement guidelines (2007). Standard bid documents prepared under project 1 for the procurement of civil works and goods were used in the MFF that led to standardization and quality control. Procurement activities were executed smoothly and were among the IPMU’s core strengths. The performance of the consultants was rated generally satisfactory. Some issues with the performance of design and construction supervision personnel were slow internal mobilization and frequent replacement of team members. The IPMU’s close monitoring of design and construction supervision consultants led to the consultants’ improved output deliveries. The performance of the initial IPPMS consulting firm engaged to prepare project performance monitoring reports was less than satisfactory and had to be terminated. Following the termination, a new firm was appointed, and its performance was satisfactory. On the whole, the performances of the contractors and supplier were satisfactory. Some deficiencies were noted in the designs of 17 water supply and sewerage contracts (of the total 132 contracts) as contract periods were not realistic and had to be extended to also accommodate contractor’s poor mobilization of resources. This validation has similar assessment as the ratings from the PCR.

54 The PCR stated that the Rajasthan government provided counterpart funding to complete spillover works after loan

closure.

17

IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings

45. Overall, the PCR rated the MFF and project 3 successful. This validation assesses them less than successful. Under the core criteria, the project was relevant, less than effective, efficient, and less than likely sustainable. Significant progress was made to improve urban services to the project towns. However, there were shortfalls in the expected outcomes. The MFF and project 3 were relevant to the government’s development objectives and ADB’s priorities, although some design flaws were noted. They were less than effective due to shortfalls in the documentation and achievements of outcomes related to water supply, sanitation, SWM, and institutional development of ULBs. The project was efficient. Recomputed EIRRs were above 12%; however, methodological shortcomings may have biased the estimation upwards. Finally, the lack of financial and institutional capacity at ULBs to perform O&M activities suggested that the project was less than likely sustainable.55

Overall Ratings

Validation Criteria PCR IED Review Reason for Disagreement

and/or Comments Relevance L2366 L2506 L2725 MFF0015

Relevant Relevant Relevant Relevant

Relevant Relevant Relevant Relevant

Effectiveness L2366 L2506 L2725 MFF0015

Effective Effective Effective Effective

Less than effective Less than effective Less than effective Less than effective

Shortfall in the achievement of outcomes (water supply, sewerage, solid waste management, and institutional development of ULBs)

Efficiency L2366 L2506 L2725 MFF0015

Efficient Efficient Efficient Efficient

Efficient Efficient Efficient Efficient

Sustainability L2366 L2506 L2725 MFF0015

Likely

sustainable Likely

sustainable Likely

sustainable Likely

sustainable

Less than likely

sustainable Less than likely

sustainable Less than likely

sustainable Less than likely

sustainable

The financial sustainability of the ULBs with existing revenue collection levels is not sufficient to meet O&M costs of assets. There is a heavy dependence of transfers from state and government. Management of several assets is still performed by PHED rather than ULBs.

Overall Assessment

55 SARD disagrees with this validation's assessment of effectiveness, sustainability, and overall assessment for the

reasons mentioned in the footnotes 40 and 49 (see Appendix).

18

Validation Criteria PCR IED Review Reason for Disagreement

and/or Comments L2366 L2506 L2725 MFF0015

Successful Successful Successful Successful

Less than successful Less than successful Less than successful Less than successful

Preliminary Assessment of Impact L2366 L2506 L2725 MFF0015

Satisfactory Satisfactory Satisfactory Satisfactory

Less than satisfactory Satisfactory Satisfactory Satisfactory

It is difficult to fully attribute identified impacts to project outcomes

Borrower and executing agency L2366 L2506 L2725 MFF0015

Satisfactory Satisfactory Satisfactory Satisfactory

Satisfactory Satisfactory Satisfactory Satisfactory

Performance of ADB L2366 L2506 L2725 MFF0015

Satisfactory Satisfactory Satisfactory Satisfactory

Satisfactory Satisfactory Satisfactory Satisfactory

Quality of PCR

Less than satisfactory Para. 54. Inconsistent information on outcome and output targets is noted in the main text and appendix 1. Aspects that could have been improved are (i) EIRR had shortcomings on the assumptions used for benefit and cost; (ii) no FIRR was calculated at completion; (iii) additional documentation of inputs used from TA; and (iv) further documentation on safeguard implementation, particularly on environmental impact. Most of the outcome and output performance indicators did not have a baseline, which made it difficult to quantify existing needs and understand the links to the level of subproject interventions.

ADB = Asian Development Bank, EIRR = economic internal rate of return, FIRR = financial internal rate of return, IED = Independent Evaluation Department, PCR = project completion report, PHED = Public Health and Engineering Department, TA = technical assistance, ULB = urban local body. Source: ADB (IED).

B. Lessons

46. The main lessons identified in the PCR can be summarized as follows: (i) project design could have been enhanced to include 24/7 water supply and synchronize the provision of sewer networks with household connections; (ii) sewerage outcomes in terms of number of households connected to the network was less than 15% in the selected ULBs; (iii) enhancing subproject

19

preparedness before contract awards improves implementation and contract design must also consider realistic implementation periods to avoid extensions and legal issues; (iv) close monitoring of consultants performance is necessary to ensure the quality and timely delivery of services; (iv) having a dedicate cadre of staff in the IPMU, rather than deputed staff, contributes to ensuring continuity in implementation; (v) overestimation of contingencies results in partial loan cancellation or the need to extend the loan implementation to utilize the contingency amounts; and (vi) DMF indicators must be consistent and uniform across the MFF and its projects to ensure a coherent performance assessment. 47. This validation builds on the key lessons in the PCR and suggests the following lessons: 48. Sector-level lesson. It is important to provide mechanisms to ensure connections to the sewerage network in the areas where the network is expanded. This will also help guarantee the utilization of the infrastructure, the generation of revenues to contribute to financial sustainability, and the realization of project’s economic benefits. For water supply, it is important to consider service reliability (hours of water supply per day) in addition to quantity and quality of water delivered, as it affects the benefits that households obtain from the services. In terms of SWM, the inclusion of sanitary landfill sites as part of the program interventions is critical to ensure that collected waste is adequately disposed and does not pose a risk to the population or the environment. 49. The ULBs’ effective management of infrastructure assets critically hinges on an adequate infrastructure financing strategy that ensures a self-sustainable service provision. Capacity development efforts for ULBs should also consider the promotion of operational efficiency (i.e., operating ratio and nonrevenue water reduction), and the capacity to collect revenues in relation to existing service level provision. 50. Results framework and methodology level lesson. It is important to include relevant indicators to assess project’s results and achievements objectively. This is particularly important to assess project effectiveness. Outcome and output indicators should not be mixed with each other. Baselines and performance targets should be clearly defined to understand project achievements. At the same time, project outcome indicators, such as the: (i) number of new household connections; (ii) the additional effective water consumption and service reliability; (iii) effective domestic wastewater treatment (rather than installed capacity); (iv) unaccounted for water; and (v) amount of solid waste disposed in sanitary landfills are critical to understand project achievements. Finally, it is important to be concise in selecting indicators to avoid unnecessary data collection efforts. The DMF at appraisal was too complex, containing too many indicators, thus, some of them provided limited additional value.

C. Recommendations for Follow-Up 51. Based on PCR’s recommendations, this validation suggests to monitor: (i) the evolution of household sewer connections and water consumption to understand the demand and utilization level of the infrastructure assets built in project towns; (ii) capacity of ULBs to manage infrastructure assets, ensuring adequate O&M and raising sufficient revenue to cover expenses; and obtaining the pertaining consents to operate for water treatment plants and sewage treatment plants.

V. OTHER CONSIDERATIONS AND FOLLOW-UP A. Monitoring and Reporting

20

52. The PCR reported that monitoring and reporting covenants were complied with. These included quarterly progress reports, semiannual safeguard reports, and the project completion report. The IPMU also conducted and reported baseline and end-term surveys. 53. The PCR emphasized inconsistencies between the MFF’s DMF indicators and the subsequent projects 2 and 3’s DMFs. This validation notes that the DMF had design weaknesses that limit its ability to measure project’s achievements. Most of the outcome and output performance indicators did not have a baseline that made it difficult to quantify existing needs and understand the links to the level of subproject interventions, such as infrastructure capacity. On another hand, the use of the term urban infrastructure services’ reach and access seemed to be interpreted inconsistently for water supply and sewerage services as it was not clear whether household connection was included in the definition. This limited the ability of the DMF to assess the realization of project benefits. Finally, other indicators could have been included to measure project’s impacts on the environment, such as pollution levels.

B. Comments on Project Completion Report Quality56 54. This validation notes the PCR’s quality less than satisfactory.57 The PCR recognized some of the DMF design’s shortcomings. However, this validation notes that the efforts to document indicator definitions and values were uneven. Inconsistent information on outcome and output targets was noted in several parts of the document (main text and appendix 1), particularly in terms of target values and years. Documentation of main data sources and methods for measurement should have been made more explicit. Other aspects that could have been improved include: (i) the methods to recalculate EIRR had some shortcomings on the assumptions used for benefit and cost estimations; (ii) further evidence could have been provided to support the sustainability rating, particularly the financial part; and (iii) additional documentation of consulting service inputs at project completion, or the inputs used from TA. C. Data Sources for Validation

55. Data sources for this validation were the PCR, the RRP, the government’s PCR,58 back-to-office reports, and aide-mémoires relating to loan and midterm review missions. India’s country partnership strategies for 2003–2006, 2009–2012, and 2018–2022 were also reviewed.

D. Recommendation for Independent Evaluation Department Follow-Up

56. A program performance evaluation report should be prepared so that the MFF is evaluated in its entirety against the impact, outcomes, and outputs expected at appraisal. Particular attention should be given to the facility’s effectiveness (increased water consumption, sewer connections, domestic wastewater treatment capacity utilization, quality of water supply service) and sustainability (O&M cost recovery); and development impacts, such as flood damages, household incomes, incidence of waterborne diseases, pollution, and travel time reduction.

56 IED noted that some of the differing views between IED and SARD stem from the lack of results data by component

and ULB breakdown. IED had to use IPPMS report data. SARD later noted that their evaluation on effectiveness was mainly based on the breakdown data collected during the PCR mission (that was not presented in the PCR, but kept by the project officer), not the IPPMS report data that was used by IED for the project validation report.

57 SARD disagreed with this validation's assessment of PCR quality. 58 Government of Rajasthan. Office of Project Director RUIDP. 2018. Project Completion Report for Rajasthan Urban

Sector Development Investment Program (MFF and Tranche 3). Jaipur.

21

APPENDIX: LINKED DOCUMENTS PCR Validation Report: Rajasthan Urban Sector Development Investment Program (Tranche 3 and Multitranche Financing Facility) a. SARD Comments on Final PVR

Document available upon request. b. IED Response to SARD Comments

https://www.adb.org/sites/default/files/linked-documents/Linked%20Document%20for%20Appendix%20-%20IED%20Response%20to%20PARD%20Comments%20on%20the%20Revised%20Draft%20PVR%20IND-2506%202725%2C%20MFF0015.pdf

https://www.adb.org/sites/default/files/linked-documents/Linked%20Document%20for%20Appendix%20-%20IED%20Response%20to%20PARD%20Comments%20on%20the%20Revised%20Draft%20PVR%20IND-2506%202725%2C%20MFF0015.pdfhttps://www.adb.org/sites/default/files/linked-documents/Linked%20Document%20for%20Appendix%20-%20IED%20Response%20to%20PARD%20Comments%20on%20the%20Revised%20Draft%20PVR%20IND-2506%202725%2C%20MFF0015.pdfhttps://www.adb.org/sites/default/files/linked-documents/Linked%20Document%20for%20Appendix%20-%20IED%20Response%20to%20PARD%20Comments%20on%20the%20Revised%20Draft%20PVR%20IND-2506%202725%2C%20MFF0015.pdfhttps://www.adb.org/sites/default/files/linked-documents/Linked%20Document%20for%20Appendix%20-%20IED%20Response%20to%20PARD%20Comments%20on%20the%20Revised%20Draft%20PVR%20IND-2506%202725%2C%20MFF0015.pdf

Project Basic DataI. Project DescriptionII. Evaluation of Performance and RatingsIII. Other Performance AssessmentsIV. Overall Assessment, Lessons, and RecommendationsV. Other Considerations and Follow-UpAppendix: Linked Documents