LINKED DOCUMENT 6: STAKEHOLDERS’ PERCEPTIONSLINKED DOCUMENT 6: STAKEHOLDERS’ PERCEPTIONS A. Survey Objective and Methodology 1. The evaluation of the Asian Development Bank’s

Appendix, Linked Document 6

1

LINKED DOCUMENT 6: STAKEHOLDERS’ PERCEPTIONS

A. Survey Objective and Methodology

1. The evaluation of the Asian Development Bank’s (ADB) results-based lending (RBL) pilot program

entailed a survey that aimed to gather feedback from government counterparts in relevant developing

member country (DMC) governments and ADB staff on the design and implementation of the RBL

modality, its added value, and the achievement of expected results and benefits. The survey results,

together with the analyses of data and information from other sources such as key informant interviews,

documents review, and portfolio analyses, were intended to contribute to the formulation of

recommendations to improve ongoing RBL programs and give directions for future portfolio.

2. The structured survey questionnaires were administered online1

to three groups of target

respondents with known email addresses: (i) officers and staff of national and central government

agencies in nine countries implementing RBL operations; (ii) officers and staff of executing and

implementing agencies of RBL programs; and (iii) ADB staff and RBL program officers or team leaders

(PTLs). The names of target respondents and their email addresses were mostly compiled from the

Independent Evaluation Department (IED) missions undertaken for this evaluation.

3. The survey was conducted from 16 August to 10 September 2017. A follow-up email was sent

to each target respondent.

B. Survey Response Rates and Respondents’ Profiles

1. Response rates

4. The survey questionnaire was sent to 21 central government officers and 49 executing or

implementing agency officers and staff (70 staff). A total of 40 government staff responded, a response

rate of 57% (Table A6.1). The survey questionnaire was sent to 26 ADB RBL program officers, of which

18 or 69% responded.

Table A6.1. Number of Stakeholders and Response Rates, by Type of Respondent

Stakeholders Sent to No. responded Response rate Margin of error a

Central government 21 9 43% 25%

Executing or

implementing agencies

49 31 63% 11%

Total DMC clients 70 40 57% 10%

PTLs 26 18 69% 13%

Total 96 58 60 8%

DMC = developing member country., PTLs=project or program team leaders

a At 95% confidence interval.

Source: Independent Evaluation Department.

2. Number of Respondents, by Country

5. Table A6.2 shows the distribution of survey respondents by country of RBL operations. Except for

Solomon Islands where the survey team was not able to discover e-mail addresses of DMC government

contacts, countries with ADB-supported RBL projects were represented by at least one respondent

government officer. The 18 PTLs cover all countries but one, the Philippines.

1 Survey Monkey online tool (www.surveymonkey.com).


2

Table A6.2. Number of Respondents, by Country of RBL Program

Central Government

Executing and

Implementing

Agencies

Total DMC

Clients PTLs

Armenia 1 2 3 3

China, People’s Republic 0 3 3 4

Indiaa - 1 1 3

Indonesia 1 1 2 3

Nepal 0 3 3 1

Pakistan 1 4 5 1

Philippines 1 5 6 0

Solomon Islandsa - - - 1

Sri Lanka 2 6 8 4

Total 6 25 31 18

No answerb 3 6 9

DMC = developing member country, PTL = program team leader.

a No email addresses of contacts.

b Since the survey is anonymous, the respondent can opt not to indicate their country of operations.


3. Length of involvement in ADB-supported RBL Program

6. One of every five DMC client respondents (particularly the executing or implementing agency

officers) and one of every three PTLs had been involved with RBL program since it started 3 years ago.

More DMC clients (33%) than PTLs (11%) were relatively new in the RBL program (Table A6.3).

Table A6.3. Years of RBL Project Involvement

Central Government

Executing and

Implementing

Agencies Total DMC Clients PTLs

No. % No. % No. % No. %

One (1) year or less 4 44.4 9 29.0 13 32.5 2 11.1

More than 1 year to 2

years 4 44.4 10 32.3 14 35.0 5 27.8

More than 2 years to 3

years 0 0.0 5 16.1 5 12.5 5 27.8

More than 3 years 1 11.1 7 22.6 8 20.0 6 33.3

Total 9 100.0 31 100.0 40 100.0 18 100.0



4. Activity Involvement, Roles, and Responsibilities in the RBL Program

7. DMC central government officers who participated in the survey had been involved in

(i) identifying the program to be supported by RBL (44%), (ii) policy dialogues (33%), and (iii) verification

of results and disbursements against disbursement-linked indicators (DLIs) (33%). Most executing and

implementing agency respondents were involved in (i) verification of results and disbursements against

DLIs (64%), (ii) implementation and monitoring of the program action plan, PAP (64%), and

(iii) assessment of program systems (45%)—Table A6.4.

8. Conversely, most PTL respondents (at least two-thirds) had been involved in: (i) assessment of

program soundness, expenditure and financing framework, results and links with disbursements (89%);

(ii) support to government in identifying and designing the country’s RBL program (89%); (iii) assessment

of programs systems (83%); (iv) assessment of the RBL program (83%); (v) supervision of RBL program

implementation (72%); (vi) monitoring of RBL program performance and risks (67%); and (vii) working

with development partners to support the country in implementing measures to improve the RBL

program (Table A6.5).


3

Table A6.4. Activity Involvement of DMC Client Respondents

Central Government

Executing and

Implementing

Agencies All DMC Clients

No. % No. % No. %

Identification of program to be supported ADB

through the results-based lending modality 4 44.4 12 38.7 16 40.0

Assessment of program soundness, expenditure and

financing frameworks, and/or results and links with

disbursements

1 11.1 13 41.9 14 35.0

Assessment of monitoring and evaluation system,

fiduciary system, safeguards system, and/or other

program systems

1 11.1 14 45.2 15 37.5

Design of the RBL program 1 11.1 7 22.6 8 20.0

Development of the program action plan 1 11.1 16 51.6 17 42.5

Implementation and monitoring of the program

action plan 2 22.2 20 64.5 22 55.0

Monitoring and managing fiduciary and safeguard

issues and other risks 0 0.0 6 19.4 6 15.0

Addressing performance problems and ensuring

effective feedback loops 0 0.0 7 22.6 7 17.5

Participation in policy dialogues 3 33.3 12 38.7 15 37.5

Verification of results and disbursements against

disbursement-linked indicators 3 33.3 20 64.5 23 57.5

Othersa 1 11.1 1 3.2 2 5.0

Total 9 100.0 31 100.0 40 100.0

DMC = developing member country.

a Others include: for central government: overall coordination of the program, mainly financial aspects; for the executing or implementing agencies:

(i) program management, (ii) facilitation of the submission of the ADB required financial reports available at the Department of Education Accounting

Division, coordinate with the Department of Education Budget Division regarding financial reports prepared by their office, coordinate with the

Commission on Audit and Department of Education Project Management Staff regarding ADB financial requirements; attend selected meetings

regarding ADB loan; (iii) implementation of the RBL program.


Table A6.5. Activity Involvement of ADB Staff and/or PTL Client Respondents

PTLs

No. %

Assessment of program soundness, expenditure and financing

frameworks, and/or results and links with disbursements 16 88.9

Assessment of monitoring and evaluation system, fiduciary

system, safeguards system, and/or other program systems 15 83.3

Assessment of the results-based lending (RBL) program in

coordination with the government 15 83.3

Support to the government in identifying and designing

measures to improve the RBL Program, its systems, and risk-

mitigating measures

16 88.9

Working with development partners as appropriate to support

the country in implementing improvement measures 12 66.7

Monitoring of RBL program performance and risks 12 66.7

Supervision of RBL program implementation 13 72.2

Provision of technical advice in support of RBL implementation

including resolution of issues 10 55.6

Others a 3 16.7

Total 18 100.0

PTL = program team leader.

a Others include: (i) program team leader and (ii) support the mission leader in processing the RBL.



4

C. Summary of Stakeholders’ Perceptions

1. Identifying sector priorities to be supported by RBL and defining program boundaries

9. Most DMC clients (87%) find their engagement with the ADB in identifying sector priorities to be

supported by RBL, “extremely/very useful” (Table A6.6).

Table A6.6. Usefulness of ADB’s Engagement with the Government in

Identifying Sector Priorities for RBL

Central Government

Executing and

Implementing

Agencies Total DMC Clients

No. % No. % No. %

Extremely/very usefula 8 88.9 25 86.2 33 86.8

Moderately useful 1 11.1 4 13.8 5 13.2

Slightly/not at all useful 0 0.0 0 0.0 0 0.0

Don’t know 0 0.0 0 0.0 0 0.0

Total 9 100.0 29 100.0 38 100.0


a Executing or implementing agency comments: (i) ADB has given the most priority to the development of the education sector

plan and its implementation through the technical and financial support; (ii) conducting research and publication of insightful

reports have contributed to the establishment of accurate policies; (iii) especially ADB's RBL support for distribution systems,

staff training and certification programs.


10. Identifying country sector priorities to be supported by RBL is “not difficult” for two-thirds (67%)

of PTL respondents. It seems that PTLs find it slightly more difficult to define the boundaries of the RBL

program as distinct from the government program; 50% of PTLs rated defining boundaries as

“moderately difficult” (Table A6.7).

Table A6.7. PTLs’ Level of Difficulty in Identifying Sector Priorities and

In Defining RBL Boundaries

PTLs’ Level of Difficulty

Identifying Sector Prioritiesa Defining Boundaries

b

No. of PTLs % No. of PTLs %

Extremely/very

difficult 1 5.6 1 5.6

Moderately difficult 5 27.8 9 50.0

Slightly/not at all

difficult 12 66.7 8 44.4

Don’t know 0 0.0 0 0.0

Total 18 100.0 18 100.0


a Comments: (i) “Not difficult”—The first RBL financed by ADB in Sri Lanka was in the education sector and the first RBL in

PRC was in the same sector. Expanding RBL to other sectors, such as transport and energy is much more difficult because

of safeguard requirements; (ii) “Difficult” —Difficult due to the breadth and range of sector priorities; some priorities were

selected but the enabling policy environment was not considered.

b Comments: (i) This largely depended on the mission leader, his/her knowledge and familiarity with the RBL. (ii) Negotiating

boundaries was intense but not really difficult. Government expected more from ADB.


2. Technical Assistance Capacity Building Support for Clients and Staff

11. Not all DMC clients and PTLs think that the identification of the country’s needs for capacity

building or TA related to the RBL program has been accurate. Only 18% of DMC client respondents and

28% of PTLs think that the identification of RBL-related capacity building or TA needs of the country has

been “highly accurate” (Table A6.8). The majority of DMC government respondents (82%) and of PTLs

(72%) indicated that TA needs identification was less than “highly accurate” with the majority (about


5

two-thirds) of both groups of respondents saying it has been “moderately accurate.” About 13% of DMC

respondents considered the TA needs identification “slightly accurate” and 6% of PTLs found it to be

“not accurate.”

Table A6.8. Accuracy of Identification of Country’s TA Capacity Building Needs

Central Government

Executing and

Implementing

Agencies Total DMC Clients PTLs

No. % No. % No. % No. %

Highly accurate 2 22.2 5 17.2 7 18.4 5 27.8

Moderately accurate 4 44.4 20 69.0 24 63.2 12 66.7

Slightly accurate 2 22.2 3 10.3 5 13.2 0 0.0

Not accurate a 0 0.0 0 0.0 0 0.0 1

a 5.6

a

Don’t know 1 11.1 1 3.4 2 5.3 0 0.0

Total 9 100.0 29 100.0 38 100.0 18 100.0


a PTL’s comment: TA was discouraged by ADB and reduced to focus on risk mitigation of ADB funds. Government wanted

more assistance with policy reforms.


12. For two-thirds of DMC clients (68%), the project preparatory technical assistance (PPTA) provided

by ADB in the preparation of the country's RBL Program had been “extremely/very important” (Table

A6.9). A respondent cited the case of the preparatory TA for Nepal’s School Sector Development Plan

provided by ADB, which he/she said was so useful that the government was able to use it as the strategic

plan for the development of the sector.

Table A6.9. Importance of the PPTA for the RBL Program

Central Government

Executing and

Implementing

Agenciesa Total DMC Clients

No. % No. % No. %

Extremely/very important 7 77.8 18 64.3 25 67.6

Moderately important 1 11.1 5 17.9 6 16.2

Slightly/not at all

important 0 0.0 1 3.6 1 2.7

No PPTA was provided 0 0.0 0 0.0 0 0.0

Don’t know 1 11.1 4 14.3 5 13.5

Total 9 100.0 28 100.0 37 100.0


a Comment: During the preparation of the School Sector Development Plan in Nepal, the TA provided by ADB was so

useful that it was used as the strategic plan for the development of school education sector through the joint efforts

made by Government of Nepal and financing partners.


13. Like the PPTA, the capacity development technical assistance (CDTA) provided by ADB in building

the country’s capacity and institutional strengthening related to RBL was considered important by 66%

of DMC clients (Table A6.10). However, 26% of government clients indicated that the CDTA was

“moderately or slightly important”, which could mean that RBL capacity building efforts so far may not

have been enough.


6

Table A6.10. Importance of Capacity Building Technical Assistance

Central Governments

Executing and

Implementing


No. % No. % No. %

Extremely/very importanta 7 77.8 18 62.1 25 65.8


Slightly/not at all

importantb 1 11.1 0 0.0 1 2.6

No CDTA was provided 0 0.0 1 3.4 1 2.6

Don’t know 1 11.1 1 3.4 2 5.3

Total 9 100.0 29 100.0 38 100.0


a Comment: Technical assistance given by ADB in different sectors to strengthen the education system of

Nepal has also enhanced the institutional capacity of the ministry system.

b Comments: (i) CDTA has a memorandum of understanding but the intended CDTA assistance is not

provided to concerned offices. (ii) Focus must be sensitive to project and project implementer's needs.


3. ADB’s due diligence efforts in terms of pre-design diagnostics and assessments

14. The DMC clients were asked to rate the quality of RBL program pre-design diagnostics and

assessments done by the ADB staff, as follows: (i) program soundness, (ii) expenditure frameworks and

financing plans, (iii) results and links with disbursements, (iv) monitoring and evaluation systems, (v)

fiduciary systems, (vi) procurement systems, and (vii) environment and social safeguard systems. It can

be noted from Table A6.11 that the majority of the respondents (at least 60%) consider the quality of

each of these efforts as “very good or good”. The biggest proportion of government respondents (86%)

that gave “very good/good” rating referred to ADB’s efforts in ensuring “program soundness.” Fewer

government respondents (60%) gave the highest rating for the quality of ADB’s assessment of monitoring

and evaluation (M&E) system for the RBL program (Table A6.11).

Table A6.11. DMC Clients’ Rating of the Quality of ADB’s Due Diligence Efforts

Very Good/

Good Acceptable

Poor/Very

Poor

Don't

know Total

No. % No. % No. % No. % No. %

Central Governments

Program soundness 8 100.0 0 0.0 0 0.0 0 0.0 8 100.0

Expenditure frameworks and

financing plans 6 75.0 2 25.0 0 0.0 0 0.0 8 100.0

Results and links with disbursements 5 62.5 3 37.5 0 0.0 0 0.0 8 100.0

Monitoring and evaluation systems 6 75.0 2 25.0 0 0.0 0 0.0 8 100.0

Fiduciary systems 7 87.5 1 12.5 0 0.0 0 0.0 8 100.0

Procurement systems 7 87.5 1 12.5 0 0.0 0 0.0 8 100.0

Environment and social safeguard

systems 7 87.5 1 12.5 0 0.0 0 0.0 8 100.0

Executing and Implementing Agencies



financing plans 16 59.3 8 29.6 1 3.7 2 7.4 27 100.0






systems 16 59.3 8 29.6 1 3.7 2 7.4 27 100.0

Total DMC Clients



7

Very Good/

Good Acceptable

Poor/Very

Poor

Don't

know Total

No. % No. % No. % No. % No. %


financing plans 22 62.9 10 28.6 1 2.9 2 5.7 35 100.0






systems 23 65.7 9 25.7 1 2.9 2 5.7 35 100.0


Executing or implementing agency comment: Consider introduction of monitoring and evaluation software tool in order to ensure

efficiency and accuracy of monitoring.


15. On the other hand, the PTLs’ rating of their level of difficulty in doing pre-design diagnostics and

assessments was somewhat mixed. As is shown in Table A6.12, the majority of the PTLs (56%) find

assessment of program soundness slightly or not at all difficult. Moderately difficult for at least half of

the PTLs are the pre-diagnostics of the environmental and social safeguard systems (61%), fiduciary

systems (56%), and procurement systems (50%).

Table A6.12. Project Team Leaders’ Level of Difficulty in Pre-Design Diagnostics and Assessments

Extremely/Very

Difficult

Moderately

Difficult

Slightly/Not at

all Difficult Total

No. % No. % No. % No. %

Program soundness 3 16.7 5 27.8 10 55.6 18 100.0

Expenditure frameworks and financing

plans 5 27.8 8 44.4 5 27.8 18 100.0

Results and links with disbursementsa 6 33.3 6 33.3 6 33.3 18 100.0

Monitoring and evaluation systems 4 22.2 8 44.4 6 33.3 18 100.0

Fiduciary systems 2 11.1 10 55.6 6 33.3 18 100.0

Procurement systems 3 16.7 9 50.0 6 33.3 18 100.0


systems 1 5.6 11 61.1 6 33.3 18 100.0

a Comment: design of disbursement-linked indicators for the second results-based lending program in the People’s Republic of

China (also in the education sector) was extremely difficult due to anticipated high expectation from the Board.


16. With regard to the sufficiency of resources, particularly the amount of time to complete pre-

design diagnostics and assessments for the RBL program, 61% of PTL respondents though that resources

are sufficient, but 56% feel that time available is inadequate (Table A6.13).


8

Table A6.13. Sufficiency of Resources and Adequacy of Time to Complete

Pre-Design Diagnostics and Assessments

Resources Timea

No. of PTLs % No. of PTLs %

More than

sufficient/adequate 2 11.1 1 6.3

Sufficient/adequate 9 50.0 6 37.5

Less than sufficient/adequate 7 38.9 5 31.3

Not at all sufficient/adequate 0 0.0 4 25.0

Don’t know 0 0.0 0 0.0

Total 18 100.0 16 100.0


a Comments: (1) “Not adequate”—Everything is always rushed at ADB due to artificial deadlines linked to

bunching season; (2) “Adequate”—The program design used a lot of assessments completed by other

development partners and ADB, which are available to the program team.


17. According to 75% of PTL respondents, at least equal, if not greater efforts, are required to

complete the preparation and processing of RBL program compared with those needed for other lending

modalities (Table A6.14). About 38% indicated that RBL requires greater effort while 19% said that it

requires less effort than other modalities. One PTL who claimed greater effort was required by RBL

explained that much more work was required in developing the results chain and DLIs as RBL requires a

much more holistic appreciation of constraints, policy context, budget allocation to sequence DLI and

implementation arrangements. Moreover, he/she noted that closer engagement with government was

required under RBL.

Table A6.14. Effort Required in the Preparation and Processing of RBL Compared with Other Lending

Modalities

PTL

No. %

RBL requires greater effort than other modalities 6 37.5

RBL requires equal effort than other modalities 6 37.5

RBL requires less effort than other modalities 3 18.8

Don’t know 1 6.3

Total 16 100.0


Comments: (i) Much more work is required in developing the results chain and DLIs—much more

holistic appreciation of constraints, policy context, budget allocation is required to sequence the

disbursement-linked indicators and implementation arrangements. Closer engagement with

government is required which is really good; (ii) It depends on how much assessment has been done

on the sector and country systems. Nevertheless, efforts to complete the preparation and processing

of the RBL program should reduce over time, with more assessments done for the sector and country

systems.


4. Disbursement-Linked Indicators (DLIs)

18. Quality of DLIs. The majority of DMC stakeholders (at least 85%) and PTLs (at least 78%) have

encouraging views on the DLIs of the country’s RBL program (Table A6.15). They generally agree that:

(i) DLIs were effectively formulated and their achievement would ensure attainment of the RBL program’s

intended outcomes, (ii) DLIs are sufficiently ambitious, yet realistic, (iii) the DLI verification protocols are

clear in terms of defining how their achievement will be measured and verified, and (iv) the DLIs and the

PAP complement each other. However, it should be noted that PTLs had slightly less positive views than

DMC government clients on the quality of DLIs.

19. Most DMC stakeholder respondents (94%) agreed that “The DLI verification protocols are clear

in terms of defining how their achievement will be measured and verified” while most PTLs (89%) believed


9

that “The DLIs and the PAP complement each other.” Both, however, seem to be less than fully convinced

that the “DLIs are sufficiently ambitious, yet realistic” as this statement had the least agreement from

DMC clients (79%) and PTLs (78%).

Table A6.15. Agreement or Disagreement with Statements on the Quality of DLIs

Strongly

Agree/ Agree

Neutral/ No

Opinion

Disagree/Strongly

Disagree Total

No. % No. % No. % No. %

DMC Clients

The disbursement-linked indicators (DLIs) were

effectively formulated such that their achievement

ensures attainment of the Program’s intended

outcomes.

31 91.2 1 2.9 2 5.9 34 100.0

The DLIs are sufficiently ambitious, yet realistic. 27 79.4 4 11.8 3 8.8 34 100.0

The DLI verification protocols are clear in terms of

defining how their achievement will be measured

and verified.

32 94.1 1 2.9 1 2.9 34 100.0

The DLIs and the Program Action Plan (PAP)

complement each other. 29 85.3 2 5.9 3 8.8 34 100.0

PTLs

The DLIs were effectively formulated such that their

achievement ensures attainment of the Program’s

intended outcomes.

14 77.8 2 11.1 2 11.1 18 100.0

The DLIs are sufficiently ambitious, yet realistic. 14 77.8 1 5.6 3 16.7 18 100.0

The DLI verification protocols are clear in terms of

defining how their achievement will be measured

and verified.

14 77.8 1 5.6 3 16.7 18 100.0

The DLIs and the PAP complement each other. 16 88.9 2 11.1 0 0.0 18 100.0


Comments: (1) Executing or implementing agency comment: There must be an established system of revising, refining, or polishing indicators,

so they remain relevant to both the program and the department; (ii) PTL comment: Of the three programs I am involved with, DLIs for one of

them were not very effective.


20. Independent verification of DLIs. About 76% of DMC stakeholder respondents thought that the

independent verification of DLIs was “extremely/very important” in ensuring the achievement of the

intended results of the RBL Program. By contrast, only 9% believed otherwise (Table A6.16). A respondent

commented that independent verification of DLIs would ensure whether progress was achievable or not,

and suggest necessary action for on-time completion of each and every activity.

Table A6.16. Importance of Independent Verification of DLIs

Central Government

Executing and

Implementing


No. % No. % No. %

Extremely/very important 7 87.5 18 72.0 25 75.8


Slightly/not at all

important 0 0.0 3 12.0 3 9.1

Don’t know 0 0.0 0 0.0 0 0.0

Total 8 100.0 25 100.0 33 100.0


Comments: (i) Independent verification of DLIs will ensure whether progress is achievable or not, and suggest necessary action

for on-time completion of each and every activity. (ii) Independent verification will need more achievement data to prove the

result of the RBL program.



10

21. Management of the DLI results verification process. About 64% of executing agency or

implementing agency respondents against only 27% of PTLs rated the management of the DLI results

verification process “extremely/very effective”. The rest, 24% of government stakeholders and 60% of

PTLs, gave a lower rating of “moderately effective.” Some 12% of the respondents replied with “don’t

know,” which indicates that it is too early to say (Table A6.17).

Table A6.17. Effectiveness of the Management of the DLI Results Verification Process

Executing and

Implementing Agencies PTLs Total

No. % No. % No. %

Extremely/very effective 16 64.0 4 26.7 20 50.0

Moderately effective 6 24.0 9 60.0 15 37.5

Slightly/not at all effective 0 0.0 0 0.0 0 0.0

Don’t know 3 12.0 2 13.3 5 12.5

Total 25 100.0 15 100.0 40 100.0


Executing or implementing agency comments: (i) We have not raised any withdrawal application. Hence could not answer this

question at this point of time. (ii) It is still early to give the decision on this questionnaire. It could be answered once independent

verification agencies' output is seen.

PTL comments: (i) As far as the verification protocol is clear and fully understood by the executing and implementing agencies, and

verification sources (e.g., independent verification) are available, it could be very effective. (ii) External third-party verification is

really required. Must be transparent and reported. (iii) Verification protocols are sometimes intentionally kept general, not very

detailed, as we cannot anticipate everything upfront and design the verification protocol very rigidly. Therefore, during

implementation, the verification protocol needs to be further refined. Also, executing agencies need to be guided on the

preparation of evidence or documentation of evidence.


5. RBL Program Action Plan

22. Usefulness of the PAP. The majority of both DMC clients (88%) and PTLs (61%) found the PAP

useful in supporting RBL implementation. However, they differed in their opinion of the PAP’s usefulness

in capacity building, strengthening program systems, and in mitigating risks (Table A6.18).

23. In capacity building, only 44% of PTLs compared with 67% of DMC clients indicated that the PAP

was useful. Some 11% of PTLs even said that the PAP was not useful at all for the purpose of capacity

building. For the strengthening of program systems, 88% of DMC respondents versus 44% of PTL

respondents thought that the PAP was useful. Likewise, 73% of DMC clients against 33% of PTLs believed

that the PAP was useful.

Table A6.18. Usefulness of the RBL Program Action Plan

Extremely/

Very Useful

Moderately

Useful

Slightly/Not

At All Useful

Don't

Know Total

No. % No. % No. % No. % No. %

DMC Clients

In capacity building 22 66.7 10 30.3 1 3.0 0 0.0 33 100.0

In strengthening program systems 29 87.9 3 9.1 1 3.0 0 0.0 33 100.0

In mitigating risks 24 72.7 4 12.1 4 12.1 1 3.0 33 100.0

In supporting implementation 29 87.9 4 12.1 0 0.0 0 0.0 33 100.0

ADB Staff PTLs

In capacity building 8 44.4 7 38.9 2 11.1 1 5.6 18 100.0

In strengthening program systems 8 44.4 6 33.3 3 16.7 1 5.6 18 100.0

In mitigating risks 6 33.3 9 50.0 2 11.1 1 5.6 18 100.0

In supporting implementation 11 61.1 6 33.3 0 0.0 1 5.6 18 100.0


PTL comment: The PAP is useful but is not prioritized by government as it tends to focus on DLIs, especially those that can be

addressed quickly. Policy reforms have been overlooked a little, which undermines the effectiveness and achievement of intended

results.



11

24. Completion of PAP activities. According to 76% of DMC client respondents, most or more than

half of the activities of the RBL program specified in the PAP had been completed on time, with 12%

saying that all activities in fact have been completed as scheduled (Table A6.19).

Table A6.19. Proportion of PAP Activities Completed as Scheduled

Central Government

Executing and

Implementing

Agencies

Total DMC Clients

No. % No. % No. %

All activities are being completed as scheduled 0 0.0 4 16.0 4 12.1

Most/More than half of activities are being completed as

scheduled 4 50.0 17 68.0 21 63.6

Half of the activities are being completed as scheduled 2 25.0 2 8.0 4 12.1

Less than half of the activities are being completed as

scheduled 0 0.0 1 4.0 1 3.0

No activity is being completed as scheduled 0 0.0 0 0.0 0 0.0

Don’t know 2 25.0 1 4.0 3 9.1

Total 8 100.0 25 100.0 33 100.0

DMC = developing-member country


25. Effectiveness of PAP implementation. The implementation of the PAP has been “extremely/very

effective” as noted by 60% of executing or implementing agency respondents but only 27% of PTLs (Table

A6.20). Of the PTL respondents, 20% even indicated that PAP implementation was “slightly/not effective”.

Table A6.20. Effectiveness of PAP Implementation

Executing and

Implementing

Agencies PTLs Total

No. % No. % No. %



Slightly/not effective 0 0.0 3 20.0 3 7.5

Don’t know 1 4.0 1 6.7 2 5.0

Total 25 100.0 15 100.0 40 100.0


Executing or implementing agency comments: (i) RBL PAP is very useful for the implementation of secondary

school programs, it drives the program and makes the team accountable for the result through in-time

implementation of the programs. (ii) Some PAPs need a time schedule and process to be completed in

Perusahaan Listrik Negara (State Electricity Company)—PLN.

PTL comments: (i) PAP requirements or recommendations were too often overlooked or avoided. (ii) There are

differences between different RBL programs. In one RBL, PAP implementation is very good while in the other

one, PAP is not very much followed up by the executing agency.


6. Technical Advice Provided by ADB in Support of RBL Program Implementation

26. Most DMC client respondents (80%) considered the technical advice provided by ADB in support

of the implementation of the country's RBL program to be adequate (Table A6.21).


12

Table A6.21. Adequacy of ADB Technical Advice

Central Government

Executing and

Implementing


No. % No. % No. %

More than adequate 1 12.5 2 8.0 3 9.1

Adequate 6 75.0 18 72.0 24 72.7

Less than adequate a 0 0.0 3 12.0 3 9.1

Not adequate 0 0.0 1 4.0 1 3.0

Don’t know 1 12.5 1 4.0 2 6.1

Total 8 100.0 25 100.0 33 100.0

DMC = developing member country

a Executing or implementing agency comment: in some areas, the technical assistance needed to be

expanded for the successful implementation of all program activities within a given time.


27. According to 60% of PTLs, ADB’s technical advice during RBL program implementation had been

adopted by the executing or implementing agencies “to a somewhat high degree” (Table A6.22). But

13% of PTLs indicated that adoption of the technical advice was “to a somewhat low degree.” No

respondents thought that there was a “very high degree” of acceptance of ADB’s technical advice.

Table A6.22. Extent of the Country’s Adoption of ADB’s Technical Advice

PTL

No. %

To a very high degree 0 0.0

To a somewhat high degree a 9 60.0

To a somewhat low degree a 2 13.3

To a very low degree 0 0.0

Don’t know 4 26.7

Total 15 100.0


a Comments: (1) ADB needs to put more effort into the provision of technical advice. (2) I

have a mixed case. One executing agency is very receptive while the other is not so much.


7. Monitoring of critical risks

28. Risks critical to the implementation of the RBL program are monitored to a “high degree” as

indicated by 72% of executing and implementing agencies and 60% of PTLs. However, there are cases

where apparently risks are monitored to a “low degree” according to 24% of executing agency and

implementing agency respondents and 20% of PTLs (Table A6.23).

Table A6.23. Extent of Monitoring of Critical Risks

Executing and Implementing

Agencies a PTLs

No. % No. %

To a very high degree 5 20.0 3 20.0

To a somewhat high degree 13 52.0 6 40.0

To a somewhat low degree 6 24.0 1 6.7

To a very low degree 0 0.0 2 13.3

Don’t know 1 4.0 3 20.0

Total 25 100.0 15 100.0


a Comment: It appears as high-risk because of the set disbursement-linked indicators (targets and protocol) and political

transition of the country.



13

29. Risks that may arise during RBL program implementation seemed to have been anticipated. Two-

thirds (67%) of PTL respondents estimated that the number of risks that materialized during

implementation were the same or fewer than those identified during the risk assessment (Table A6.24).

However, one PTL noted that more risks arose in the course of implementation than were identified

during risk assessment.

Table A6.24. Number of Risks during Implementation Compared with Risk Assessment

PTLs

No. %

Greater number of risks during implementation than in risk

assessment 1 6.7

Same number of risks during implementation as in risk assessment 7 46.7

Fewer risks during implementation than in risk assessment 3 20.0

Don’t know 4 26.7

Total 15 100.0



8. Implementation of the country’s RBL-supported program

30. About 85% of DMC government stakeholders, but only 67% of PTLs were satisfied with the

implementation of the RBL-supported program. No DMC client was not satisfied with how the program

had implemented, but 13% of PTLs are somewhat dissatisfied (Table A6.25).

Table A6.25. Satisfaction or Dissatisfaction with RBL Implementation

Central

Governments

Executing and

Implementing

Agencies All DMC Clients PTLs Total

No. % No. % No. % No. % No. %

Very/somewhat satisfied 7 87.5 21 84.0 28 84.8 10 66.7 38 79.2

Neither satisfied nor

dissatisfied 1 12.5 3 12.0 4 12.1 1 6.7 5 10.4

Somewhat/Very dissatisfied 0 0.0 0 0.0 0 0.0 2 13.3 2 4.2

Don’t know 0 0.0 1 4.0 1 3.0 2 13.3 3 6.3

Total 8 100.0 25 100.0 33 100.0 15 100.0 48 100.0

DMC = developing-member country, PTL = program team leader.


31. While the majority (70%) of DMC clients considered ADB’s support and supervision of the

country’s RBL implementation to have been “extremely/very effective”, some 9% thought that ADB’s

support and supervision had been “slightly/not effective” (Table A6.26).

Table A6.26. Effectiveness of ADB's Support and Supervision of RBL Implementation

Central Government

Executing and

Implementing

Agencies All DMC Clients

No. % No. % No. %

Extremely/very

effective 5 62.5 18 72.0 23 69.7



Don’t know 1 12.5 1 4.0 2 6.1

Total 8 100.0 25 100.0 33 100.0

DMC = developing-member country



14

9. Extent to Which RBL Programs Strengthen the Country’s Program Systems

32. For the majority (77%) of stakeholders (85% of DMC clients and 60% of PTLs), the extent to which

RBL programs strengthen program systems is “high” (Table A6.27). However, a significant proportion of

PTLs (33%) responded “don’t know,” which could mean that there is still no evidence yet of this expected

value added of RBL.

Table A6.27. Extent to Which RBL Programs Strengthened DMC Program Systems

Central Government

Executing and

Implementing


No. % No. % No. % No. % No. %

To a very high degree 2 25.0 9 36.0 11 33.3 4 26.7 15 31.3

To a somewhat high

degree 6 75.0 11 44.0 17 51.5 5 33.3 22 45.8

To a somewhat low

degree 0 0.0 3 12.0 3 9.1 1 6.7 4 8.3

To a very low degree 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0

Don’t know 0 0.0 2 8.0 2 6.1 5 33.3 7 14.6

Total 8 100.0 25 100.0 33 100.0 15 100.0 48 100.0


Comments: DMC Clients: (i) The Government of Indonesia still needs adjustments to some regulations related to the implementation of RBL. (ii)

In some areas, government departments have been reluctant or lethargic in adapting the recommendations or requirements of the RBL program

which reduced the effectiveness of the program. PTLs: (i) Too early to assess. (ii) This supports government efforts to move toward performance-

based contracting and better performance management of staff and contractors.


10. Extent to which RBL has led to Greater Country Ownership of ADB-Supported

Operations

33. Close to 85% of DMC government respondents and 60% of PTLs perceived that the RBL modality

had to a “high extent” led to greater country ownership of programs supported by ADB (Table A6.28).

On the other hand, 9% of DMC clients and 33% of PTLs may have not yet felt or observed this result of

RBL based on their “don’t know” responses.

Table A6.28. Extent RBL has Led to Greater Country Ownership of ADB-supported Operations

Central Government

Executing and

Implementing

Agencies

All DMC

Clients PTLs Total

No. % No. % No. % No. % No. %

To a very high degree 3 37.5 6 24.0 9 27.3 4 26.7 13 27.1

To a somewhat high

degree 5 62.5 14 56.0 19 57.6 5 33.3 24 50.0

To a somewhat low

degree 0 0.0 2 8.0 2 6.1 1 6.7 3 6.3

To a very low degree 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0

Don’t know 0 0.0 3 12.0 3 9.1 5 33.3 8 16.7

Total 8 100.0 25 100.0 33 100.0 15 100.0 48 100.0


Comments: Executing and implementing agencies: For example, the RBL program was extended to PLN region of Sulawesi, following the PLN

region of Sumatera. PTL: The agenda remains the responsibility of the government and they are becoming more realistic and clear in what

they want to achieve.



15

11. Effectiveness of RBL in Promoting Partnerships among Other Donors

34. Two–thirds of PTLs (67%) confirm the effectiveness of the RBL modality in promoting partnerships

among other donors, including cofinancing (Table A6.29).

Table A6.29. Effectiveness of RBL in Promoting Partnerships

Central Government a PTL

b Total

No. % No. % No. %




Don’t know 3 37.5 1 6.7 4 17.4

Total 8 100.0 15 100.0 23 100.0


a Comment: There has been no learning for the results of RBL implementation in Indonesia. Still in progress.

b Comments: (i) The cofinancing with KFW under the second proposed results-based lending program is very effective. (ii) This

will improve once donors see improvements in delivery and more serious attention directed towards achievement of specified

results. It is building credibility currently. Donors are watching carefully and are likely to support future reforms now that they

see ADB and World Bank investing wisely and are more serious in supporting government to achieve specified results.


12. Extent to Which RBL Complements other ADB Lending Modalities

35. About 67% of PTL respondents confirmed that the RBL modality complemented other ADB

lending modalities to a “high degree,” while 20% observed that they were complementary only to a low

degree (Table A6.30).

Table A6.30. Extent to Which RBL Complements Other ADB Lending Modalities

PTLs

No. %

To a very high degree 1 6.7

To a somewhat high degree 9 60.0

To a somewhat low degree 2 13.3

To a very low degree 1 6.7

Don’t know 2 13.3

Total 15 100.0


Comment: There have been few RBL programs in a limited number of sectors so far.

RBL should be able to complement other modalities more if more RBLs in more

sectors are processed.


13. Usefulness of RBL in Reducing Transaction Costs

36. More than half of PTLs (60%) affirmed the usefulness of RBL in reducing transaction costs (Table

A6.31). One PTL commented that the lower cost may be attributed to the use of government systems.


16

Table A6.31. Usefulness of RBL in Reducing Transaction Costs

PTLs

No. %

Extremely/very useful 9 60.0

Moderately useful 4 26.7

Slightly/not useful 1 6.7

Don’t know 1 6.7

Total 15 100.0


Comments: (i) It may take more time from both ADB and government at the

beginning of implementation to fully understand the requirements of the DLI

verification process; (ii) RBL is using more government systems.


14. Extent to Which RBL’s Intended Benefits are Being Achieved in the Country

37. At least 88% of DMC stakeholders and 71% of ADB PTLs believe that the intended benefits of

RBL are being achieved in countries, namely: (i) greater responsiveness to the needs of the country,

(ii) supporting good governance, (iii) increased accountability and incentives for development results,

and (iv) enhanced development and aid effectiveness (Table A6.32).

38. RBL would be able to achieve enhanced development effectiveness to a “high degree” according

to 94% of DMC clients and greater responsiveness to the needs of the country (91%). The high extent of

achievement of RBL in terms of greater responsiveness to the needs of the country and increased

accountability and incentives for development results were acknowledged by 86% of PTLs.

Table A6.32. Extent of Achieving RBL Program’s Intended Benefits

To a Very

High

Degree

To a Somewhat

High Degree

To a Somewhat

Low Degree

To a very

Low

degree

Don't

know Total

No. % No. % No. % No. % No. % No. %

DMC Clients

Greater responsiveness to the

needs of the country 6 18.2 24 72.7 1 3.0 0 0.0 2 6.1 33 100.0

Supporting good governance 10 30.3 19 57.6 3 9.1 0 0.0 1 3.0 33 100.0

Increased accountability and

incentives for development results 7 21.2 22 66.7 2 6.1 1 3.0 1 3.0 33 100.0

Enhanced development / aid

effectiveness 5 15.2 26 78.8 1 3.0 0 0.0 1 3.0 33 100.0

ADB Staff PTLs

Greater responsiveness to the

needs of the country 4 28.6 8 57.1 1 7.1 0 0.0 1 7.1 14 100.0

Supporting good governance 2 14.3 8 57.1 2 14.3 0 0.0 2 14.

3 14 100.0

Increased accountability and

incentives for development results 6 42.9 6 42.9 1 7.1 0 0.0 1 7.1 14 100.0

Enhanced development / aid

effectiveness 3 21.4 8 57.1 1 7.1 0 0.0 2

14.

3 14 100.0



15. Two Greatest Strengths of the RBL modality

39. The responses of DMC clients and PTLs when asked to choose two strengths of the RBL modality

are summarized in Table A6.33. It can be noted that “focus on results” was the top strength of cited by

all three groups of respondents: DMC central government (75%), executing and implementing agencies

(80%), and PTLs (86%).


17

40. The second most-frequently cited strength of RBL by both executing and implementing agencies

(24%) and PTLs (43%) is the “use of government program systems” (Table A6.33).

Table A6.33. Two Greatest Strengths of the RBL Modality

Central Government

Executing

and

Implementing

Agencies

All DMC

Clients PTLs Total

No. % No. % No. % No. % No. %

Efficiency in processing the

preparation of the operation 3 37.5 4 16.0 7 21.2 1 7.1 8 17.0

Use of government program

systems 2 25.0 6 24.0 8 24.2 6 42.9 14 29.8

Focus on results 6 75.0 20 80.0 26 78.8 12 85.7 38 80.9

Focus on program institutional

strengthening 0 0.0 6 24.0 6 18.2 1 7.1 7 14.9

Ability to include capacity-building

measures and focus on incentives 1 12.5 3 12.0 4 12.1 3 21.4 7 14.9

Enhanced ADB staff support to

government in identifying and

addressing key issues

1 12.5 2 8.0 3 9.1 4 28.6 7 14.9

Usefulness of ADB assessments in

designing the results-based

lending program

1 12.5 2 8.0 3 9.1 1 7.1 4 8.5

Other (Control of funding for a

designated-results oriented task) 0 0.0 1 4.0 1 3.0 0 0.0 1 2.1

Total 8 100.0 25 100.0 33 100.0 14 100.0 47 100.0



16. Two Greatest Challenges of the RBL Modality

41. The greatest challenge of RBL is “the novelty of RBL and issues related to its newness” according

to 76% of DMC clients and governments and 57% of PTLs. The second greatest challenge for 39% of

DMC government respondents is “delays in processing the preparation of the operation,” which to PTLs

seem to be less of a problem as only 7% marked this as a top challenge. For 29% of PTLs, the greater

problems are “exclusions of high-value contracts” and “heaviness in conducting ADB assessments” (Table

A6.34).

42. Other challenges identified by the respondents include: (i) difficulty in revising unrealistic targets

identified at the implementation process (executing and implementing agencies); (ii) lack of

understanding of RBL among ADB Management and the Board (PTL); (iii) pressure to put higher amount

of disbursement on outcome-level indicators (PTL); (iv) burden on project officer in verifying the results

and addressing issues arising from bottlenecks to the result achievement, which can be beyond one

person's capacity or the program scope (PTL).


18

Table A6.34. Two Main Greatest Challenges of the RBL Modality

Central

Government

Executing and

Implementing


No. % No. % No. % No. % No. %

The novelty of RBL and issues related to

its newness 7 87.5 18 72.0 25 75.8 8 57.1 33 70.2

Delays in processing the preparation of

the operation 3 37.5 10 40.0 13 39.4 1 7.1 14 29.8

Exclusions of high-value contracts and

activities due to their potential adverse

impact on the environment and/or

affected people

0 0.0 1 4.0 1 3.0 4 28.6 5 10.6

Heaviness in conducting ADB assessments 1 12.5 5 20.0 6 18.2 4 28.6 10 21.3

Ability to include capacity-building

measures and focus on incentives a

0 0.0 3 21.4 3 6.4

Difficulties in applying ADB's Results-

based Lending Anti-Corruption Guidelines 1 12.5 1 4.0 2 6.1 2 14.3 4 8.5

Other b 0 0.0 1 4.0 1 3.0 2 14.3 3 6.4

Total 8 100.0 25 100.0 33 100.0 14 100.0 47 100.0


a Not included in the choices of challenges for central governments and executing and implementing agencies.

b Other challenges specified are: (i) difficulty in revising unrealistic targets identified at the implementation process (executing and implementing

agencies); (ii) lack of understanding of RBL among management and board (PTL); (iii) pressure to put a higher amount of disbursement on

outcome-level indicators (PTL); (iv) burden on project officer in verifying the results and addressing issues arising from bottlenecks to the result

achievement, which can be beyond one person's capacity or the program scope (PTL).


17. Appropriateness of the RBL Modality in Supporting the Country’s Sector Program

43. While a significant proportion of government respondents (64%) think that the RBL modality is

appropriate in supporting their country’s sector program, there are a few (18%) who think otherwise

(Table A6.35). On the other hand, 93% of PTLs believe that the RBL modality is suited for the country’s

sector program.

Table A6.35. Appropriateness of the RBL Modality for the Country’s Sector Program

Central

Government

Executing and

Implementing


No. % No. % No. % No. % No. %

More than

appropriate 1 12.5 5 20.0 6 18.2 4 28.6 10 21.3

Appropriate 7 87.5 14 56.0 21 63.6 9 64.3 30 63.8

Less than appropriate 0 0.0 4 16.0 4 12.1 1 7.1 5 10.6

Not at all appropriate 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0

Don't know 0 0.0 2 8.0 2 6.1 0 0.0 2 4.3

Total 8 100.0 25 100.0 33 100.0 14 100.0 47 100.0


PTL comment: Things are really changing as a result of RBL. After years of policy stagnation government is now addressing challenges.


18. Satisfaction Rating of the RBL Program

44. Overall, 85% of DMC clients are satisfied with the outcomes and benefits so far of the RBL

Program. Likewise, 79% of PTLs expressed overall satisfaction with the RBL program.


19

19. DMC Clients’ Satisfaction Rating of ADB’s Efforts and Performance

45. All DMC government respondents rated ADB’s RBL-related efforts and performance in providing

support to the country’s RBL Program, “satisfactory,” of which more than half (58%) gave a rating of

“very satisfactory” (Table A6.37).

Table A6.36. Satisfaction or Dissatisfaction of the RBL Program

Central Governments

Executing and

Implementing

Agencies

All DMC

Clients PTLs Total

No. % No. % No. % No. % No. %

Very satisfied 2 25.0 8 32.0 10 30.3 4 28.6 14 29.8

Somewhat satisfied 6 75.0 12 48.0 18 54.5 7 50.0 25 53.2

Neither satisfied nor

dissatisfied 0 0.0 3 12.0 3 9.1 1 7.1 4 8.5

Somewhat dissatisfied 0 0.0 0 0.0 0 0.0 1 7.1 1 2.1

Very dissatisfied 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0

Don’t know 0 0.0 2 8.0 2 6.1 1 7.1 3 6.4

Total 8 100.0 25 100.0 33 100.0 14 100.0 47 100.0


PTL Comment: “Somewhat dissatisfied”—There is no accountability, there is no buying in by the borrower, unless there are dedicated and

assigned individuals, the expected results can’t be achieved through the system.


Table A6.37. DMC Clients’ Satisfaction Rating of ADB’s RBL Efforts and Performance

Central Governments

Executing and

Implementing Agencies All DMC Clients

No. % No. % No. %

Very satisfactory 4 50.0 15 60.0 19 57.6

Somewhat satisfactory 4 50.0 8 32.0 12 36.4

Neither satisfactory nor

unsatisfactory 0 0.0 2 8.0 2 6.1

Somewhat unsatisfactory 0 0.0 0 0.0 0 0.0

Very unsatisfactory 0 0.0 0 0.0 0 0.0

Don't know 0 0.0 0 0.0 0 0.0

Total 8 100.0 25 100.0 33 100.0


Executing and implementing Agency comment: The efforts and performance of ADB in providing our country's RBL is very good indeed.

Thank you for these kind efforts.


20. Effectiveness of ADB support and Strengthening of its Own Capacity to Design and

Implement RBL programs

46. In strengthening its own capacity to design and implement RBL programs, only a few PTLs (14%)

believe that ADB has provided “extremely/very effective” support (Table A6.38). On the other hand, 21%

of PTL respondents consider ADB support to be “slightly/not effective” and more than half (64%) as

“moderately effective” indicating there is a need for improvement in building ADB staff capacity in terms

of designing and implementing RBL programs.


20

Table A6.38. Effectiveness of ADB Support and Strengthening of its Own Capacity

PTL

No. %

Extremely/very effective a 2 14.3

Moderately effective 9 64.3

Slightly/not effective 3 21.4

Don’t know 0 0.0

Total 14 100.0


a Comment: ADB is supporting staff well in this area.


Documents

LINKED DOCUMENT 6: STAKEHOLDERS’ PERCEPTIONSLINKED DOCUMENT 6: STAKEHOLDERS’ PERCEPTIONS A. Survey Objective and Methodology 1. The evaluation of the Asian Development Bank’s