Document of The World Bankdocuments.worldbank.org/curated/en/... · HERS Higher Education Reform Strategy HESP Higher Education Support Project IfR Interim unaudited Financial Report

Document of

The World Bank

FOR OFFICIAL USE ONLY

Report No: ICR00004049

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(IDA CREDIT-48860-PK PORTION B; IDA CREDIT 48870-PK PORTION A)

ON THE

CREDITS

IN THE AMOUNT OF

SDR70.5 MILLION (PORTION A OF THE CREDIT)

AND

SDR121.7 MILLION (PORTION B OF THE CREDIT)

(US$300 MILLION EQUIVALENT)

TO THE

Ministry of Finance

FOR A

PAKISTAN: TERTIARY EDUCATION SUPPORT PROJECT ( P118779 )

December 29, 2017

Education Global Practice

South Asia Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective September 18, 2017)

Currency Unit = Pakistan Rupee (PKR)

PKR 105.39 = US$1

US$1.42 = SDR 1

FISCAL YEAR

July 1 - June 30

Regional Vice President: Annette Dixon

Country Director: Patchamuthu Illangovan

Senior Global Practice Director: Jaime Saavedra Chanduvi

Practice Manager: Keiko Miwa

Task Team Leader(s): Scherezad Joya Monami Latif

ICR Main Contributor: Juan Prawda Witenberg

ABBREVIATIONS AND ACRONYMS

AC Affiliated Colleges

CBA Cost benefit analysis

CMS Campus Management Solution Module

CPS Country Partnership Strategy

DLIs Disbursement Linked Indicators

EEP Eligible expenditure program

EIRR Economic internal rate of return

ESMP Environmental and Social Management Plan

FM Financial management

GoP Government of Pakistan

HEC Higher Education Commission

HEIs Higher education institutions

HEMIS Higher Education Management Information Systems

HERS Higher Education Reform Strategy

HESP Higher Education Support Project

IfR Interim unaudited Financial Report

IPR Implementation progress report

ISR Implementation Status and Results Report

KPI Key performance indicators

M&E Monitoring and evaluation

MoF Ministry of Finance

MTDF-HE I Medium Term Development Framework for Higher Education 2005-2010

MTDF-HE II Medium Term Development Framework for Higher Education 2011-2015

MTDF-HE III Medium Term Development Framework for Higher Education 2016-2025

ORICs Offices of Research Innovation and Commercialization

PDO Project Development Objective

POM Project Operational Manual

QECs Quality Enhancement Cells

QER Quality Enhancement Review

RF Results framework

SLP Students Loan Program

TA Technical assistance

TEIs Tertiary education institutions

TESP Tertiary Education Support Project

TTL Task Team Leader

TTS Tenure track system

WB World Bank

TABLE OF CONTENTS

DATA SHEET .......................................................................................................................... 1

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5

A. CONTEXT AT APPRAISAL ......................................................................................................5

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ........................................................... 12

II. OUTCOME .................................................................................................................... 14

A. RELEVANCE OF PDOs ......................................................................................................... 14

B. ACHIEVEMENT OF PDOS (EFFICACY) ................................................................................... 15

C. EFFICIENCY ........................................................................................................................ 17

D. JUSTIFICATION OF OVERALL OUTCOME RATING ................................................................. 18

E. OTHER OUTCOMES AND IMPACTS (IF ANY) ........................................................................ 18

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 19

A. KEY FACTORS DURING PREPARATION ................................................................................ 19

B. KEY FACTORS DURING IMPLEMENTATION .......................................................................... 20

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 24

A. QUALITY OF MONITORING AND EVALUATION (M&E) ......................................................... 24

B. ENVIRONMENTAL, SOCIAL AND FIDUCIARY COMPLIANCE .................................................. 26

C. BANK PERFORMANCE ........................................................................................................ 27

D. RISK TO DEVELOPMENT OUTCOME .................................................................................... 28

V. LESSONS AND RECOMMENDATIONS ............................................................................. 29

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 31

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 43

ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 45

ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 46

ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 49

ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 50

ANNEX 7. ASSESSMENT OF APPROPRIATENESS OF PDO AND INTERMEDIATE INDICATORS .. 51

ANNEX 8. EXECUTIVE SUMMARY OF THE COMPLETION PROJECT REPORT FROM THE GOVERNMENT ..................................................................................................................... 53

The World Bank Pakistan: Tertiary Education Support Project ( P118779 )

Page 1 of 54

DATA SHEET

BASIC INFORMATION

Product Information

Project ID Project Name

P118779 PAKISTAN: TERTIARY EDUCATION SUPPORT PROJECT (

P118779 )

Country Financing Instrument

Pakistan Specific Investment Loan

Original EA Category Revised EA Category

Partial Assessment (B) Partial Assessment (B)

Organizations

Borrower Implementing Agency

Ministry of Finance Higher Education Commission

Project Development Objective (PDO) Original PDO

The development objective of the proposed Project is to improve the conditions of teaching, learning and research for enhanced access, quality and relevance of tertiary education.


Page 2 of 54

FINANCING

Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$)

World Bank Financing IDA-48870

110,000,000 110,000,000 107,199,051

IDA-48860

190,000,000 114,347,172 104,387,463

Total 300,000,000 224,347,172 211,586,514

Non-World Bank Financing

Borrower 1,715,000,000 0 0

Total 1,715,000,000 0 0

Total Project Cost 2,015,000,000 224,347,172 211,586,515

KEY DATES

Approval Effectiveness MTR Review Original Closing Actual Closing

24-Mar-2011 30-Nov-2011 17-Jun-2013 31-Dec-2015 30-Jun-2017

RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s) Amount Disbursed (US$M) Key Revisions

11-Dec-2013 78.06 Change in Results Framework Change in Components and Cost Cancellation of Financing Change in Financing Plan Reallocation between Disbursement Categories Change in Legal Covenants Change in Implementation Schedule

17-Dec-2015 186.75 Change in Loan Closing Date(s)

KEY RATINGS

Outcome Bank Performance M&E Quality

Satisfactory Satisfactory Substantial


Page 3 of 54

RATINGS OF PROJECT PERFORMANCE IN ISRs

No. Date ISR Archived DO Rating IP Rating Actual

Disbursements (US$M)

01 21-Sep-2011 Moderately

Unsatisfactory Moderately Unsatisfactory 0

02 28-Apr-2012 Moderately Satisfactory Moderately Satisfactory 0

03 16-Dec-2012 Moderately Satisfactory Moderately Unsatisfactory 32.59

04 23-Jun-2013 Moderately

Unsatisfactory Unsatisfactory 34.93

05 08-Jan-2014 Moderately

Unsatisfactory Unsatisfactory 51.24

06 19-Jun-2014 Moderately Satisfactory Moderately Satisfactory 89.30

07 24-Dec-2014 Moderately Satisfactory Moderately Satisfactory 94.74


09 08-Dec-2015 Moderately Satisfactory Moderately Satisfactory 121.93


11 18-Jan-2017 Satisfactory Moderately Satisfactory 138.23

SECTORS AND THEMES

Sectors

Major Sector/Sector (%)

Education 100

Tertiary Education 100

Themes

Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 110

Jobs 100

Public Private Partnerships 10


Page 4 of 54

Public Sector Management 20

Public Finance Management 10

Public Expenditure Management 10

Public Administration 10

Transparency, Accountability and Good Governance

10

Human Development and Gender 80

Education 80

Access to Education 20

Science and Technology 20

Teachers 20

Standards, Curriculum and Textbooks 20

ADM STAFF

Role At Approval At ICR

Regional Vice President: Isabel M. Guerrero Annette Dixon

Country Director: Rachid Benmessaoud Patchamuthu Illangovan

Senior Global Practice Director: Michal J. Rutkowski Jaime Saavedra Chanduvi

Practice Manager: Amit Dar Keiko Miwa

Task Team Leader(s): Halil Dundar Scherezad Joya Monami Latif

ICR Contributing Author: Juan Prawda Witenberg


Note on Page 3: Actual disbursement figures presented under “Ratings of project performance in ISRs” are in Special Drawing Rights (SDR).

Page 5 of 54

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

A. CONTEXT AT APPRAISAL

Context

1. Pakistan’s broad landscape. At the time of appraisal in late 2010, Pakistan was the sixth most populous country in the world with a population of approximately 165 million people, with the highest population growth among South Asia countries (2.1 percent per annum) and the second youngest population. Despite significant economic progress from 2000 to 2007, Pakistan’s economy had been facing a major economic crisis since 2008 and continued to be one of the poorest countries in the world with a per capita income of less than US$950 (2008).

2. Pakistan lagged far behind neighboring countries in many aspects of education, ranking 125th out of 169 countries on the education index of the UNDP Human Development Report (2010). The Net Enrollment Rate for primary school in 2008/09 for example, was 66 percent, substantially lower than Bangladesh (88 percent) and India (89 percent). Despite significant achievements, Pakistan’s tertiary education faced major challenges, affecting its competitiveness and economic growth. For example, in the area of higher education and training, Pakistan ranked 118th out of 133 economies on the Global Competitiveness Index 2009-2010.

3. The following paragraphs summarize the five key issues addressed by the Pakistan Tertiary Education Support Project (TESP). These tertiary education issues, curtailed the country’s intellectual and human capital development and slowed Pakistan’s transition to a middle-income country in the global knowledge economy of the 21st century.

4. Low and inequitable participation in tertiary education. Between 2002 and 2009, total enrollment grew at an average annual rate of about 17 percent, increasing from approximately 276,000 students in 2002 to 803,000 in 2009, excluding enrollments in affiliated colleges (AC) and private/external degree programs, still lagging behind the rates in neighboring countries and significantly behind those of middle-income countries. Much of the growth in enrollment originated from distance and private education sub-sectors. While 17.3 percent of the college age population (17-23 years old) belonging to the highest income group was enrolled in tertiary education, only 0.4 of the poorest 20 percent of this same age cohort participated in tertiary education. There were also large geographical differences in enrollment ranging from 2.3 percent in Baluchistan to 6 percent in Punjab and Sindh. The gross enrollment rate in urban areas of 8.8 percent was twice the one in rural areas (3.8 percent).

5. Poor quality of teaching and research. Significant institutional reforms together with investments had taken place to improve the quality and relevance of tertiary education but these reforms had not translated into improvements in the quality of teaching, learning and research except in a few universities. There was a shortage of qualified faculty, combined with a lack of incentives and adequate accountability mechanisms. Most universities were not producing an adequate number of graduate students. Curricula


Page 6 of 54

had remained unchanged for decades, oblivious to the changes happening outside of the academic world. Pakistani universities on the whole except for a few outliers, have had a long history of diminishing teaching quality, lack of updated library and computer facilities and access to the Internet, and outdated textbooks and other reading materials.

6. Weak research performance and relevance. Despite the efforts of the Government of Pakistan (GoP) to increase research productivity by improving research conditions and providing incentives for faculty, the production of PhDs by the Pakistani Tertiary Education Institutions (TEIs) had been strikingly low, making the basis for strong graduate programs and research production very fragile. There was little applied and/or basic research undertaken in the sciences and engineering programs. Research was, by and large, absent in higher learning. Pakistan’s research performance lagged behind comparable countries as measured by the number of publications in high impact journals.

7. Low levels of expenditure. Notwithstanding the GoP’s commitment to the development of the education sector and the increase in tertiary education budget as measured by its share of GDP (0.27 percent in 2000 to 0.45 in 2008), given the increase in student enrollment, and as compared to other countries in the region, tertiary education in Pakistan was still significantly underfunded at the time of appraisal. In addition, the lack of adequate releases of funds (from the Ministry of Finance to the Higher Education Commission (HEC)) further affected the successful implementation of many of the reform and development initiatives undertaken by the HEC. Thus, the rapid expansion of the tertiary education system had left a considerable financing gap between available funds and the anticipated cost of investments needed to implement the planned reforms. Over 90 percent of the recurrent budget allocated by the GoP was spent on staff salaries, overcrowding non-salary items like the ones targeted to finance quality improvement interventions and support effective teaching, learning and research. There was limited scope for further revenue mobilization through non-budgetary sources.

8. Inadequate institutional governance and management. Despite improvements in sector-wide governance and management following the launch of the GoP’s overall program in the early 2000s, at the time of appraisal there was still inadequate institutional governance and management, including weak Higher Education Management Information Systems (HEMIS), hampering long-term sector and institutional planning, monitoring, and accountability. First, there was insufficient institutional capacity for planning, policy analysis and monitoring and evaluation (M&E) at the HEC, the autonomous institution established in 2002 empowered to foster badly needed changes in tertiary education. For example, HEC: (i) lacked a long-term strategic perspective and agreed targets against which to monitor progress; (ii) displayed a largely dysfunctional internal management structure; (iii) lacked a national accreditation and quality assessment process and thus, an internal evaluation system at higher education institutions (HEIs); and (iv) lacked a performance evaluation scheme. Second, HEC was not completely immune to changes in the political landscape and its autonomy was not fully guaranteed. Third, HEC lacked the communication strategy and resources required to reach out to higher education stakeholders and the public at large.

9. GoP’s Higher Education Reform Strategy (HERS). To address the above-mentioned issues and improve the quality of tertiary education attuned to the needs of the 21st century, the GoP developed the Medium-Term Development Framework for Higher Education 2005-2010 (MTDF-HE I) aiming at: (i) increasing access to tertiary education; (ii) improving the quality and relevance of learning and research;


Page 7 of 54

and (iii) strengthening governance and management. The HERS included supply-side interventions such as, for example, faculty development and recruitment, infrastructure expansion and upgrading, curriculum revision, restructuring of specific academic programs, quality assurance, competitive research funding and capacity development activities, as well as demand-side measures. 10. The World Bank (WB) supported the implementation of the MTDF-HE I through the Higher Education Support Project (HESP, 2009) in the amount of US$100 million, using a Development Policy Credit. Despite initial progress in the implementation of the actions undertaken under the MTDF-HE (I), there were still a large and unfinished reform agenda, and the gains made in the last few years were not sufficiently consolidated. Recognizing this, the GoP drafted the new MTDF-HE II for the period 2010-2015 reflecting accomplishments and adjustments to its priorities and providing a vision for the sector’s medium-term development. The TESP was a response to a specific request by the GoP for continued financial and technical support to the implementation of the MTDF-HE II. It bears mentioning that the TESP was the first results based loan in higher education for the Bank, a lending modality that had sufficient ownership by the HEC and the government and was meant to initiate a results focus in the achievement of the HERS.

Theory of Change (Results Chain)

11. Table 1 (next page) illustrates the proposed TESP Results Chain linking the operation’s interventions (activities) grouped in sub-components to their outputs (intermediate results) and these, to the desired short (PDOs) and long-term (out of the scope of TESP) outcomes.


Page 8 of 54

Table 1. TESP Theory of Change (Results Chain)

PDO

Improved

conditions of teaching,

learning and research under

a context of enhanced

access, quality and relevance

of tertiary education

Outcome 1.1 Improved sector planning, fiscal sustainability and effectiveness of public expenditures

1. Recurrent and development funds released in a timely manner as per the medium term DLI budgetary framework

Activities -Implementation of MTBF -development of monitoring protocol for funds release -refinement of resource allocation formula used by HEC for university fund allocation --diversification of sources of funding --timely allocations of HEC recurrent grant to universities

Outcome 1.2 Enhanced Quality and Relevance of Teaching, Learning and Research

DLI 2. Indigenous post graduate scholarships awarded

Activities -scholarship program to increase supply of qualified faculty -Implement tenure track system -interim faculty placement program for returning scholars -implement faculty development program -develop and implement minimum quality standards for ACs -implement set of interventions to enhance quality education at MA level -Establish ORICs and ORIC quality standards

DLI 3. Increased number of faculty members on tenure track system

DLI 4. Increased number affiliated colleges meeting quality standards

DLI 5. Provision of enhanced quality education at MA level for external students

DLI 6. Increased no. of ORICs established and performing satisfactorily students

Outcome 1.3

Improved Equitable Access to Tertiary Education

DLI 7. Increased no. of students benefiting from financial aid program

Outcome 1.4

Strengthened Management and Governance

DLI 8. Increased no. of QECs performing satisfactorily

DLI 9. Increased no. of HEIs assessed against evaluation standards DLI 10. Improved strategic management and planning in public HEIS

Activities -implement financial aid program -expansion and upgrading of existing physical facilities --scaling up distance education facilities in public and private universities --Enhance HEIs PPPs for private provision of higher education

Activities -strengthen internal governance and mgt of HEIs --systematic monitoring of QECs, extension of accreditation councils --develop and utilize HEMIS --strengthen fiduciary control systems --improve communication capacity


Page 9 of 54

12. The above table illustrates that ensuring a conducive environment for the achievement of outcomes sought by the Project was necessary. The importance of a strategic vision as provided by the MDTF HE II however cannot be discounted either. The conducive conditions are specifically reflected in TESP by an: (i) enhanced program financing at the tertiary education system (Component 1); and (ii) improved capacity building policy design and monitoring and evaluation (Component 2). The expected outcomes and end-of-project targets for each one of these two components are described in table 1 above. Project Development Objectives (PDOs) 13. The PDOs were to improve the conditions of teaching, learning and research for enhanced access, quality and relevance of tertiary education. 14. TESP’s main beneficiaries included all tertiary education students, academic staff (including newly trained faculty through in-country or foreign scholarship programs), and administrators, benefiting from improved conditions of the teaching, learning and research environment in both public and private tertiary education institutions across the country. A total of about 1.2 million students and approximately 20,000 academic staff in 73 public and 24 private universities and about 800 ACs across the country were expected to benefit from specific interventions under TESP. A total of 18 private tertiary education institutions (out of 57) were initially also eligible to receive funding from TESP with the intention of increasing the number of beneficiary private institutions to 24 by the closing date.

Key expected outcomes and outcome indicators 15. Progress towards meeting these objectives were to be assessed by the following four outcome indicators with its respective end-of-project targets as compared to a baseline: For outcome 1: enhanced program financing at the tertiary education system:

a. Number of regular students in public and private HEIs (to be increased from a baseline of 463,803

to 604,803); b. Number of faculty members on the Tenure Track System (TTS) (to be increased from a baseline of

849 to 2,350); and c. Number of Quality Enhancement Cells (QECs) performing satisfactorily as measured by scorecards

(to be increased from a baseline of 0 to 15).

For outcome 2: improved capacity building policy design and monitoring and evaluation:

d. Number of Offices of Research Innovation and Commercialization (ORICs) performing satisfactorily as measured by scorecards (to be increased from a baseline of 0 to 45).


Page 10 of 54

Components 16. The above-mentioned PDOs were to be achieved through the implementation of the following 2 components and 6 sub-components: a. Component 1 - Program Financing (total US$1.995 billion1; IDA: US$280 million) supporting the

four pillars of the MTDF-HE II; and b. Component 2 - Capacity Building, Policy Design and M&E (total US$20 million; IDA: US$20 million]. 17. Component 1 was the results-based part of the Credit (IDA-48860) disbursing against the achievement of 10 agreed Disbursement-Linked Indicators (DLIs)2 and included the following four subcomponents: a. Sub-component 1.1 - Improved fiscal sustainability and expenditure effectiveness aimed at

supporting: (i) the implementation of the medium-term budgetary framework for higher education MTBF-HE; and (ii) ensuring the development and monitoring of a protocol for the timely release and use of funds. It also aimed at supporting the: (iii) refinement of the resource allocation formula used by HEC to determine the amount of funds to be allocated to universities; and (ii) diversification of sources of funding, including self-generating income. The eligible expenditure program (EEP) for this subcomponent included the university recurrent grant allocated by HEC to universities.

b. Sub-component 1.2 - Enhanced quality and relevance of teaching and research aimed at: (i) supporting the increase of the supply of new qualified faculty and the skills improvement of faculty in service; (ii) improving the quality and relevance of courses and programs delivered by universities and degree awarding institutions and the quality of the services provided to ACs and external students3; and (iii) promoting research and its applications4. Accordingly, this sub-component financed the: (i) continuation of the scholarship program to increase the supply of

1 Disbursements under component 1 were made against two main features: (i) selected (eligible) key budget line items of the HEC annual approved budget referred to as EEP, up to capped absolute amounts (constituting the counterpart funding amounting to about US$1,715 million); and (ii) Credit disbursement based on the third party verified achievements of pre-specified results as measured by 10 DLIs (IDA funding amounting to about US$280 million). 2 A procurement plan was not required for component 1 since its EEP relate to non-procurable expenditures like salaries, allowances, scholarships, research and recurrent grants to HEIs provided as institutional block grants based on a formula funding. 3 Through: (a) curriculum revision and full scale adoption of the revised curricula of universities; (b) design and implementation of a comprehensive strategy aimed at providing ACs with quality assurance support by universities in order to enhance academic performance; (c) design of a policy mostly based on distance learning mode, to improve the quality of programs delivered to external (private) students and pilot implementation of the policy in six public universities having large numbers of external students; and (d) scaling up the examination systems in HEIs, notably through standardization measures. 4 Through: (i) the increase in the supply of funds available on a competitive basis for research in priority areas; and (ii) strengthening the relevance and value added of research undertaken in HEIs, including the commercialization of the outputs of academic research through, inter alia, the establishment of ORICs to liaise with the industry and to facilitate the development of technology incubators at selected HEIs.


Page 11 of 54

qualified faculty staff, including the gradual replacement of foreign scholarships by indigenous scholarships programs associated with split PhD programs; (ii) completion of the interim faculty placement program for returning PhD scholars in parallel with the PhD scholarship program; (iii) strengthening of the performance-based TTS to ensure the steady improvement in the quality of academic staff; and (iv) implementation of the Faculty Development Program to improve pedagogical skills and subject knowledge of existing faculty staff through short-term professional development courses. EEPs for this subcomponent included: (i) PhD scholarships; (b) tenure track faculty salaries; (c) research grants (e.g., National Research Program for Universities); and (d) development of the Pakistan Education and Research Network.

c. Sub-component 1.3 - Improved equitable access to tertiary education aiming at supporting the

GoP’s program to expand access to tertiary education and reduce inequalities resulting from disparities in students’ socioeconomic and regional background. The key interventions supported by this subcomponent included: (i) selective expansion and upgrading of existing physical facilities; (ii) scaling up of distance education facilities in public and private universities; (iii) design and implementation of financial aid programs to ensure equity of access to HEIs5; and (iv) enhancement of the scale and scope of private-public partnerships in HEIs in view of promoting further private provision of higher education services. There are no EEP associated with this subcomponent.

d. Sub-component 1.4 - Strengthened governance and management at the national and institutional

levels through: (i) streamlining and strengthening the internal governance and management of HEIs; (ii) institutionalizing quality assurance mechanisms through the reinforcement of the quality assurance mechanism, including the systematic monitoring of the QECs at the university level and through the extension of accreditation councils to new professional areas; (iii) scaling up of the development of a comprehensive, reliable and timely HEMIS to allow information-based management of HEIs; (iv) strengthening the fiduciary control systems focusing on both procurement and financial management reforms, especially at institutional levels; and (v) strengthening the implementation capacity of HEC, with a focus on the implementation of a communication strategy.

18. Component 2 aimed at strengthening capacity for program implementation and M&E systems and compliance with the DLIs through the financing of essential technical assistance (TA) and capacity building activities clustered in the following two sub-components: a. Sub-component 2.1 capacity building of HEC and HEIs aiming at supporting capacity building at

the HEC and participating HEIs in the areas of financial management (FM) and procurement, MTDF implementation, M&E (including sectoral HEMIS), program coordination, sector planning, as well as communication; and

b. Sub-component 2.2 policy design and TA supporting: (i) the design and implementation of selected

policies directly linked to the first component, especially in the area of expenditures effectiveness,

5 Through (a) expansion of needs-based scholarships to remove financial obstacles for meritorious students from low-income families in public and private universities; and (b) development and implementation of a Students Loan Program (SLP) to allow students to reduce the costs of attending a higher education institution


Page 12 of 54

resource diversification and student aid (e.g., SLP); (ii) design and development of policies in areas where little technical or strategic work had been undertaken to date, including research performance and labor market outcomes; and (iii) M&E activities, including third party validation studies. A limited amount of funds was to be allocated for the provision of office equipment and furniture for HEC, as well as for unexpected analytical studies and training activities.

19. Table 2 displays the original ten agreed DLIs by sub-component under component 1.

Sub-component DLIs

SC 1.1: Improved fiscal sustainability and effectiveness of expenditures

DLI 1: Allocation and timely release of recurrent development funds to HEC as per agreed MTBF-HE

SC 1.2: Improved quality and relevance of teaching, learning and research

DLI 2: Number of new indigenous postgraduate scholarships awarded

DLI 3: Number of faculty members on the Tenure Track System

DLI 4: Number of affiliated colleges meeting the minimum quality standards set by HEC

DLI 5: Provision of enhanced quality education at MA level for external students

DLI 6: Number of ORICs established and performing satisfactorily as measured by scorecards

SC 1.3: Improved equitable access to tertiary education

DLI 7: Number of students benefiting from financial aid program, including a needs-based scholarship program under a student loan program

SC 1.4: Strengthened management and governance

DLI 8: Number of quality enhancement cells (QECs) performing satisfactorily as measured by scorecards

DLI 9: Number of HEIs assessed against institutional performance evaluation standards

DLI 10: Improved strategic management, planning and accountability in public HEIs

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDOs and Outcome Targets 20. The original PDOs and the end-of-project outcome targets were not modified during the December 2013 TESP’s restructuring. Revised PDO indicators

21. Not applicable because the PDO and the end-of-project outcome targets remained unchanged during the restructuring.


Page 13 of 54

Revised Components

Table 3. Changes made to the Intermediate Outcome Indicators (IOI) at the time of restructuring:6

22. The two components were not modified during the restructuring however, some of their DLI targets were scaled down, their metric further refined, their costs decreased and their implementation schedule modified as further explained in Section II-B and more specific in Table 3 above. Accordingly, during the restructuring: (i) DLI 1 was dropped; (ii) DLIs 4 and 7 fine-tuned their metric; and (iii) DLI 4, DLI 9 and the key performance indicator (KPI 7) decreased their original targets.

6 The following changes were made in the DLIs at the time of the restructuring: a) DLI 1 for years 2 and 3 were canceled due to non-achievement; b) DLI 4 was modified from “number of ACs colleges maintaining minimum quality standards set by HEC” to

“HEC has finalized, adopted and implemented a capacity building plan for the ACs assessed during year 1 to meet the selected minimum quality standard”

c) The SLP of the original DLI 7 was dropped due to limited progress in its operationalization; and d) The number of public HEIs under DLI 9 was decreased from 30 to 13 for year 2 due to logistical constraints

and security concerns and lack of qualified assessors in the higher education sector, and the one for year 3 was dropped for the same reasons.

Original statement Original end of project target

Restructured statement of the metric

Restructured end of project target

Allocation and timely release of recurrent and development funds to HEC as per agreed MTBF-HE (IOI 5 and DLI 1)

Timely release

Deleted

Number of affiliated colleges meeting minimum quality standards set by HEC (IOI 12 and DLI 4)

800

Number of assessed affiliated colleges with successfully complete capacity building activities under HEC – implemented Capacity Building Plan for Assessed Affiliated Colleges

102

Number of administrators in affiliated colleges participating in management training (IOI 7)

750

102

Number of students benefiting from financial aid program, including a needs-based scholarship program under a student loan program (IOI 15 and DLI 7)

6,000

Number of students benefiting from financial aid program, including a needs-based scholarship program (DLI 7)

6,000

Number of HEIs assessed against Institutional Performance Evaluation Standards (IOI 18 and DLI 9)

60

13


Page 14 of 54

Other significant changes 23. At the request of the GoP and based on the factors described in paragraph 74: (i) the TESP was restructured in December 2013; (ii) US$77.82 million (SDR 49.9 million) of the original Credit amount of US$300 million were canceled7; and (iii) the WB agreed at the end of 2015 to extend the closing date by 18 months from January 1, 2016 to June 30, 2017. Rationale for changes and their implication in the original Theory of Change. 24. Two factors, both force majeure but critical to a smooth project implementation period hounded the Project from the time of Board approval to the project’s restructuring in end 2013. These are described in more detail further in the ICR but they are important to note in setting the context of the restructuring. One, was the fiscal difficulties faced by the Government of Pakistan due to natural calamities in the form of devastating floods; and, the other is the passing of the 18 Amendment by Parliament which overnight led to the question which continues, on whether to devolve or not to devolve the provision of higher education in Pakistan. The first issue meant there were insufficient fund releases made to the HEC to implement the project because the government simply did not have the funds and the second, a question to the HEC’s authority and its mandate to administer reforms to higher education institutions country wide. 25. TESP was restructured in December 2013 at the request of the GoP grounded on the following two factors: (i) there was a need to focus on unmet but achievable DLIs; and (ii) the slow implementation progress during the first two years since Credit effectiveness as explained in Section III-B. Under the restructuring, TESP was restructured down to focus on achievable results with a view to meeting HEC’s medium term goals in the sector. TESP’s implementation jump-started after the restructuring. The changes made did not affect TESP’s Theory of Change.

7 US$33.9 million from non-achievement of DLI 1, US$12.72 million from DLI 4, US$12.72 million from DLI 7, US$8.48 million from DLI 9 and US$10 million from component 2.

II. OUTCOME

A. RELEVANCE OF PDOs Assessment of relevance of PDOs and ratings 26. Relevance of objectives is high. At the lending stage, TESP PDOs and design was consistent with the Country Partnership Strategy (CPS) for Pakistan (FY10-13). The CPS supported the GoP’s overall vision, embodied in its Second Poverty Reduction Strategy Paper concerning stimulating growth in the production sector, creating adequate employment opportunities, improving income distribution, and harnessing the country’s economic competitiveness through economic liberalization, deregulation and transparent privatization. The CPS aimed at contributing to the improvement of: (i) economic governance; (ii) human development and social protection; (iii) infrastructure to support growth; and (iv) security and conflict-reduction.


Page 15 of 54

8 CPS Pakistan 2015-2019; Report No. 84645-PK, April 4, 2014.

27. In line with the GoP’s growth and competitiveness objectives, the TESP aimed at supporting Pakistan’s drive to improve human development, including a broad spectrum of strategic tertiary education reforms with an enhanced focus on results. Accordingly, TESP was aligned with the CPS priorities to prepare a skilled workforce needed to spearhead the country’s growth ambitions. 28. At the closing of the project TESP PDOs and design remained consistent with the CPS for Pakistan8 (2015-2020). The TESP is aligned with the overarching goal of the Pakistan CPS which is to help the country accelerate poverty reduction and build shared prosperity. The fourth strategic result - service delivery – and fourth theme within service delivery - increased school enrollment and adoption of education quality assessment are particularly aligned with the TESP’s tertiary education focus and the project’s its development objective.

29. In addition, the PDOs were appropriately aligned with the GoP’s medium and long-term higher education vision and strategy. The PDOs were stated with sufficient operational precision allowing for gauging their attainment through the observation and measurement of a metric comprising four outcomes. Baseline data was included at the onset of project implementation and end-of-project outcome and intermediate targets were realistic apart from 3 that were decreased at restructuring. The operational architecture of the design consisting of 2 components and 6 sub-components took the required holistic approach.

B. ACHIEVEMENT OF PDOS (EFFICACY) Assessment of achievement of each objective/outcome 30. Achievement of the PDOs rated Substantial. The following assessment of the PDOs has been organized around the two key outcomes/objectives triggering the improvement of the conditions for teaching, learning and research leading to enhanced access, quality and relevance of tertiary education as further detailed in Annex 1 of this ICR. 31. With respect to the first PDO outcome – enhanced program financing (component 1) (See Annex 1 for details): (i) The number of regular students in public private HEIs at the end of the project (867,562)

exceeded by 43.4 percent the original end-of-project target (604,803); (ii) The number of faculty members on the TTS at the end-of the project (3,838) exceeded by 78.6

percent the original end-of-project target (2,149); and (iii) The number of ORICs performing satisfactorily as measured by scorecards at the end of the

project (41) exceeded its original end-of-project target (15) by 173.3 percent. 44 ORICs were established under TESP, but only 41 performed satisfactorily as confirmed by third party validation.

32. With respect to the second PDO outcome – improved capacity building policy design and monitoring and evaluation (component 2) the number of QECs performing satisfactorily at the end of


Page 16 of 54

the project as measured by scorecards (90) exceeded in 100 percent the end-of-project target of 45. 147 QECs were established under TESP, but only 90 performed satisfactorily as confirmed by third party validation [see outcome indicator 1 in Annex 1 of the ICR]. 33. As fully detailed in Section III-B below, all activities were completed by end-of-project. This was despite a slow start to project implementation. Once certain DLIs with unrealistic target (like DLI4, DLI9 and KPI7) had been realistically downsized (see table 3 above), another two (DLI 4 and DLI 7) had their metrics fine-tuned (see table 3 above) and another, for reasons outside the control of the project (DLI 1), canceled, the project gathered the traction with a more focused approach to achieve results in the given project period. 34. In addition to completing all activities, meeting or exceeding in some cases the end-of-project targets, total IDA documented disbursement reached 74 percent of the total appraised cost or 100 percent of the estimated total after the cancelation at restructuring. 35. As explained in Section IV-A and in Annex 1, there is a significant relevance of the four outcome indicators in contributing to the setting of the appropriate environment for improved teaching, learning and research (outcome indicators 1, 2 and 4) and enhanced access to tertiary education (indicator 3). 36. As per Annex 1: (a) 11 intermediate outcome indicators (6, 8, 9, 10, 11, 13, 15, 17, 20, 21 and 22) exceeded or achieved their original end-of-project target set at appraisal; (b) other 3 indicators (7, 12 and 18) achieved or exceeded their restructured end-of-project targets; (c) 3 indicators (14, 16 and 19) fell short; and (d) one indicator (5) was eliminated at the restructuring. 37. Contribution to the higher-level objectives. TESP aimed at supporting Pakistan’s drive to improve human development, including a broad spectrum of strategic tertiary education reforms with an enhanced focused on results. This higher-level objective statement was explicitly translated in the results chain of table 1 above into the following three long-term outcomes (outside the scope of TESP but within the larger MDTF HE II): (a) Increased access to tertiary education. As per outcome indicator #3 in Annex 1, enrollment of regular students in public and private HEIs shows an increasing trend. Furthermore, the increased number of new postgraduate foreign scholarships (indicator #14 in Annex 1), the increased number of student benefiting from financial aid (indicator #15 in Annex 1) and the increased number of new postgraduate indigenous scholarships awarded (indicator #21 in Annex 1) seems to strongly suggest that there is a promising ongoing path to achieve this first long-term outcome; (b) Improved quality and relevance of learning and research. As per outcome indicators #2 and #4 in Annex 1, an increased number of faculty members in the TTS and a larger number of ORICs performing satisfactorily as measured by scorecards and verified by a qualifying third party are necessary foundations leading to the achievement of this second long-term outcome. Furthermore, the increased budgetary allocation of HEC as percentage of GDP (indicator #6 in Annex1), the increased share of universities’ self-generated revenue in overall total resources (indicator #9 in Annex 1), the increased number of technology companies established and incubated in HEIs (indicator #8 in Annex


Page 17 of 54

1), the increased number of revised and restructured undergraduate and/or graduate programs in HEI (indicator #11 in Annex 1), the increased number of academic programs accredited by Accreditation Councils (indicator #17 in Annex 1), just to mention few, seems to strongly support the achievement of this second long-term outcome; and (c) Strengthened governance and management of tertiary education. As per outcome indicator #1 in Annex 1, the increased number of QECs performing as measured by scorecards and verified by a third qualifying party, in conjunction with other KPIs in Annex 1 that have shown a growing trend like #7, #12 and #18, just to mention some, will most certainly contribute to the achievement of this third long-term outcome. Justification of overall efficacy rating 38. Overall efficacy is rated Substantial. Improving the relevance and quality of higher education attuned to the requirements of the 21st century involves two interdependent, and to certain degree, broad sequential processes:

(i) A conducive environment for the improvement of relevance and quality of higher education, which was the primary objective of TESP and to a large extent that of the two previous related operations - HESP/MTDF-HE I 2005-2010 and MTDF-HE II 2011-2015; and

(ii) The project took a holistic approach to improving the quality and relevance of tertiary education while increasing equitable access and strengthening governance and management of service delivery on the basis of the MDTF HE II. Subcomponent linkages were an important part of the TESP which was designed to reform the sector as a whole instead of supporting piece meal and ad hoc interventions. Each sub component was designed to include a set of priority programs and interventions, which were also linked to the other subcomponents, and which were implemented at different levels of institutions—public and private universities and Affiliated Colleges (the latter, provincial subjects). This is what the new tertiary education operation in Pakistan (MTDF-HE III 2016-2025), currently under preparation, is likely to also aim for because this aspect of the design was laudable.

39. As mentioned in earlier paragraphs, TESP put into motion the process that positively impacted the setting of a conducive environment for improved teaching, learning and research and to a certain extent, also contributed to the achievement of the higher long-term outcomes explained above. Interventions such as ORICs, score cards validated by a Third Party and improvement plans for Affiliated Colleges as discussed above (paragraph 37 (b)), all were major contributions and set up building blocks for longer term improvement of the sector. For this achievement despite the external bottlenecks faced by the project, the ICR justifies the substantial efficacy rating.

C. EFFICIENCY

Assessment of efficiency and rating

40. The overall efficiency of TESP was Substantial. TESP had an economic internal rate of return (EIRR) of 14 percent indicating that the project was economically viable (See Annex 4 for details).


Page 18 of 54

9 This rate of return is comparable to the findings of Montenegro & Patrinos (2014) who report a rate of return to higher education in Pakistan as 15 percent.

41. Economic Internal Rate of Return. A cost-benefit analysis (CBA) was conducted at the time of the ICR, which reflects a positive rate of return for the project. Given that a cost-benefit analysis was not conducted at the time of appraisal, there is no way to compare the expected rate of return to the time of appraisal and project completion. The project benefits were attributed to: (a) increased number of college and university graduates (resulting in increased employment); and (b) a higher wage premium for college and university graduates (due to increased productivity as a result of improved quality of tertiary education). Costs included public expenditures on tertiary education (which includes both recurrent and development expenditure incurred by HEC) and household private expenditures on higher education as well as the opportunity cost of higher education, in the form of forgone wages. Overall the project’s EIRR of 14 percent reflects that the project was efficient in ensuring that economic benefits of the project exceeded the economic costs (refer to annex 4 for more details).9

D. JUSTIFICATION OF OVERALL OUTCOME RATING

42. The ICR rates the overall outcome as Satisfactory, because:

a. The project objectives were rated highly relevant and they were achieved or in some cases exceeded;

b. Efficacy was rated as substantial as the conducive environment for teaching, learning and conducting research for enhanced access, quality and relevance of tertiary education was improved; and

c. The overall efficiency of TESP displaying an estimated EIRR of 14 percent was substantial. E. OTHER OUTCOMES AND IMPACTS (IF ANY)

Gender 43. Not applicable as no gender explicit interventions were included in the TESP.

Institutional Strengthening 44. TESP, following the gains of HESP (2009)/MDTF-HE I, has been bringing about an important, albeit gradual, institutional change process in the Pakistani tertiary education sector. The project brought about a behavioral and management change from an input/process-oriented business as usual, where recurrent and developmental financial resources (grants) to the HEIs was transferred without major accountability, to an outcome-oriented business modality where the allocation of resources was contingent on accountability and results. This shift in performance based budgeting to a certain extent has been characteristic from 2004 onwards. 45. The following are examples of the institutional transforming seeds supported by TESP: (i) the TTS; (ii) the establishment of performance standards for HEIs; (iii) the establishment of QECs and ORICs performing against standards (scorecards); (iv) the accreditation of HEIs programs; (v) the capacity building in the ACs; and (vi) the benchmarking of the hybrid distance education programs.


Page 19 of 54

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME

A. KEY FACTORS DURING PREPARATION

49. Strengths of preparation. First, the lending phase benefited from the findings of an assessment carried out on HESP/MTDF-HE I 2005-2010, a precursor of the project, whose main lessons learned were incorporated into the TESP design. Based on the above assessment and other related documentation like the Punjab and Sindh Education Sector Projects and WB-financed higher education and programmatic lending operations in other countries: (i) the key tertiary education sector issues to be addressed by TESP were clearly identified (as detailed above); and (ii) the suitable institutional and implementation arrangements as well as the lending instrument--a results based Sector Investment Credit rather than a Development Policy Credit used by HESP. 50. Second, TESP is aligned to the GoP’s vision formulation and strategic thinking at the national level, which has been translated into a long-term higher education reform strategy that currently transcends MTDF-HE II 2011-2015. 51. Third, the documentation in the project files attest to a strong GoP ownership at the time of preparation, especially from the HEC whose Chairman became a champion of the proposed TESP. This ownership was reflected by the full commitment, support and engagement provided by the HEC, especially in the design of component 1 through its different working groups, whose overall coordination was under the responsibility of the then HEC Chairman. These included the following HEC Divisions: (i) DLI 2, Human Resource Development Advisor; (ii) DLI 3 and DLI 9, Quality Assurance and Learning Innovation Advisor; (iii) DLI 4, Academic Advisor; (iv) DLI 6, Director General of Research and Development; (v) DLI 7, Director General of Finance; (vi) DLI 8, Director General of Quality Assurance Agency; and (vii) DLI 10, Academic Members. The operational design of DLI 1 was the responsibility of the Ministry of Finance (MoF)10. The WB, HEC and MoF acted almost as one team during the entire lending phase.

10 The implementation of DLI 5 would be a responsibility of participating HEIs, while the envisaged TESP Secretariat to be established during implementation process, was to be responsible for the implementation of component 2.

Mobilizing private sector financing 46. Not applicable. Poverty reduction and shared prosperity 47. TESP contributed to poverty reduction and shared prosperity through increasing and exceeding the target of the number of students benefiting from a financial aid program, including a Needs-Based Scholarship Program and a number of new postgraduate scholarships. Other unintended outcomes and impacts

48. None.


Page 20 of 54

52. Fourth, the project design appropriately addressed key sector issues—access, quality and relevance of learning and research, governance and management and improved financing in a holistic fashion. Linking the disbursements of the most important component (1) to ten tangible and verifiable DLIs, most of them associated with ongoing activities of the HEC (the implementing entity), minimized possible disbursements hurdles.

53. Areas which could have strengthened project preparation. First, even though an institutional capacity assessment of HEC was carried out flagging its weaknesses, this assessment seems to have underestimated the seriousness of these gaps. It seems that the earlier HESP/MTDF-HE I 2005-2010 operation had given the preparation team enough confidence in the institutional capacity of HEC to implement TESP with no major envisaged issues, and thus, no additional digging into HEC’s institutional capacity was warranted at the preparation process. This assumption was further confirmed by the satisfactory conclusions of the November 29, 2010 Quality Enhancement Review (QER) meeting. As a consequence of this limited implementation capacity by HEC, some activities were not implemented on a timely basis and procurement of required TA was stalled for a long time.

54. Second, preparation could have benefitted from an institutional capacity assessment of a sample of HEIs and ACs. Without this, there was limited identification of potential hurdles for these institutions to achieve the intended targets (DLI 4, DLI 9 and intermediate outcome indicator 7) and propose mitigation measures. 55. Third, because WB management did not permit the use of Designated Accounts/Special Accounts in new loans/credits in Pakistan, including TESP, due to some unresolved issues with other closed WB/IDA-financed operations in the country, a new disbursement arrangement had to be agreed for component 2. Accordingly, agreement was reached during preparation that the MoF would transfer funds to HEC for the implementation of component 2 and the WB would then reimburse the GoP against evidence of eligible expenditures (TA). The operational complexities of this new disbursement arrangement were not properly assessed and no appropriate mitigation measures were reflected in a POM. Consequently, the transfer of funds from MoF to the HEC for component 2 in the absence of a TESP designated account became an implementation hindering issue, especially from effectiveness to April 2013 (18 months). 56. The main factors hindering the early implementation stages were outside the control of the WB team and difficult to predict. This led to a restructuring in December 2013 and an 18-month extension of the closing date and these were not a lack of implementation readiness conditions or the design weaknesses. Rather the following two factors outside the control of the WB team impacted the early implementation: (i) a serious deteriorating fiscal situation that will prevent compliance with DLI 1 and other DLIs; and (ii) uncertainty caused by the enactment of the 18th Constitutional Amendment in Parliament leading to grey areas concerning the devolution of HEC’s responsibilities to the provincial governments.

B. KEY FACTORS DURING IMPLEMENTATION 57. Effective and timely implementation of the TESP was underpinned in two critical conditions: (a) MoF compliance in ensuring an adequate budget framework for the MTDF-HE II and timely release of


Page 21 of 54

funds, especially the development grants (DLI 1)11; and (b) HEC’s capacity to ensure the achievement of the corresponding DLIs under component 1. In addition, capacity to carry out, through the TESP Secretariat, the selection and contracting of the required qualified TA and produce TESP reporting and required studies under Component 2 on a timely basis as specified in the TESP Financing and Project Agreements. 58. These two necessary conditions were not complied with during the first two years of implementation after Credit effectiveness (November 2011-December 2013) as reflected in the first 5 Implementation Status Reports (ISRs) rating the implementation progress between unsatisfactory to moderately unsatisfactory. Therefore, the narrative of this section of the ICR is divided in two distinctive time periods – before the restructuring of December 2013 and after, where implementation was consistently rated moderately satisfactory by the ISRs 6 to 11. Factors outside Government control – natural disasters 59. There was severe flooding in the summer of 2010 due to heavy monsoon rains which affected approximately one-fifth of Pakistan's total land area and about 20 million people. This shifted the financial allocation priorities of the government because of the necessary emergency relief efforts that the GoP had to undertake from 2011 to 2013 to address the destruction of property, livelihood and infrastructure. This situation was further compounded by slow economic growth, declining reserves, increased fiscal deficit and limited cash-flow, which required an IMF loan to the GoP of about US$6.6 billion in September 2013 to support stabilizing the economy, boosting growth while expanding the social safety net to protect the poor. This grim situation greatly explains the lack of compliance in ensuring an adequate budget framework for the MTDF-HE II and timely release of funds in the period 2011-2013. 60. Accordingly, counterpart funding was consistently rated moderately unsatisfactory until June 2015 because of insufficient development fund releases to the HEC explained in the paragraph above. Factors generally subject to Government control 61. The first half of the implementation cycle covering the period March 2011-January 2014 is characterized by having achieved, by and large, six DLIs and displaying a shortfall in achieving the following four: DLI 1, DLI 4, DLI 7 and DLI 9 as well as showing a poor implementation progress with component 2 largely due to financial and implementation bottlenecks. There are 3 key factors subject to the GoP control that affected this first half of the implementation cycle leading to the restructuring. 62. First, there was an eight-month delay in TESP effectiveness and launching after Board approval due to the uncertainty surrounding the status of the devolution of HEC’s functions to the Provincial Government complying with the 18th Amendment to the Constitution approved by Parliament in April 2010 at the time of project identification. Even after effectiveness, this confusion continued to delay the implementation of some TESP activities. This implementation delay, combined with others generated by the limited implementation capacity of HEC/TESP Secretariat, especially as it concerns component 2, will result in a request to extend the Credit closing date by 18 months at the end of 2015.

11 Which is the fundamental premise of all DLI-based operations, as well as P4Rs financed by the WB.


Page 22 of 54

63. Second, the budgetary allocations to the HEC fell short of the original budget framework agreed with the WB at the time of TESP approval. Third, there were considerable delays from the MoF in releasing approved TESP-related funds to HEC, creating serious implementation issues, especially with the carrying out of several studies under component 2 listed as legal covenants, like the baseline survey of the higher education sector and a quality of education services, which were dropped under the restructuring. 64. Accordingly, the TESP was restructured at the very end of 2013 at the request of the GoP, with 15 months left to arrive at the Credit’s original closing date. Once the restructuring was completed and approved, the ban on recruiting new staff was lifted by the MoF for the TESP in 2013, compliance with the GoP financial commitment towards MTDF-HE II also significantly improved. The ratings of counterpart funding improved from unsatisfactory and moderately unsatisfactory between November 2011 to June 2015 (ISRs 1 to 8) to moderately satisfactory and satisfactory since then (ISRs 9 to 12) and the rating of component 2 was improved to moderately satisfactory from early 2016 onwards. The implementation of the TESP significantly improved, the PDOs were achieved and total disbursements reached the percentages mentioned in other parts of this ICR. Factors generally subject to implementing entity (HEC) control 65. HEC was entrusted with the overall responsibility of implementing and managing TESP with the support of the TESP Secretariat established during the implementation phase. Three HEC-related factors hindered progress implementation in implementation cycle up until January 2014, roughly. First, the HEC went through turbulent times from August 2012 to February 2013 experiencing major changes in key HEC staff, including the replacement of the Executive Director, several advisers and director generals who were involved in the design and early implementation stages of the TESP12. With each new arrival, a time-consuming TESP learning curve was required. These changes resulted in a quasi-stalemate of the implementation adding to the initial delays in the start-up phase and cancellation of two implementation support missions in September 2012 and March 2013 (ISR 4). 66. Second, the TESP Secretariat (Project Implementation Unit) was established at the HEC to: (a) provide strategic support for the carrying out of different TESP activities, especially as it concerns component 2; (b) coordinate with various focal persons from different HEC’s organs participating in the achievement of the DLIs; and (c) report the implementation progress to the HEC, MoF and the WB. Accordingly, the Project Agreement stipulated that the Secretariat was to be timely staffed with a Project Coordinator, four program officers, two procurement specialists, an M&E specialist and a communication officer. None of these positions were filled on timely basis and there were frequent changes once the positions were filled. Delays in staffing the TESP Secretariat had significant negative impact on recruiting TA, especially for the third-party verification of compliance with the DLIs’ by-annual targets and studies that needed to be canceled for component 2. Due to lack of qualified M&E person, the monitoring and

12 HEC Chairman changes: one up to 2013; second for the period 2013-2014; and third from 2014 onwards. With the arrival of a new HEC Chairman, the lobbying capacity with MoF for the timely release of approved recurrent and developmental funds could improve or weaken depending on past historical personal relationships between the corresponding actors. Executive Director changes: one up to 2012; second few days in 2012; third one month in 2012; fourth one year in 2013; fifth one year in 2014; sixth 8 months in 2015; seventh from 2016 onwards. TESP Project Coordinator at the TESP Secretariat changes: one up to end of 2013; second from early 2014 onwards.


Page 23 of 54

evaluation at both the project and the MTDF-II levels was very weak until the restructuring. This resulted in untimely and incomplete reporting including the bi-annual implementation progress reports (IPRs)13, in the early years of the project, thus preventing to inform effectively and timely on the achievement of the DLIs and KPIs included in the RF. Procurement of TA under component 2, rated consistently moderately unsatisfactory, finally improved in early 2016. 67. Third, counterpart funding supported the construction, renovation and rehabilitation of academic, administrative and residential buildings for various departments of Universities and HEIs to cater their growing space requirements for the male and female students and staff. Accordingly, compliance with the environmental safeguards as stipulated in the agreed Environmental and Social Management Plan (ESMP) became a covenant in the Financial and Project Agreements. Weak implementation capacity at both the HEC and participating HEIs to implement the TESP in accordance with the ESMP, resulted in moderately unsatisfactory rating of this compliance until June 2014 as reflected in the ISRs 1 to 6, at which point, the rating improved to moderately satisfactory to satisfactory (as displayed in the ISRs 7 to 11). 68. The Steering Committee that was supposed to provide overall strategic guidance for TESP planning and implementation attuned to the MTDF-HE II as well as oversight of financial management, including reviewing progress reports, internal and external audit and third-party reports, was established late. Despite having met about 5 times during the entire implementation cycle, this oversight body had, by and large, a marginal influence in steering the TESP process. Moreover, the frequent changes in the HEC Executive Director, who was the Chair of the Steering Committee, and the continuous rotation of other members of this Committee, explains to a large degree the limited impact of this oversight arrangement. 69. The TESP was restructured in December 2013 at the request of the GoP grounded on the following two facts: (i) several DLIs were unlikely to be achieved during the remaining life of the project and there was a need to focus on unmet but achievable DLIs; and (ii) the poor implementation progress during the roughly 24 months since Credit effectiveness as explained above. It bears mentioning that poor implementation progress had a lot to do with MoF’s ban on recruiting new staff during 2012 which negatively impacted the implementation of component 2, since the procurement and M&E specialists’ consultants could not be recruited. This ban was lifted in early 2013. TESP was restructured to focus on achievable results with a view to meeting HEC’s medium term goals in the sector. US$77.82 million (SDR 49.9 million) of the original Credit amount of US$300 million were canceled14. TESP’s implementation jump-started after the restructuring and to be highlighted is the fact that the restructuring did not dilute the ultimate results, relevance or efficacy of the project. 70. A training support plan was developed for TESP for component 2 in October 2013 which once it started being implemented, started exhibiting a moderately satisfactory implementation progress as recently as early 2016 including: (i) further capacity building for HEC and HEIs staff, especially in the areas of governance, information technology and financial management; (ii) supporting the updating of the

13 One report covering the period October 1 to March 31 and the second covering April 1 to September 30, within the same academic year. 14 US$33.9 million from non-achievement of DLI 1, US$12.72 million from DLI 4, US$12.72 million from DLI 7, US$8.48 million from DLI 9 and US$10 million from component 2.


Page 24 of 54

Quality Assurance Manual developed almost a decade ago; (iii) extending support to affiliated colleges; and (iv) contributing towards the preparation of MTDF-HE III 2016-2025. 71. Somewhat ambitious end-of-project targets related to DLI 4 (Affiliated Colleges improvement), generated implementation constraints in achieving the original targets of the DLI. During the restructuring the metric was modified and the end-of-project targets significantly reduced. Likewise, lack of capacity of HEIs to draft business plans and of additional resources to implement these plans, were constraints that initially affected compliance of DLI 10 until addressed at the latter stages of the implementation cycle. 72. Lack of appropriate understanding concerning the Distance Education program among several HEIs including delays devising the adequate framework for its provision and absence of risk mitigation plans on how to deal with server and electricity problems, lack of information technology awareness at HEIs and students’ access to IT equipment and internet as well as absence of qualified teachers for this hybrid modality during the weekends (only time distance education students could complement their learning process in a classroom environment) further constrained the achievement of DLI 5 until these issues were properly addressed. 73. To address the poor performance of component 2 until the end of 2015, the GoP and the WB agreed at the end of 2015 to extend the closing date by 18 months from January 1, 2016 to June 30, 2017 to: (i) effectively deliver on the TA under component 2, especially concerning the undertaking of required studies for the design of MTDF-III 2016-2025 thus extending support for sustainable reforms in the Higher Education sector; and (ii) allow for an orderly closing of the Project. 74. After the restructuring, the PDOs were achieved (see Annex 1) and total documented IDA disbursement reached SDR 142.25 million equivalent to 74 percent of the estimated total agreed at appraisal or 100 percent of the estimated total after the cancelation at restructuring. 75. Weighing all of the above, primarily that the implementation shortcomings were satisfactorily addressed by the team at the time of restructuring which led to the successful completion of the project.

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME

A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 76. M&E design was appropriate, but could have been improved by reducing the number of indicators. The M&E design included 4 outcome indicators and 18 intermediate outcome indicators (that translated into 10 DLIs and 12 KPIs), this may have been too many. This many number of indicators resulted in a complex, costly and time-consuming effort in collecting and analyzing the required information and reporting on the updated RF. The M&E architecture, although very much connected to the PDOs, could have been simplified by eliminating those indicators not strongly connected with the PDOs and merging others into an overall new metric (see Annex 7).


Page 25 of 54

77. In addition, as mentioned above, there were ambitious targets such as: (a) DLI 4 concerning ACs; (b) intermediate outcome indicator 7 concerning number of administrators in ACs participating in management training; and (c) DLI 9 concerning number of HEIs assessed against institutional performance evaluation standards. The corresponding targets were significantly decreased during the restructuring. M&E Implementation 78. M&E implementation. The assessment of the TESP M&E implementation can be divided in two periods – before and after restructuring. During the former period, M&E was consistently rated by the WB supervision missions as unsatisfactory to moderately unsatisfactory (ISRs 1 to 4), mainly due to the: (i) weak M&E capacity at the national (HEC/TESP Secretariat) and institutional (HEIs) levels not appropriately addressed by the project (no qualified M&E specialist on board at the TESP Secretariat); (ii) lack of progress reporting (Draft Quarterly Progress Report and Annual Progress Report) and/or reporting which was incomplete, not appropriately organized and delayed; and (iii) lack of some verifiable reports concerning compliance of DLIs because the issues of procuring the corresponding TA under component 2 to carry out these verifications on a timely basis. 79. After TESP’s restructuring, the M&E staffing at the TESP Secretariat was resolved in 2014 and the IPR reports, of good quality and relevance were produced and submitted on time. In addition, by appropriately addressing the transfer of funds from the MoF to the HEC to unblock the implementation stagnation of component 2 at the end of 2015, a series of nine studies, including, Public Expenditure Tracking Survey and the Employer Perception Survey were commissioned and were completed by project closing. Accordingly, the rating of: (a) the M&E was improved from moderately satisfactory to satisfactory from 2014 onwards (ISRs 5 to 12); and (b) component 2 improved to moderately satisfactory in early 2016 (ISRs 10 and 12). 80. As mentioned earlier, the baseline study of HEIs, as well as the impact study of the TESP on learning and labor markets outcomes, were not carried out mainly due to the issues associated with the implementation of component 2. 81. On the positive side, the employers perception survey, the public expenditure tracking study and the student satisfaction survey were completed and it is hoped that the findings would be factored in the design of the follow up operation currently under preparation. 82. The ICR identified a series of issues negatively affecting the implementation of the M&E framework, some that were addressed at different times of the implementation cycle and others like (d) below that continue to be a challenge:

a) Data specialists such as statistical officers were not placed in many HEIs, while frequent changes of department heads were a recurring phenomenon, resulting in lack of ownership of data within the institutions;

b) The information and communication technologies facilities in many HEIs needed to be improved

for the maintenance and development of accurate databases required to feed the updating of the TESP RF;


Page 26 of 54

c) Online systems needed to be strengthened and training of focal M&E persons at the HEIs and affiliated colleges needed training to provide accurate and quick information to the HEC; and

d) Reliability of data continues to be a problem like in the case of the number of regular students in

public and private higher education institutions. M&E Utilization 83. The M&E was utilized in an appropriate fashion for the following two reasons. First, third party validation of the information associated with the compliance of each DLI was a requirement for the WB reimbursement of eligible TESP-related expenditures under component 1 and this process seemed to have worked well. 84. Second, the IPR includes valuable lessons learned derived from the analysis of each DLI and KPI, which will primarily inform decision-making and resource allocation in the forthcoming MTDF-HE III operation, currently under preparation. Justification of Overall Rating of Quality of M&E 85. The ICR assigns a Substantial rating to the quality of M&E for the following three reasons. First, from the design point of view, the theory of change is clear and most of the indicators in the RF, are connected to the PDOs. Second, the key M&E implementation shortcomings were satisfactorily addressed at the restructuring stage. Third, the current M&E arrangement and utilization of findings has reached a level of methodological soundness, quality and reliability thus increasing the likelihood of its long-term sustainability and robust preparation and implementation of the follow-up operation.

B. ENVIRONMENTAL, SOCIAL AND FIDUCIARY COMPLIANCE 86. There were issues with compliance with the environmental safeguards until mid-2014 (ISRs 3 to 6) mainly due to the lack of capacity of the HEIs to implement the ESMP in the construction, renovation and rehabilitation of academic, administrative and residential buildings catering to their growing space requirements being financed by counterpart funding. 87. Once environmental monitoring committees were constituted at 50 universities, training on compliance with the ESMP provided and 42 of these started reporting on this compliance as per prescribed template, the rating of the compliance with the environmental safeguards improved to moderately satisfactory from the end of 2014 onwards. 88. At preparation, no negative social safeguard issues and impacts on the Project were anticipated. Nor did these appear during implementation. Therefore, social safeguards was deemed N/A and not monitored or rated during Project implementation. 89. An assessment of the financial management arrangements in place was carried out for HEC during preparation rating its overall FM risk as moderate. Staffing, budgeting, accounting, internal controls, financial reporting and audit arrangements were found satisfactory. Cognizant that during 2013 there


Page 27 of 54

were some delays in producing the corresponding interim unaudited financial report (IFR, ISR 3) and in another occasion the IFR was overdue (ISR 5), these matters were addressed satisfactorily and, by and large, there were no more issues with financial management during the entire implementation cycle which was rated moderately satisfactory during 2014 and satisfactory from 2015 onwards. 90. A procurement capacity assessment of HEC was carried out during preparation. The assessment reviewed the organizational structure, procurement procedures and practices, staff responsible for procurement and types of procurement taking place and identified issues and risks. The main risk, rated moderate, was lack of experience in conducting procurement under the WB’s procurement guidelines. The mitigating measures included: (i) appointing two dedicated procurement specialists at the TESP Secretariat prior to disbursement under Component 2; and (ii) providing procurement training to familiarize key staff with the WB’s procurement guidelines and procedures July 1, 2011. None of these measures were addressed on time. 91. There was a delay in recruiting the two dedicated procurement specialists at the TESP Secretariat (ISR 4) and in maintaining a procurement website, publicly accessible and a record keeping system acceptable to the WB (ISR 3). While HEC had a fairly developed procurement system centered on Public Procurement Rules 2004 and the capacities in managing goods contracts were adequate, it was weak in the procurement of services (TA under component 2). 92. The delimitation of roles and responsibilities between the human resource and the procurement units within HEC was rather blurred and, though TESP Secretariat was identified as a single point of contact in the Project Agreement, Schedule, Section I A (b), various units of HEC insisted on intermixing of WB’s and government’s procurement policies resulting in avoidable delays. In conclusion typically employed mitigation measure of recruiting procurement staff as individual consultant, POM and trainings were stipulated. The appraisal presumably did not foresee the role of fully functional Directorates of HR, Procurement and IT that were likely to play a critical role in project implementation. The interface attrition between TESP Secretariat and various HEC Directorates was in a way a key contributor to delays. Moreover, the lack of clarity within HEC, chiefly due to the financial arrangements, led to intermixing of Bank and Government’s processes thereby causing implementation delays. TESP also witnessed highest number of procurement staff turnover mainly due to a dearth of competent procurement specialists in the Pakistani market. 93. The above issues were eventually solved, with strong Bank implementation support and accordingly, the ISRs 5 to 7 and 9 to 12 rated the TESP-related procurement activities and procedures as moderately satisfactory.

C. BANK PERFORMANCE Quality at Entry 94. Several preparation strengths were displayed by the WB team during the period April 2010 to February 2011 covering the identification, preparation/design, appraisal and negotiation of the TESP. It can be clearly surmised that this preparation process led to full ownership by the GoP.


Page 28 of 54

Quality of Supervision 95. The task team leader (TTL) for the lending cycle continued in that role until 2016 at which time, the co-TTL, located in the WB Islamabad Office since mid-2013, took over the TTL responsibilities. This arrangement not only provided continuity to the WB supervision but also, after mid-2013, consistent and timely on-site implementation support and close communication with the key counterpart players on an ongoing basis. This helped resolve implementation hurdles on the spot as they happened, leading to the restructuring, provided timely technical inputs on the development of benchmarking instruments required by some DLIs and on-hand support and training for the implementation jump start of component 2. 96. Overall, the supervision missions included the appropriate skill-mix to carry out their envisaged terms of reference as shown in Annex 2, including procurement, financial management and the social safeguards as well as the higher education specialist. Supervisions included visits to some participating HEIs and ACs. The findings of each supervision mission were conveyed to WB management and to the GoP clearly and in a straightforward manner, including accomplishments and implementation shortcomings, potential risks and implementation flags, as reflected in the ISRs. The WB team intensified its support at the time of restructuring in a more targeted and frequent fashion, mainly because the proximity of the co-TTL to the field. As a result, after restructuring implementation improved significantly, the PDOs were achieved and the pace of disbursements increased. Justification of Overall Rating of Bank Performance 97. The ICR rates the overall Bank performance as Moderately Satisfactory because the WB supervision team was effective in helping to improve implementation significantly during the second half of the TESP cycle, after the restructuring, with achievement of the PDOs and a disbursement rate of 74 percent against the estimated total at appraisal or 100 percent of the estimated total at restructuring.

D. RISK TO DEVELOPMENT OUTCOME 98. Commendably, the HEC has signaled its ambitious plan to continue the comprehensive reform commitments of the government in the area of higher education in Pakistan initiated in 2004 and clearly seen in the HEC Vision 2025. However, there are, like all ambitious reform plans, technical, financial/economic, security and political risks to sustain and expand the gains of TESP. This is further explained below. 99. There may be weak ownership of reform activities in a number of HEIs and ACs posing an important implementation challenge to the HEC and the higher education sector as a whole. In addition, and this is a typical risk in many countries, the institutional capacity built up at the participating HEIs and ACs supported by TESP could weaken in the short to medium terms if trained staff migrate out of their current jobs. 100. Pakistan’s security environment and law and order situation remains volatile impacting the scarce public financial resources that the GoP could be allocating to the higher education reform. Most recent WB estimates recorded in the current CPS show the annual costs of conflict in Pakistan at no less than 2


Page 29 of 54

percent of GDP. The perception that Pakistan is a high-risk country discourages private and foreign investment for growth and job creation, further impacting the fiscal picture. Adding to the above, the fiscal deficit, currently estimated at about 5.3 percent of GDP (2015), could be further increased by the costs associated with energy shortages, lower tax base, natural disasters caused by floods and torrential rains, increasing debt servicing requirements, higher than budgeted subsides and maintenance derived from the gradual dilapidation in the socio-economic infrastructure. 101. Finally, the lack of clarity on the 18th Amendment concerning the devolution of HEC functions to the Provincial Governments still persists. Going forward, a role that the Bank could play with the HEC and the Provincial Governments is building up the institutional capacities required to implement and maintain the activities associated with the DLIs. Otherwise there could be a stagnation of the tertiary education system and the gains made through this project and the MDTF HE II.

V. LESSONS AND RECOMMENDATIONS

102. Continue strengthening the ongoing shift from a process oriented and low institutional accountability paradigm in the tertiary education sector to an outcome oriented strong institutional accountability model of Government support. Whereas both the MTDF-HE I 2005-2010 and HE-II 2011-2015 supported the establishment of conducive environment for improved teaching, learning and research in the tertiary education system of Pakistan, the MTDF-HE-III 2016-2025 needs to further support the ongoing migration of the paradigm in this sector by allocating public recurrent and development grants based on observable and measurable agreed outcomes with the participating institutions, thus significantly increasing their accountability. Under this envisaged paradigm it becomes: (a) equally important to account for enrollment (as in TESP) as on the tracking of the employability of higher education graduates and their performance in the labor market; (b) as important to account for scholarships provided to study abroad (as in TESP) as to track the job status of beneficiaries and their performance upon their return to the country; and (c) equally important to account for the number of ORICS meeting the quality benchmark (as in TESP) as it is to account for the revenue generated by these ORICs as a percentage of the institution’s annual budget. 103. HEC being in the driver seat owning and implementing the program without the need of parallel structures, except for M&E and procurement of TA under component 2 managed by the TESP Secretariat, greatly contributed to the satisfactory achievement of the stated PDOs and ensured technical, institutional and financial sustainability of TESP’s key interventions after the closing date. Ensuring and maintaining “client ownership” from the early stages is a key to success. This empowerment has been gradually and slowly down streamed to the participating HEIs and ACs, albeit with a need to sustain, and if possible, expand the gains. 104. An agreed draft Operational Manual should be made a condition for negotiations. TESP’s draft POM was made available to the WB in 2012 about a year after having negotiated the Credit, and the no-objection to a further revised draft, was provided by the WB a year later. As a consequence, for example; (a) there was no sufficient clarity for about 18 months after Credit effectiveness on how to operationalize the transfer of budgetary funds from MoF to the HEC/TESP Secretariat to implement component 2; and


Page 30 of 54

(b) there was role confusion between different areas of HEC to carry out TESP-related procurement processes. 105. A lesson learned from HESP 2009, still relevant for TESP is that in an environment in which resource availability is uncertain or constrained, like with the compliance of DLI 1 in this project, the financial sustainability aspects need to be factored early into the program design. These considerations should influence the nature and scope of the proposed interventions, instruments and mechanisms to achieve the desired goals without unduly compromising the objectives of the proposed program. 106. A simpler but relevant RF could have facilitated the monitoring of the implementation progress thus contributing to a more-timely corrective and/or preventive decision-making process. The TESP’s RF with 4 outcomes and 18 intermediate output indicators comprising 10 DLIs and 12 KPIs resulted in a complex, time-consuming and costly monitoring scheme. 107. Use of designated accounts for procurable non-DLI eligible expenditures provides flexibility and opportunity during the implementation of project activities. .


Page 31 of 54

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS

A. RESULTS INDICATORS A.1 PDO Indicators

Indicator Name Unit of Measure Baseline Original Target Formally Revised

Target

Actual Achieved at Completion

Number of QECs performing satisfactorily as measured by scorecards

Number 0.00 45.00 45.00 90.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 100%. 147 QECs have been established, but only 90 have achieved the satisfactory performing rating as measured by scorecards.


Target


Number of faculty members on the Tenure Track System

Number 849.00 2349.00 2149.00 3838.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017


Page 32 of 54

Comments (achievements against targets): Target exceeded by 78.6 percent. It is an ongoing regular activity within the HEC that continues after the closing date.


Target


Number of regular students in public and private HEIs

Number 463803.00 604803.00 604803.00 867562.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 43.5 percent.


Target


Number of ORICs performing satisfactorily as measured by the scorecard system

Number 0.00 15.00 15.00 41.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 173.3 percent. 44 ORICs were established under the TESP, but only 41 performed satisfactorily as per scorecard system.

A.2 Intermediate Results Indicators

Component: Allocation and timely release of recurrent and development funds to HEC as per agreed medium term budgetary framework for higher


Page 33 of 54

education (MTBF-HE)

Unlinked Indicators


Target


Budgetary allocation to HEC as percentage of GDP

Percentage 0.25 0.26 0.26 0.30

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded.


Target


Number of administrators in affiliated colleges participating in management training

Number 0.00 750.00 102.00 278.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Revised target at the restructuring exceeded by 172.5 percent. With respect to the original target set up at appraisal the achievement felt short by 62.9 percent.


Target


Number of technology Number 12.00 45.00 45.00 180.00


Page 34 of 54

companies established and incubated in HEIs

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 300 percent


Target


Share of universities' self-generated revenues in overall total resources

Percentage 44.00 46.00 46.00 48.39

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 5.2 percent.


Target


Improved strategic management, planning and accountability in public HEIs

Number 0.00 70.00 70.00 70.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target achieved.


Target


Number of revised and restructured undergraduate and/or graduate programs in

Number 0.00 45.00 45.00 128.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017


Page 35 of 54

HEIs

Comments (achievements against targets): Target exceeded by 184.4 percent


Target


Number of assessed affiliated colleges which successfully complete all training/capacity building activities for selected MQS under HEC implemented Capacity Building Plan (Revised Indicator)

Number 0.00 800.00 102.00 102.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): The revised target at the restructuring was achieved. However, with respect to the original target the achievement felt short by 87 percent, partially because using an inappropriate benchmarking instrument.


Target


Provision of enhanced quality education at M.A. level for external students

Number 0.00 6.00 6.00 12.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 100 percent

Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at


Page 36 of 54

Target Completion

Number of new postgraduate foreign scholarships awarded

Number 300.00 2250.00 2250.00 1491.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Achieved target fell short by 33.7 percent from the end-of-project target agreed at appraisal.


Target


Number of students benefiting from a financial aid program, including a Needs-Based Scholarship Program and a Student Loan Program

Number 1000.00 6000.00 20161.00

24-Mar-2011 31-Dec-2015 30-Jun-2017

Comments (achievements against targets): Target exceeded by 236 percent but only with respect to the scholarship part of the metric. The student loan program never got off the ground and was deleted during restructuring.


Target


Number of private TEIs eligible for public support as per approved policy

Number 7.00 31.00 31.00 29.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Achieved target fell short by 6.4 percent from the end-of-project target agreed at appraisal.


Page 37 of 54


Target


Number of academic programs accredited by Accreditation Councils established by HEC

Number 60.00 400.00 400.00 1027.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 156.7 percent


Target


Number of HEIs assessed against Institutional Performance Evaluation Standards

Number 0.00 60.00 13.00 25.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 92.3 percent against the revised target set at restructuring, but falling short by 58.3 percent against the appraised end-of-project target.


Target


Number of public HEIs with a fully functional Campus Management Solution Module (CMS)

Number 8.00 38.00 38.00 8.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017


Page 38 of 54

Comments (achievements against targets): Achieved target fell short by 79 percent from the end-of-project target agreed at appraisal.


Target


Number of HEC, Provincial education officials, and HEIs staff participated in staff development programs

Number 0.00 100.00 100.00 4992.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by almost 4892 percent (forty fold). Unless tracking the performance of this trained staff, the indicator, as stated, seems disconnected from the PDOs.


Target


Number of new postgraduate indigenous scholarships awarded

Number 700.00 3000.00 3000.00 4500.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017

Comments (achievements against targets): Target exceeded by 50 percent. It seems that the number recorded at the end-of-the project is the cumulative, including the baseline, and not the additional.


Target


Student-full time faculty (with PhD) ratio in the public university sector

Percentage 93.00 88.00 88.00 74.00

24-Mar-2011 31-Dec-2015 30-Jun-2017 30-Jun-2017


Page 39 of 54

Comments (achievements against targets): Target fell short by 15 percent from the end-of-project target agreed at appraisal.


Page 40 of 54

B. KEY OUTPUTS BY COMPONENT

Objective/Outcome 1: Enhanced program financing at the tertiary education system

Outcome Indicators 1. Indicator 3. Number of regular students in public and private HEIs 2. Indicator 2. Number of faculty members in the TTS (DLI 3) 3. Indicator 4. Number of ORICs performing satisfactory as measured by scorecards (DLI 6)

Intermediate Results Indicators

1. Indicator 5. Allocation of timely release of recurrent and development funds to HEC as per agreed medium term budgetary framework for higher education (DLI 1) 2. Indicator 6. Budgetary allocation to HEC as percentage of GDP 3. Indicator 8. Number of technology companies established and incubated in HEIs 4. Indicator 9. Share of universities’ self-generated revenues in overall total resources 5.Indicator 11. Number of revised and restructured undergraduate and/or graduate programs in HEIs 6. Indicator 12. Number of AC meeting minimum quality standards set by HEC 7. Indicator 13. Provision of enhanced quality education at M.A. level for external students (DLI 5) 8. Indicator 14. Number of new postgraduate foreign scholarships awarded 9. Indicator 15. Number of students benefiting from a financial aid program, including a Needs-based scholarships Program and a Loan Program (DLI 7) 10. Indicator 16. Number of private TEIs eligible for public support as per approved policy. 11. Indicator 17. Number of academic programs accredited by Accreditation Councils established by HEC 12. Indicator 18. Number of HEIs assessed against institutional performance evaluation standards (DLI 9) 13. Indicator 21. Number of new indigenous postgraduate scholarships awarded (DLI 2) 14. Student-full time faculty (with PhD) ratio in the public university sector.


Page 41 of 54

Key Outputs by Component (linked to the achievement of the Objective/Outcome 1)

1. Recurrent and development funds agreed in the mid-term budgetary framework are timely release (DLI1) 2.Formula used by HEC to determine the amount of funds to be allocated to universities is refined and implemented 3. Technology companies in HEIs established under TESP generating revenue 4. HEIs increase the percentage of yearly self-generated revenue out of the overall total yearly financial resources 5. New indigenous postgraduate scholarships awarded (DLI 2) 6.TTS being implemented as per agreed annual operation plan (DLI3) 7. Improved faculty 8. ACs implement measure to meet minimum quality standards set by HEC (DLI4) 9. MA level for external students is enhanced (DLI5) 10. Establishing ORICs and ensuring satisfactory performance as measured by scorecards (DLI6) 11. Provision of financial aid program, including a needs-based scholarship program under a student loan program (DLI7) 12. Granting new postgraduate foreign scholarships 13. Targeted public HEIs facilities physically upgraded 14. Distance education in both private and public universities scaled up 15. Higher education supply increased through private-public partnerships in HEIs 16. Quality assurance mechanisms fully functional and institutionalized 17. Undergraduate and/or graduate programs in HEIs revised and restructured 18. Institutional performance evaluation standards for HEIs established by HEC and fully functional 19. Professional accreditation councils, especially in new areas, fully functional 20. A comprehensive, reliable and timely HEMIS develop and fully functional 21. Identification of private TEIs eligible to receive public support as per approved policy 22. Fiduciary control systems, especially at the institutional level, strengthen 23. Communication capacity at the institutional level improved


Page 42 of 54

Objective/Outcome 2: Improved capacity building, policy design and monitoring and evaluation

Outcome Indicators 1. Indicator 1. Number of QECs performing satisfactory as measured by scorecards (DLI 8)

Intermediate Results Indicators

1. Indicator 7. Number of administrators in AC participating in management training 2. Indicator 10. Improved strategic management, planning and accountability in public HEIs (DLI 10) 3. Indicator 19. Number of HEIs with fully functional Campus Management Solution Module 4. Indicator 20. Number of HEC, Provincial education officials and HEIs staff participated in staff development programs

Key Outputs by Component (linked to the achievement of the Objective/Outcome 2)

1. Establishment of QECs and ensuring their satisfactory performance as per scorecards 2. Development of a Campus Management Solution Module 3. Internal governance of targeted HEIs streamlined and strengthen 4. Improved strategic management, planning and accountability in public HEIs (DLI10) 5. Management training provided to administrators of HEIs 6. Staff development programs developed and implemented


Page 43 of 54

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION

Names Title

Lending

Javaid Afzal Senior Environmental Specialist

Zia Al Jalaly Senior Social Development Spec

Asif Ali Senior Procurement Specialist

Zubair Khurshid Bhatti Education Specialist

Halil Dundar Senior Education Specialist, TTL

Mehnaz Haider Senior Operations Officer

Ayesha Khan Consultant

Benoit Millot Consultant

Hanid Mukhtar Senior Economist

Yoko Nagashima Senior Education Specialist

Naveed Hassan Naqvi Senior Education Specialist, TTL

Naoko Ohno Senior Operations Officer

Saeeda Sabah Rashid Financial Management Specialist

Furqan Ahmad Saleem Sr Financial Management Specialist

Huma Ali Waheed Research Analyst

Supervision/ICR

Juan Carlos Alvarez Senior Counsel

Zubair Khurshid Bhatti Senior Education Specialist

Halil Dundar Senior Education Specialist

Neelam Ejaz Operations Analyst

Akram Abd El-Shorbagi Financial Management Specialist

Rehan Hyder Senior Procurement Specialist

Rahat Jabeen Environmental Specialist

Nasreen Shah Kazmi Program Assistant

Ayesha Khan Consultant

Scherezad Joya Monami Latif Lead Education Specialist, TTL

Benoit Millot Consultant – Higher Education

Hanid Mukhtar Senior Economist

Naoko Ohno Senior Operations Officer

Rubina Quamber Program Assistant

Saeeda Sabah Rashid Financial Management Specialist

Jean Risopoulos Economist

Martin Serrano Senior Counsel

Shrinivasan Thiruvengadachari Remote Sensing Specialist

John Weatherhogg Agricultural economist

Chaohua Zhang Safeguard Specialist


Page 44 of 54

A. STAFF TIME AND COST

Stage of Project Cycle Staff Time and Cost

No. of staff weeks US$ (including travel and consultant costs)

Preparation

FY10 4.125 38,830.39

FY11 58.299 309,908.54

FY12 17.775 79,012.09

FY13 0 6,651.28

Total 80.20 434,402.30

Supervision/ICR

FY11 0 86.35

FY12 36.921 246,765.52

FY13 59.570 284,466.57

FY14 47.647 211,625.83

FY15 19.887 119,851.54

FY16 10.264 107,443.91

FY17 14.075 145,649.63

FY18 2.652 46,514.34

Total 191.02 1,162,403.69


Page 45 of 54

ANNEX 3. PROJECT COST BY COMPONENT

Components Amount at Approval

(US$M) Actual at Project

Closing (US$M) Percentage of Approval

(US$M)

Component 1: Program Financing

280.00 212.18 76

Component 2: Capacity Building, Policy Design and Monitoring and Evaluation (M&E)

20.00 10.00 50

Total 300.00 222.18 74

Note: SDR49.9 Million, equivalent to US$77.82 was cancelled at the time of the restructuring on December 2013.


Page 46 of 54

ANNEX 4. EFFICIENCY ANALYSIS

1. This annex summarizes the economic analysis for TESP by presenting both the economic and financial analysis of the project.

A. Economic Analysis

2. This economic analysis presents a CBA of TESP using the present discounted value method. Following the standard practice, the economic costs and benefits are estimated over the five years of the project relative to a counterfactual. Benefits include the number of additional college and university graduates and the improved quality and relevance of higher education due to project interventions which is assumed to result in higher wage premiums for college graduates. Costs include HEC expenditure, and private costs which include direct household expenditure and opportunity costs of schooling over the project period.

3. The following assumptions and data sources were used to estimate the net present value (NPV) of project benefits, and the EIRR.

(a) Data on higher education graduates are obtained from the HEC from 2010 through to 2016. The total number of graduates over this period were 657,263.15

(b) TESP costs include HEC expenditure over the project period including the recurrent grant budget and the development grant budget, as well as the technical assistance expenditure under the project. The total expenditure over the five years of the project is US$4.43 billion in nominal terms.

(c) Private per child household expenditure on education are calculated using PSLM survey data 2013–14, by calculating the average annual expenditure per student enrolled in tertiary education. On average, households spend approximately US$409 per year for students enrolled in tertiary education. It is assumed that 54 percent of tertiary age individuals would have been engaged in paid labor had they not enrolled in school, based on the labor force participation rates derived from PSLM survey data (2013–14).

(d) Lifetime labor earnings are estimated using the PSLM survey data (2013–14) to determine benefits due to increased earnings for secondary and university completers. Lifetime earnings are calculated for five cohorts over a period of 20 years for secondary school completers as well as university graduates. Given that the project involved activities that were expected to improve the quality and relevance of higher education, it is assumed that university graduates will experience a 5 percent productivity premium over the existing wage premium (2 percent is assumed for the low case in the sensitivity analysis while 7 percent is assumed for the high case). Labor force participation (LFP) rates are derived from PSLM 2013-14 which is 54 percent for secondary completers, and 61 percent for university graduates. For the sensitivity analysis, LFP rates are assumed to be 3 percentage points

15 This number assumes that all the 1,191 students studying through project funded scholarships abroad all graduated from their respective programs.


Page 47 of 54

above the base case for the high case scenario, and 3 percentage points below the base case for the low case scenario.

(e) The discount rate utilized for the analysis is 10 percent, while inflation is projected to be 7 percent per year, based on a three-year average of the consumer price index.

4. Based on the assumptions above, the NPV of economic benefits under TESP is estimated to be US$1.29 billion, with an estimated EIRR of 14 percent. Table 1.1 presents a summary of the sensitivity analysis with the NPV of economic benefits, and the EIRR, under the base, low, and high case scenarios.16 The NPV ranges from US$705 million to US$1.79 billion while the EIRR ranges from 12 percent to 15 percent under the various scenarios indicating that TESP was economically viable.

Table 1.1 Sensitivity Analysis

NPV (US$, millions) Internal Rate of Return (%)

LFP

Base Case

LFP Low Case

LFP High Case

LFP Base Case

LFP Low Case

LFP High Case

Quality premium base case

1,287 1,014 1,559 14 13 14

Quality premium low case

962 705 1,219 13 12 13

Quality premium high case

1,503 1,220 1,787 14 13 15

5. Economic analysis at appraisal. Given that there was no economic analysis conducted at appraisal, it is not possible to compare the EIRR at the time of appraisal with the EIRR at project completion.

B. Financial Analysis

6. Over the project period, the GoPakistan demonstrated strong commitment to the higher education sector by increasing expenditure by 113 percent (in nominal terms) between 2011-12 and 2016-17. The project’s contribution to the overall federal higher education sector has ranged from less than 1 percent in the final year of the project to as high as 11 percent reflecting that the project has had a substantial impact on the federal higher education sector.

16 Assumptions relating to the LFP rate and quality premium under the project are varied for the sensitivity analysis and are specified in the assumptions section above.


Page 48 of 54

Table 1.2 HEC Budget Expenditure

Source: HEC

2011-12 2012-13 2013-14 2014-15 2015-16 2016-17

Budget Head

Recurring Grant Budget (million

332.34

420.43

504.74

544.09

657.12

708.90

Development 120.58

136.56

219.32

245.50

305.47

227.55

TA 0.19

1.25

1.87

5.22

2.33

Total 452.93

557.19

725.30

791.46

967.81

938.78

IDA financing of overall budget

49.86

28.20

67.12

38.28

22.55

3.80

% IDA financing 11% 5% 9% 5% 2% 0.4%


Page 49 of 54

ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS

None.


Page 50 of 54

ANNEX 6. SUPPORTING DOCUMENTS (IF ANY)

Bank preparation documents

World Bank. Islamic Republic of Pakistan. Country Partnership Strategy (CPS) for FY2010-2013.

World Bank. Statements of mission objectives, Aide-Memoires and Back-to-Office Reports from the identification mission carried out in April 2010 up to the appraisal mission carried out in December 2010.

World Bank. Quality Enhancement Review (QER) Panel of November 29, 2010. Minutes dated December 3, 2010.

World Bank. Project Appraisal Document (PAD) for the Islamic Republic of Pakistan: Tertiary Education Support Project (TESP) Report No. 59153-PK, February 24, 2011.

World Bank. Agreed Minutes of Negotiations dated February 10, 2011.

Bank project implementation documents

World Bank (IDA). Financing Agreement between the Islamic Republic of Pakistan and the International Development Association for the Tertiary Education Sector Support Project Credit 4887-PK (Portion A) and Credit 4886-PK (Portion B) - dated September 22, 2011.

World Bank (IDA). Project Agreement between the International Development Association and the Higher Education Commission for the Tertiary Education Sector Support Project Credit 4887-PK (Portion A) and Credit 4886-PK (Portion B) - dated September 22, 2011.

World Bank (IDA). Restructuring Paper on a Proposed Project Restructuring of the Tertiary Education Sector Support Project Credit 4887-PK (Portion A) and Credit 4886-PK (Portion B) – Report RES11280 dated November 25, 2013.

World Bank. Aide-Memoires and Implementation Status and Results Reports (ISRs) of all the supervision missions recorded in the Data Sheet of this ICR and in the Project’s electronic files from September 9, 2011 (ISR 1) to June 30, 2017 (ISR 12)

Higher Education Commission, Islamabad. Implementation Progress Report of the TESP May 31, 2017.

National University of Sciences and Technology (NUST), Risalpur Campus, Self-Assessment Report – Bachelor of Civil Engineering (2015)

Bank and Borrower other project implementation and post-implementation-related documents

World Bank. Islamic Republic of Pakistan. Country Partnership Strategy (CPS) for FY2015-2019, Report 84645-PK, dated April 4, 2014

World Bank. Islamic Republic of Pakistan. Implementation Completion and Results Report of the Higher Education Support Program (HESP). Report ICR00001579, June 14, 2010. Higher Education Commission (HEC), Islamabad. HEC Vision 2025. 2017 UNDP Human Development Report 2010.


Page 51 of 54

ANNEX 7. ASSESSMENT OF APPROPRIATENESS OF PDO AND INTERMEDIATE INDICATORS

The following are appropriate outcome and intermediate outcome indicators clearly and strongly linked to the PDOs: a) The trailing of the TTS (outcome indicator DLI 3), is an innovative performance-based of faculty

appointment gauging enhanced functioning and efficiency of the faculty members through a constructive competition among faculty members under financial security;

b) The QECs (outcome indicator DLI 8) assesses the universities and compiles self-assessment reports identifying strengths and weaknesses of institutional programs in teaching, learning and research and uses a performance-based scorecard for measuring progress of these QECs against the assigned targets so that the shortcomings in the process of self-assessment are identified and addressed;

c) The establishment of well-performing ORICs (outcome indicator DLI 6) allows for determining the degree of linkage effectiveness between the supply (the research done in the universities, commercialization efforts and incubator for future entrepreneurs) and the demand (how this research is being used by industry, commerce, services; self-generating revenue by the selling of prototypes and consulting and training services to the public and private sector; creation of self-employment);

d) The number of regular students in public and private higher education institutions (outcome indicator 3) is connected to the “enhanced access” part of the PDOs, and relevant in a context like Pakistan with a very low gross enrollment rate in tertiary education;

e) Budgetary allocation to HEC as percentage of GDP (intermediate outcome indicator 6) is a necessary requirement for setting the appropriate HEI conducive environment;

f) Number of technology companies established and incubated in HEIs (intermediate outcome indicator 8) as a way to generate revenue and link the HEI to the employment sector;

g) The share of universities’ self-generated revenue in overall total resources (intermediate outcome indicator 9) is an appropriate way to assess the sustainability of an HEI;

h) The intermediate outcome indicator 12 (DLI 4) concerning ACs meeting minimum quality standards, is a relevant indicator because these institutions nurture students after the 12th grade and provides the base for university education. Therefore, their role is pivotal in establishing the benchmarks of quality in tertiary education system. There are 1,993 ACs17; in the country lacking, by and large standards of quality, infrastructure, teaching and learning. Having recognized the above: (i) the original end-of-project target was ambitious and unrealistic to be achieved; and (ii) the benchmark instruments designed by HEC to achieve this DLI were not tailored to ACs but more to HEIs;

i) Number of students benefiting from a financial aid program, including a needs-based scholarship program and student loan (DLI 7), to increase the likelihood for their retention and completion of their HEI program;

j) Number of new postgraduate indigenous scholarships awarded (DLI 2) as a way to produce and retain PhD students;

17 In the administrative domain, the ACs are accountable to the Provincial government, while on the academic domain they are accountable to a given affiliated HEI.


Page 52 of 54

k) Number of new postgraduate foreign scholarship awarded (intermediate outcome indicator 4), that includes in its metric the percentage of those returning after completing their studies complying with the end-terms of their fellowship contract;

l) Number of HEI assessed against institutional performance evaluation standards (intermediate outcome indicator 18); and

m) Student-full time faculty ratio (with PhD) in the public university sector (intermediate outcome indicator 22)

The following are outcome and intermediate outcome indicators that are either not linked or weakly linked to the PDOs that could have been eliminated from the original RF or rephrased their metric: a) Intermediate outcome indicator 11 – number of revised and restructured undergraduate and/or

graduate programs in HEIs – lacking in the metric a mechanism to monitor their implementation at the corresponding HEIs;

b) Number of private TEIs eligible for public support as per approved policy (intermediate outcome indicator 16);

c) Improved strategic management, planning and accountability in public HEIs (DLI 10), resulting in the development of a Five-Year Strategic Plan and Annual Reporting by the HEI, without tracking the implementation of such plans beyond the request of the Annual Reports as a (unconvincing) proxy of monitoring; and

d) Number of HEC, Provincial education officials and HEI staff participated on staff development programs (intermediate outcome indicator 20), void of any tracking of what this staff is doing to improve teaching, learning and research after completing their training.

The following are outcome and intermediate outcome indicators that could have been merged into an overall and more relevant metric: a) Number of administrators in ACs participating in management training (intermediate outcome

indicator 7) could have been merged with DLI 4 under its revised metric at restructuring; and b) Number of HEI assessed against institutional performance evaluation standards (intermediate

outcome indicator 17) with number of public HEIs with fully functional Campus Management Solutions Module (CMS) (intermediate outcome indicator 19).


Page 53 of 54

ANNEX 8. Executive Summary of the Completion Project Report from the Government18

This report provides a summary of the activities completed under TESP, since its inception in November, 2011 to June, 2017. Initially, the progress on KPIs and DLIs (except for DLI-10) was tracked till December, 2015, only, when the MTDF-HE II was completed and the planned targets were mostly achieved. However, in order to ascertain sustainability of the initiatives supported under TESP, progress on KPIs and DLIs was further tracked as of March, 2017. The report also includes the summary of financial performance and the status of Eligible Expenditure Programs. Moreover, the report includes a summary of the best practices and lessons learnt during 2011-15 on certain DLIs. During the implementation period of MTDF-HE II, (ending on December 31st 2015), the HEC facilitated the HEIs and TEIs to achieve noteworthy results, under Component-1 of the project. As the project was extended, the performance on relevant indicators was tracked up to March 31, 2017 and it revealed that the TESP initiatives have not only been achieved but in many cases the performance surpassed the targets and continues to progress. Some significant results include the following: (i) 163 Quality Enhancement Cells have been established, with 90 of them working satisfactorily, on the basis of scorecard assessment; (ii) 49 Offices of Research Innovation and Commercialization (ORICS) have been established in universities against the projected target of 30; (iii) 1,027 academic programs have been accredited surpassing the target of 400; (iv) 4,500 postgraduate indigenous scholarships and 1,491 postgraduate foreign scholarships, under the UESTP and OSS Programs, have been awarded which meet the projected project requirements; (v) 128 curricula have been revised at undergraduate and postgraduate levels; (vi) Directorates of Distance Education (DDEs) have been established in 12 institutions; (vii) 180 technology companies have been established and incubated in HEIs; (viii) 20,161 need based scholarships have been awarded, which is far beyond the target of 6000; (ix) 70 HEIs have adopted planning and reporting mechanism through strategic business planning and annual progress reporting. As for the progress on Component-2 of the project, more than 4,000 participants from HEIs and HEC were trained under various thematic areas, related to MTDF, in the first phase, till December, 2015. Also, various research studies, such as Public Expenditures Tracking Survey, Students’ Satisfaction Survey, Employers’ Perception Survey, were conducted to improve future planning in higher education. Also, seven thematic studies were conducted to provide evidence-based feedback to the policy makers for designing HEC Vision 2025. During the extension period, the main focus of the project has been on provision of technical assistance, under the Component-2. Therefore, an extensive training program has been initiated, after conducting Level-3 Evaluation19 of the previous training program. By June, 2017, 1786 participants from HEC and HEIs have been trained under eight thematic areas. Also, an important study has been carried out to assess the current IT investments. Moreover, the project engaged a very experienced consultant to update the Quality Assurance Manual. During the life of the project, the project continued to place a strong focus on monitoring the results and

18 The complete version of the Completion Project Report prepared by the Government is available with the World Bank Team. 19 The Level-3 Evaluation has been conducted using Kirkpatrick Model and the detailed report can be accessed on

https://goo.gl/AELJtU.

https://goo.gl/AELJtU


Page 54 of 54

providing technical assistance to the HEC. For the purpose, a team of experts has been on board, including the M&E Specialists, Procurement Specialist, Quality Improvement Specialist, Financial Management Specialist and Training Specialist. The last review mission of the World Bank expressed its satisfaction with the performance of the Project. The report concludes that most of the TESP initiatives were sustainable and the HEC continues to make progress on the given indicators. Also, most of the initiatives, supported by TESP, have been incorporated in the Vision 2025. This shows the continued commitment of HEC to the objectives pursued by TESP.

Documents

Document of The World Bankdocuments.worldbank.org/curated/en/... · HERS Higher Education Reform Strategy HESP Higher Education Support Project IfR Interim unaudited Financial Report