IMPLEMENTATION COMPLETION AND RESULTS …...The World Bank Jamaica Social and Economic Inclusion of Persons with Disabilities (P144263) 03 08-Dec-2015 Moderately Satisfactory Moderately

Document of

The World Bank

FOR OFFICIAL USE ONLY

Report No: ICR00004577

IMPLEMENTATION COMPLETION AND RESULTS REPORT

ON A

SMALL GRANT

IN THE AMOUNT OF US$2.9 MILLION

TO

JAMAICA

FOR

JAMAICA SOCIAL AND ECONOMIC INCLUSION OF PERSONS WITH DISABILITIES (P144263)

NOVEMBER 30, 2018

Social Protection & Labor Global Practice

Latin America And Caribbean Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Regional Vice President: Jorge Familiar Calderon

Country Director: Tahseen Sayed Khan

Senior Global Practice Director: Michal J. Rutkowski

Practice Manager: Pablo Gottret

Task Team Leader(s): Asha M. Williams, Junko Onishi

ICR Main Contributor: Sarah Berger Gonzalez

ABBREVIATIONS AND ACRONYMS

BMIS Beneficiary Management Information System CCT Conditional Cash Transfer CPS Country Partnership Strategy ECD Early Childhood Development ESP Early Simulation Programme GOJ Government of Jamaica ICRR Implementation Completion Results and Report IFES International Foundation for Election Systems JCPD Jamaica Council for Persons with Disabilities JICA Japanese International Cooperation Agency MIS Management Information System MLSS MTR

Ministry of Labor and Social Security Mid Term Review

MTSEF Medium-Term Socio-Economic Framework NGO Non-Governmental Organization NDP National Development Plan NSPS National Social Protection Strategy PATH Programme for Advancement through Health and Education PHRD Policy and Human Resource Development PLR PSA

Performance and Learning Review Public Service Announcement

PWDs Persons with Disabilities RETF Recipient Executed Trust Fund RF Results Framework StW Steps-to-Work UN United Nations

TABLE OF CONTENTS

DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED.

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 1

II. OUTCOME ...................................................................................................................... 6

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 15

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 17

V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 22

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 25

ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 31

ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 32

ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 40

The World Bank Jamaica Social and Economic Inclusion of Persons with Disabilities (P144263)

DATA SHEET

BASIC INFORMATION

Product Information

Project ID Project Name

P144263 Jamaica Social and Economic Inclusion of Persons with

Disabilities

Country Financing Instrument

Jamaica Investment Project Financing

Original EA Category Revised EA Category

Not Required (C) Not Required (C)

Organizations

Borrower Implementing Agency

Ministry of Finance and the Public Service Ministry of Labour and Social Security

Project Development Objective (PDO) Original PDO

The Development Objective of the Grant is twofold, namely to (i) increase the employability and skills development of poor personswith disabilities (ages of 18-36 years); and (ii) improve the service delivery of special education needs to poor children withdisabilities between the ages of 0- 6 years.


FINANCING

FINANCE_T BL

Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$)

Donor Financing

TF-14258 2,897,790 2,897,790 2,719,625

Total 2,897,790 2,897,790 2,719,625

Total Project Cost 2,897,790 2,897,790 2,719,625

KEY DATES

FIN_TABLE_DATA

Approval Effectiveness Original Closing Actual Closing

17-Jul-2013 14-Aug-2013 14-Aug-2017 14-Aug-2018

RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s) Amount Disbursed (US$M) Key Revisions

02-Dec-2016 1.30 Change in Results Framework

31-May-2017 2.00 Change in Loan Closing Date(s) Change in Disbursements Arrangements

20-Dec-2017 2.00 Change in Components and Cost Reallocation between Disbursement Categories

KEY RATINGS

Outcome Bank Performance M&E Quality

Satisfactory Satisfactory Modest

RATINGS OF PROJECT PERFORMANCE IN ISRs

No. Date ISR Archived DO Rating IP Rating Actual

Disbursements (US$M)

01 29-Sep-2013 Satisfactory Satisfactory 0.00

02 21-Jan-2015 Satisfactory Moderately Satisfactory 0.34


03 08-Dec-2015 Moderately Satisfactory Moderately Satisfactory 0.68

04 06-Oct-2016 Moderately Satisfactory Moderately Satisfactory 1.30

05 13-Apr-2017 Moderately

Unsatisfactory Moderately Unsatisfactory 1.55

06 18-Sep-2017 Moderately Satisfactory Moderately Satisfactory 2.00

07 05-Dec-2017 Moderately Satisfactory Moderately Satisfactory 2.00

08 14-Aug-2018 Satisfactory Moderately Satisfactory 2.72

ADM STAFF

Role At Approval At ICR

Regional Vice President: Hasan A. Tuluy Jorge Familiar Calderon

Country Director: Francoise Clottes Tahseen Sayed Khan

Senior Global Practice Director: Arup Banerji Michal J. Rutkowski

Practice Manager: Mansoora Rashid Pablo Gottret

Task Team Leader(s): Francesca Lamanna Asha M. Williams, Junko Onishi

ICR Contributing Author: Sarah Berger Gonzalez


Page 1 of 28

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

Context

1. Jamaica is a pioneer in the Caribbean in recognizing the rights of Persons with Disabilities (PWDs). In 1973 the Government of Jamaica (GOJ) established the Jamaica Council for Persons with Disabilities (JCPD) which is responsible for implementing Government policies and programs for PWDs. The JCPD falls under the purview of the Ministry of Labor and Social Security (MLSS), while a multi-sectoral National Advisory Council on Disability appointed by the Cabinet advises the organization on policy-related matters. There are also several active NGOs in the country working and advocating for PWDs, most of which are associated with the Combined Disabilities Association. In 2000, Jamaica developed and considered a National Policy for Persons with Disabilities, with the anticipation of it eventually becoming law.1 Jamaica also ratified the UN Convention on the Protection of the Rights of Persons with Disabilities (2007) and is a signatory to the “Bill of Electoral Rights for People with Disabilities”, a project of the International Foundation for Election Systems (IFES). Jamaica has also ratified other related legislation including the Convention on the Rights of the Child (1989) and the International Covenant on Civil and Political Rights (1966).2

2. At the time this Recipient Executed Trust Fund (RETF) was prepared in 2013, Jamaica had a significant segment of the population with disabilities, many of whom faced obstacles to their social and economic inclusion. It was estimated that 15-17 percent of the Jamaican population, around 400,000 individuals (adults and children), had at least one disability. Of these, only 14 percent were employed.3 In 2013, adults with disabilities in Jamaica were also consistently facing social, physical and economic barriers to their effective inclusion in society. For children, nearly 16 percent of those between the ages of 2-9 years were classified as having at least one disability.4 The most frequently reported learning disabilities included: not understanding instructions (4.9 percent); mental incapacity (4.7 percent); and severe speech impediments (3.9 percent).5 Additionally, some 11.7 percent of children with disabilities 3-9 years were reported to have delayed speech, while 9.3 percent of children with disabilities aged 2 years could not name at least one object.6

3. In the past, resources and opportunities for persons with disabilities in Jamaica have been scarce, especially for poor persons with disabilities, making it challenging for many adults to be active members of society. In 2010/2011, the Programme for Advancement through Health and Education (PATH), the country’s flagship conditional cash transfer program, had just over 10,000 PWDs registered (or about 2.6 percent of total beneficiaries) and only a few individuals linked to the Steps-to-Work (StW) Program,7 an active labor market program which supports PATH beneficiaries with upskilling and seeking and retaining employment. While coverage of PWDs under the PATH has increased since the start of the Program in 2008, and some PWDs have been linked to STW, the MLSS acknowledged at the time of the Grant’s preparation that a more comprehensive and tailored approach was needed to link PWDs with quality and meaningful employment. Simultaneously, the Government recognized that poor children with

1 It would later become law in 2014, during the implementation of the Grant. 2 World Bank. 2013. Grant Funding Request: Social and Economic Inclusion of Persons with Disabilities. GFR 12233. Washington, DC. 3 Jamaica Council for Persons with Disabilities 4 Multiple Indicator Cluster Surveys (MICS). 2005. Jamaica. 5 Other types of disabilities are both mental and physical. 6 Ibid. 7 At the initiation of the Grant, only 5 PWDs were actively engaged in the StW Program.


Page 2 of 28

disabilities in Jamaica were highly disadvantaged, lacking access to early stimulation interventions because of the absence of information and limited access to specialized instruments to detect disabilities. While the MLSS, through the JCPD, administers an Early Stimulation Program (ESP) that targets children 0–6 years, providing screening services, early childhood education, as well as mental and psychosocial support to the families in order to increase the care of children with disabilities, its coverage was limited to Kingston and St. Andrews and at the time of the Grant, the program was only covering 60 percent of children with disabilities from PATH households.8

4. During the RETF’s preparation, Jamaica was finishing its National Social Protection Strategy (NSPS), a multi-sector, comprehensive approach rooted in Jamaica’s Vision 2030 and aimed at bringing together various policy strands for the protection and promotion of all Jamaican citizens and finalizing the building blocks for the establishment of a National Disabilities Act. The strategy, which benefitted from technical and financial support from the World Bank’s Social Protection Project (P105024), was highly consultative, drawing from public and private stakeholders across a number of social and economic sectors to define how Jamaica can best promote human capital development and economic participation of the poor and vulnerable and protect the entire population from shocks that negatively affect economic status and smooth consumption patterns.9 The strategy also embraced the ‘social protection floor’ approach, which defines a set of provisions that protect certain minimum guarantees of basic income security and social services for the most vulnerable populations: children, workers, elderly and persons with disabilities.10 Finally, the JCPD together with other entities working on increasing the rights, access and inclusion of persons with disabilities, were in the process of finalizing a White Paper for a Disabilities Act to present to Parliament, which built on the National Policy for Persons with Disabilities considered by Parliament in 2000.

5. At the same time, there was a heightened awareness about the need to actively include disability rights in the development and social and economic growth agenda. The Grant’s development also came at a time when there was increasing recognition among key development players that persons with disabilities have generally poorer health, lower education achievements, fewer economic opportunities and higher rates of poverty than people without disabilities. The World Bank and the World Health Organization jointly authored the World Report on Disability in late 2011 in an effort to contextualize and operationalize the best available evidence about what works to overcome barriers to health care, rehabilitation, education, employment, and support services, and to create the environments which would enable people with disabilities to flourish.11 Similarly, the United Nations published a ‘how to’ guide to Including the Rights of Persons with Disabilities in United Nations Programming at Country Level, pointing out ways to mainstream disability rights in country-level social protection programs and national plans.12

6. Within this context, the RETF was designed to finance a more comprehensive approach to socially and economically include poor persons with disabilities in employment and employment-

8 World Bank. 2013. Grant Funding Request: Social and Economic Inclusion of Persons with Disabilities. GFR 12233. Washington, DC. The ESP provides developmental assessment, home intervention, special early childhood education, physiotherapy, speech therapy, counselling and numerous programmes and workshops for parent. In 2013, it was providing services to a little over 1,400 children. It does not turn away any child. 9 Planning Institute of Jamaica. 2014. “National Social Protection Strategy.” Kingston, Jamaica. 10 Ibid. 11 World Health Organization and World Bank. 2011. “World Report on Disability.” Geneva, Switzerland. 12 United Nations Development Group. 2011. “Including the Rights of Persons with Disabilities in United Nations Programming at Country Level.”


Page 3 of 28

related opportunities, as well as to link children with disabilities to enhanced early simulation services and activities to improve their readiness for entering school. To that end, the activities under the RETF aimed to improve the capacity of the Government, non-governmental organizations (NGOs), civil society and private sector to mainstream disability rights and services in service delivery systems, including improved access to education and job skills trainings, increasing entry into the job world, and piloting innovative methods and approaches to working with PWDs in Jamaica in an effort to identify successful interventions for future scaling up. The RETF was a pioneering effort by both the World Bank and the Recipient to address social and economic inclusion of PWDs through training and ECD interventions, given limited experience in delivering similar projects for this population.

Theory of Change (Results Chain)

7. International evidence called for a multi-pronged approach mixing demand- and supply-side interventions to achieve rapid results and to enable persons with disabilities to be more actively included in Jamaican society. Skills development is a critical factor in enabling persons with disabilities to take part in the labor force. Those that have had the opportunity to gain marketable skills have shown it is possible for them to earn a living and contribute to the world of work.13 Yet access to appropriate skills training in Jamaica has not formally been available or affordable to poor and extremely poor disabled women and men. At the time of the Grant’s preparation, inclusive policies were not fully in place and mainstream training programmes had failed to encourage or accommodate the participation of poor PWDs in the work force. The lack of inclusion of PWDs begins at a young age, where parents and guardians often delay the initial screening visits to get a confirmed diagnosis of a child’s disability, and once confirmed, these adults are often at a loss on how to engage or work with their child. As a result, many children with disabilities in Jamaica never have the opportunity to succeed. Experiences from OECD countries and increasingly developing countries suggest that the inclusion of persons with disabilities must take a multi-pronged approach with supply-side interventions (training, coaching/guidance counseling, therapy, counseling for parents of children with disabilities) and measures aimed at increasing the quality and coverage of services for children and adults with disabilities.

8. The Grant was designed along a results-chain in which targeted investments in institutional capacity and persons with disabilities aimed to contribute to the ultimate long-term achievement of increasing economic and social inclusion of children and young adults with disabilities. With limited experience in this area, a robust management information system would be created to support the monitoring and evaluation of these interventions.

Result 1: Increased Skills Development and Employability of persons with disabilities: The Grant financed skills training for poor persons with disabilities between the ages of 18 and 36 years. The theory of change maintained that investments in skills training institutions (financial, human and administrative capacity), stipends, skill development, job coaching, and job attachments for trainees would increase the short (contractual) and long-term (permanent) employment of persons with disabilities benefitting from PATH and aged 18-36 years in Jamaica. Intermediate outputs included the completion of a labor market study that analyzed the supply- and demand-side elements of hiring PWDs, the completion of a training capacity assessment to identify possible organizations to carry out the skills development and training and assess their capacity to deliver training, and an increase in the percentage of PWDs who complete the skills/entrepreneurial training activities upon enrolment. Critical assumptions included (i) employers

13International Labour Organization. 2008. Recognizing ability: The skills and productivity of persons with disabilities- Literature Review.


Page 4 of 28

in the public and private sector would be willing and able to hire persons with disabilities (especially those for persons with physical disabilities); (ii) national policies were conducive to employers hiring persons with disabilities (iii) PWDs to be targeted under this Grant were ready for skills development and training; (iv) job growth would remain positive; (v) the skills developed would be relevant to the sectors hiring; and (vi) PWDs benefitting from PATH seeking employment would have access to other social services that reduce barriers to employment (child care, transportation, uniform for work, etc.).

Result 2: Improved service delivery for poor children (aged 0-6 years) with special education needs: The Grant financed skilled human resources for the provision of physiotherapeutic, rehabilitative and cognitive stimulation services to children with disabilities, adaptive aids, and sensitization workshops to parents and caretakers of children with disabilities. The theory of change argued that critical investments in the Early Simulation Programme (ESP) would improve the readiness of poor children with disabilities aged 0-6 years for school. Critical assumptions include that (i) human resources to provide the simulation services were readily available in Jamaica; (ii) parents were willing to commit to and capable of attending the sensitization workshops; and (iii) there would be no problems with the procurement of adaptive aids.

Project Development Objectives (PDOs)

9. The Development Objective of the Grant was to: (i) increase the employability14 and skills development of poor persons with disabilities (ages of 18-36 years); and (ii) improve the service delivery of special education needs to poor children with disabilities between the ages of 0- 6 years.

Key Expected Outcomes and Outcome Indicators

10. Key indicators used to measure the outcomes of the multiple interventions to boost the skills development of poor persons15 with disabilities and service delivery of special education needs to poor children with disabilities included:

Persons with disabilities who receive employment (work experience) after completing classroom training

Persons with disabilities trained under the project who have gained employment16 within twelve (12) months after completion of the program

Poor Children (0-6 years) with disabilities in Early Stimulation Program improved readiness17 for school

Components

11. The Grant activities were separated into four components:

Component 1: Skills Development of Persons with Disabilities. The objective of this Component was to improve the employability of poor persons with disabilities aged 18-36 years old through the provision of competency-based skills training in a range of job skills, and remedial education. The Component financed the organization of skills training programs for targeted PWDs, including the training

14 For the RETF, employability is defined as “the individual’s ability to seek and retain employment.” Japan Policy and Human Resources Development (PHRD) Technical Assistance (TA) Program to Support Disability and Development Part I: Summary of Grant Funding Proposal 15 Enrollment in PATH was used as a proxy for identifying poor PWDs. 16 In the Grant, this is referred to as permanent employment (including self-employment). 17 Readiness for school refers to a child with a disability receiving at least one of the following interventions: speech therapy, physical therapy or assistive aid.


Page 5 of 28

provision contracts, per diem for the participants, and, in some cases capacity building for the NGOs responsible for providing the training. Training was provided in four categories: skills; apprenticeship and on-the-job training; job coaching; and organizational strengthening. Additionally, a number of assessments were carried out to support the skills development and training. Among these included a training needs assessment of PWDs in PATH; an assessment of training providers; rapid assessment of Component 1 skills training activities; and a labor market study. The Component was amended during the December 2015 mid-term review (MTR) to allow financing, through the JCPD, consultancies to develop Codes of Practice for Employment, and Education and Training in support of Jamaica’s National Disabilities Act that came into effect in May 2014.

Component 2: Strengthen the service delivery of special education to children with disabilities. Component 2 aimed at enhancing the capacity of Jamaica’s Early Simulation Program (ESP) to provide assessments and early intervention services for poor children with disabilities from birth to six years old. The Component financed early stimulation services including physiotherapeutic, rehabilitative and cognitive stimulation to children with disabilities, early screening, assistive aids, and training workshops to parents and caretakers of children with disabilities, including an accompanying training curriculum. The Component also financed capacity building of ESP staff and a needs assessment of the ESP.

Component 3: Monitoring and Evaluation. Component 3 financed the monitoring and evaluation of the Grant’s activities. The Component originally aimed to finance baseline and endline surveys for the interventions in Components 1 and 2, process and outcome evaluations to ascertain the short- and long-term effects of the Project, and the development and implementation of a Management Information System (MIS) for the ESP and training to employment intervention. In the end, it financed off-the-shelf software for the development of a MIS for the ESP and two outcome assessments, measuring the effect of the interventions in Components 1 and 2.

Component 4: Project Management. Component 4 financed the management and supervision of the activities under the Grant.

Main Beneficiaries

12. The Grant’s main beneficiaries were persons with disabilities. Activities were targeted at poor adults with disabilities aged 18-36 years of age (300 persons with disabilities) and poor children with disabilities aged 0-6 years of age (352 children, or 40 percent of the 880 poor children identified) and their parents or caretakers (250 parents). To ensure that the Grant was targeting poor PWDs, the majority of Grant beneficiaries were also beneficiaries of the PATH program. Secondary beneficiaries were the institutions providing the skills training and entrepreneurial courses and staff at the ESP program.

Significant Changes during Implementation

13. The Grant was restructured three times, which contributed to improvements in the Results Framework (RF) and allowed for the effective management and completion of Grant activities. None of the changes impacted the theory of change or the Grant’s scope. The first restructuring, in December 2016, made minor revisions and additions to the Grant’s RF, including the revision of the wording of two PDO indicators, the addition of end targets for some of the indicators, and, the revision of end dates for targets to align to the Project’s completion date (Table 1). During a mid-term review (MTR) of the RETF in December, 2015, it was agreed that some of the results targets originally established were too ambitious, given that the target population was difficult to reach and the employment context in Jamaica was overall extremely challenging overall, but especially for persons with disabilities. Additionally, the design of


Page 6 of 28

activities under Component 1 included mandatory six-month job attachments for beneficiaries, but this was not being appropriately captured by the original wording of the PDO-level results indicator for this Component. A baseline was also added to the intermediate indicator, “increase in rehabilitative sessions” during the restructuring.

14. The second restructuring (May 2017) extended the Grant’s closing date by 12 months from August 14, 2017 to August 14, 2018 to allow for the full disbursement of the RETF and ensure achievement of the PDOs. The restructuring also added ‘goods’ to the category of eligible expenditures under the monitoring and evaluation Component 3, to facilitate the procurement of an off-the-shelf software solution for the MIS. The last restructuring in December 2017 reallocated funds from Component 3 (Monitoring and Evaluation) to Component 4 (Grant Management). The cost of the monitoring and evaluation activities was reduced due to the reduction in the number and complexity of the evaluations as well as the procurement of an off-the-shelf software for a management information system for the ESP (rather than the creation of a new system) under Component 3. Resources were reallocated to Component 4 to allow for the Grant to be adequately supervised through the revised closing date.

Table 1. Original and Revised PDO indicators and end targets

Original Indicator Original Target Revised Indicator Revised Target

PDO Indicator, Component 1: Persons with disabilities who receive employment after completing training

300 persons PDO Indicator, Component 1: Persons with disabilities who receive employment (work experience) after completing classroom training

No change.

PDO Indicator, Component 1: Persons with disabilities trained under the project are able to retain employment over six months after completion of the program

80 percent PDO Indicator, Component 1: Persons with disabilities trained under the project who have gained employment within twelve (12) months after completion of the program

40 percent

PDO Indicator, Component 2: Poor children (0-6 years) with disabilities in Early Stimulation Program improved readiness for school

50 percent No change. 40 percent

II. OUTCOME

Assessment of Relevance of PDOs

15. Relevance of the Grant objectives is assessed as high. Upon closure, they remained highly relevant to realizing the development objectives of the World Bank’s Country Partnership Strategy (CPS). Although this was a RETF, the PDOs continued to be well aligned at the Grant’s closure with Jamaica’s Performance and Learning Review (PLR) of the CPS (Report No. 112663-JM), an extension of the


Page 7 of 28

CPS FY14-17 (Report No. 85158-JM), which is deeply anchored in Jamaica’s long-term National Development Plan (NPD), “Vision 2030 Jamaica”. The PLR of the CPS maintains the objective of promoting sustained and inclusive growth through the provision of select solution-oriented support under three pillars: (i) public sector modernization; (ii) development of an enabling environment for private sector growth; and, (iii) strengthening social and climate resilience. Under Pillar 3, two of the three sub-objectives promote the strengthening social protection programs as well as increasing opportunities for the poor and vulnerable. The outcomes of the Grant therefore remain central for realizing the objectives set out under the PLR’s Pillar 3.

16. The PDOs also remained overwhelmingly pertinent to the Government’s national strategic efforts in the Social Protection Sector. At closure, the PDOs also resonated strongly with the Government’s National Social Protection Strategy (NSPS) and Medium-Term Socio-Economic Framework (MTSEF) 2015-2018. The NSPS thoroughly acknowledges that PWDs are one of the most vulnerable population groups in Jamaica and promotes the creation of active and inclusive policies for children, youth and adults with disabilities. Moreover, the MTSEF boasts Effective Social Protection as one of the 12 national outcomes to achieving a Jamaican society that is safe, cohesive and just through: (i) the creation of measures to identify and create employment and business opportunities for the unemployed and underemployed; (ii) the improvement of targeting of and provision of benefits to the needy; and, (iii) the increased participation of persons with disabilities in all spheres of national life.18

Table 2: Results Summary19

PDO Indicators Revised Target Actual Achieved at Completion

Persons with disabilities who receive employment (work experience) after completing classroom training (number)

300 384

Persons with disabilities trained under the project who have gained employment within twelve (12) months after completion of the program (percentage)

40.00 12.48

Poor children (0-6 years) with disabilities in Early Stimulation Program show improved readiness for school (percentage)

40.00 47.27

Intermediate Indicators Revised Target Actual Achieved at Completion

Labor market study completed 1 1

Completion of the Assessment of Training Capacity 1 1

PWD completing skill/entrepreneurial training activities upon enrolment (percentage)

80.00 95.55

Increase in rehabilitative sessions received (percentage) 10.00 7650.00

Parents of PATH beneficiaries trained on how to cope with children with disabilities (number)

250.00 259.00

PATH children in ESP who receive adaptive aids (percentage) 60.00 14.72

18 Government of Jamaica. 2015. Medium Term Socio-Economic Framework 2015-2018. Planning Institute of Jamaica. 19 Please see Annex 1 for detailed Results Framework.


Page 8 of 28

Assessment of Achievement of Each Objective/Outcome

17. The overall efficacy rating of the Grant was substantial, almost achieving all of its intended outcomes. Over the last five years, the Grant has had a notable impact on the lives of impoverished persons with disabilities in Jamaica, increasing access to economic opportunities for PWDs aged 18-36 that had not existed in the past and improving the readiness of poor children with disabilities aged 0-6 years for entering primary school, achieving two of the three PDO indicators (discussed further below). It has also given parents of children with disabilities an opportunity to be more informed, engaged and supportive of their children’s wellbeing, increasing the child’s likelihood of being more involved and included once in school. The Grant also contributed to improvements in the capacity of institutions providing and implementing services for PWDs through the expansion in the number of personnel, the strengthening of the NGOs and individuals’ knowledge, abilities, and commitment to carrying out the work in the sector, and the creation of a more coordinated approach to working with PWDs in Jamaica. While there were minor shortcomings in the implementation efficiency (discussed under Section III, Key Factors Affecting Implementation), the Grant substantially achieved its objectives.

18. The first objective, to increase the skill development and employability of poor persons with disabilities was achieved, exceeding its target for the first PDO indicator, and meeting the targets for all three intermediate indictors. At the Grant’s closure, four rounds of training (each one year-long) were completed, with 384 PWD receiving 6-month work experience and on-the-job work training with employers after completing classroom training in areas related to their training including housekeeping, cosmetology, customer service, furniture making, and data operations, among others. This PDO result exceeded its target of 300 persons. Moreover, all three of the intermediate results indicators were achieved. The completion of a Labor Market Study was carried out to better understand the challenges and perceptions of employing PWDs in Jamaica, meeting the intermediate indicator. Overall, the study confirmed that many employers hold negative perceptions about the employment abilities of PWDs in the workplace. However, the study also reinforced the ability of PWDs to retain employment and be assets to their employers. The study also pointed to several areas for improving policy related to the employment of PWDs and for the need to improve information and education to potential employers about the skills and potential of PWDs. A second intermediate indicator was met with the completion of the Assessment of Training Capacity/ Training Needs Assessment of PWDs in PATH. The results from the assessment proved helpful in identifying the educational and skill deficits and needs of PATH PWDs. It also informed the training and skills development courses carried out under the Grant for beneficiaries. Finally, the last intermediate indicator related to the training program’s completion rate was achieved and exceeded its target, with 95.5 percent of PWDs completing the skill training activities upon enrolment, well above the 80 percent end target. This demonstrated substantial engagement with the PWDs throughout the program, beyond what was expected for this traditionally disenfranchised and socially excluded group.

19. Despite these achievements, one of the PDO indicators for this first objective was not met. At the Grant’s closure, only 12.84 percent of persons with disabilities aged 18-36 years were permanently employed within 12 months of completing the training program, falling short of the targeted 40 percent.20 While the result was far from meeting the target, the result is in fact in range with other short term training programs for able-bodied persons, which are much lower than 40 percent.21 This shows that the

20 Of the 601 individuals that participated and completed the training over the four rounds, 75 of them received full-time employment. 21 For example, in the Dominican Republic, short-term skills training programs (less than 12 months) for able-bodied youth increased formal employment by 18 percent and higher earnings among women (25-31 percent); and, in Colombia’s Jovenes


Page 9 of 28

indicator’s target was highly ambitious, even after the Grant’s restructuring. Although the restructuring cautiously revised the target for this indicator down to 40 percent, from its original target of 80 percent, it was likely still too ambitious. Further, it is likely that market factors beyond the Grant’s control combined, with perceptions of some employers22 inhibited many PWDs from gaining employment within 12 months of the Program’s completion. While training providers were able to successfully secure work experience/ job attachments with employers for trainees (PDO indicator 1), this was fully subsidized through stipends provided by the Grant for this phase. Employment beyond this phase would largely depend on employer demand and willingness to hire PWDs. For those who did not gain employment following training, eligible PATH beneficiaries would continue receiving PATH grants through their participation in the program; while services and support to PWDs will continue to be offered through JCPD, which falls under MLSS, and where feasible, by the NGOs that provided training.

20. The implementation of the Grant also contributed initial analysis to develop two Codes of Practice related to Employment, and Training and Education to support operationalization of the Disabilities Act of 2014. This activity was added following a request made at the RETF’s Mid-Term Review (MTR), given the legislative passage of the Disabilities Act in 2014, and was intended to finance the full development of these two Codes of Practice. In May 2014, nearly a year after the Grant was effective, the GOJ passed the Disabilities Act recognizing that PWDs have the same fundamental rights as any other person, promoting individual dignity, freedom of choice and independence, making sure that PWDs are able to take part effectively and are included on an equal basis with others in all aspects of national life, and preventing and prohibiting discrimination.23 Furthermore, it supported the link between the skills development of poor PWDs and employment. With the passage of this Act, JCPD expressed urgency to begin work on the development of seven Codes of Practice to accompany the Act. A request was made at the MTR for assistance to develop two of these Codes of Practice, which were most relevant to the first result of the RETF, (Education and Training, and for Employment). This resulted in the addition of a consultancy to initiate the development of two of the Codes of Practice. By the Grant’s closing, although a gap analysis was completed and accepted, the development of the two Codes of Practice were not yet finished due to Government dissatisfaction with the drafts developed and the need to reopen the procurement process. The activity remains a priority and will be financed with Government resources, along with the development of the other five Codes of Practice for the operationalization of the Disabilities Act.

21. Finally, a communications campaign was carried out toward the end of the Grant’s implementation as a first step to improving employer perceptions of PWDs in the workforce. The assessment reports and training experiences under Component 1 highlighted the need for improving awareness about the employment potential of PWDs in the workplace. Although not planned at the design of the RETF, resource availability near the Grant’s completion facilitated implementation of the communications campaign to address the issues highlighted by the assessment reports and feedback from the training. The campaign promoted employment opportunities for PWDs, with Public Service Announcements (PSAs) illustrating actual employment experiences and achievements for each of the major disability groupings (blind, deaf, intellectually challenged and physically impaired). It also

program, short-term training of able-bodied youth increased permanent employment by 7 percentage points and earnings among women (20 percent). Arias, Omar. 2017. “Maximizing the impact of short-term training programs.” World Bank presentation at the Labor Market’s Course. 22 As noted in the Grant-financed Labor Market Study. 2016. 23 Government of Jamaica. 2014. “Disabilities Act.” http://admin.jcpdja.com/upload/339_TheDisabilitiesAct.pdf

http://admin.jcpdja.com/upload/339_TheDisabilitiesAct.pdf


Page 10 of 28

highlighted the achievements of work carried out under the Grant as a means of showing that with short-term training, PWDs are capable of being productive workers. Finally, it supported a one-day Employer Sensitization Workshop, bringing together employers from St. Andrews and Kingston to discuss and share information on issues of social inclusion of PWDs. A tool kit was developed and disseminated at the sessions that provided information to employers on how to engage and retain PWDs in employment solutions. At the Grant’s closure, the MLSS continued to finance these efforts and planned to carry out additional employer sensitization workshops elsewhere in the country to continue to promote the economic and social inclusion of PWDs. Furthermore, the PSAs and materials will remain relevant for use as strategic communications resources well-beyond the Grant’s closure.

22. The second objective, to improve the service delivery of special education needs to poor children with disabilities between the ages of 0- 6 years was substantially achieved, exceeding the end targets for the PDO indicator and two of the three intermediate indicators. This objective was challenging to achieve given the overwhelming need and number of PATH children with disabilities in Jamaica and the limited human and physical (assistive aids) resources available. Of the 880 PATH children with disabilities in ESP aged 0-6 years identified, 416 children participating in the ESP had improved readiness24 for school, or 47 percent, exceeding the target of 40 percent. Moreover, twenty-four educational assessments, the basis for placing children with disabilities in primary schools, have been completed for children enrolled in the ESP.

23. The Grant contributed to notable improvements in the Early Simulation Program through strengthening the quality of the education of PATH parents with children with disabilities and attendance, increasing the capacity of the ESP staff, and improving the quality, accessibility and scope of the services provided through the ESP. Through the Grant, a curriculum for parents and caretakers of children with disabilities was reviewed and updated and 259 PATH parents were trained on how to cope with children with disabilities, exceeding the target of 250 parents. Additionally, notable investments were made in the staff of the ESP. The Grant financed capacity building for 41 staff at the ESP through a Staff Development and Planning Workshop in October 2014 as well as year-long professional development courses at the University of the West Indies’ School of Continuing Studies. Program areas included General Psychology, Principles and Techniques of Counselling, Child Psychology and Development, Supervisory Management, Practical Home Nursing, among others. Training combined with the hiring of additional staff generated an unforeseen increase in the number of rehabilitative sessions, from 52 total to 3,978 sessions, resulting in an increase of 7,550 percent,25 far exceeding the intermediate indicator’s end target of a 10 percent increase (or 57 sessions).26 It is worth noting that the Inter-American Development Bank (IDB) had a project parallel to this Grant that financed new infrastructure and the rehabilitation of existing infrastructure where the ESP is implemented in Kingston, Jamaica.

24 Under the Grant, readiness for school refers to a child with a disability receiving at least one of the following interventions: speech therapy, physical therapy or assistive aid. 25 It is worth noting that the results framework miscalculated this number, putting 7,650 percent as the final outcome. This was not able to be reconciled in the Bank’s portal prior to the final ISR completion. 26 The unit of the baseline is the number of 30-minute sessions. The baseline is reported as a number, to provide the base on which the increase in sessions would be measured. The unit for the end target is percentage increase, so 10% increase of 52 sessions is 57 sessions. The current number of rehabilitative sessions is 3,978 sessions. The increase to 3,978 from the project baseline of 52 yields a 7,550 percent increase, far exceeding the 10 percent that was projected. This is far greater than the number of sessions projected due to the significant expansion of therapeutic services (speech therapy, physiotherapy, etc.) which were not being offered before the Grant's execution.


Page 11 of 28

24. Despite these achievements, one intermediate indicator under Component 2 was not met. The percent of PATH children in ESP who receive adaptive aids was only 9 percent of the 550 children with disabilities targeted to receive this support, missing the target of 60 percent. This was the result of persistent procurement challenges with sourcing these aids on the local market and completing procurement processes with potential suppliers. These issues are discussed in more detail in Section III.

Rationale for Not Applying Split Ratings

25. Although the Project’s outcome indicators and targets were formally revised, split ratings to assess both the original and revised objectives and associated outcomes did not apply in this instance. The rationale for not applying an assessment of split ratings for the Grant are as follows:

a. Given the limited international experience for delivering similar projects addressing early childhood development (ECD) and skills development of disabled poor youth, there were no comparative experiences to set the original PDO indicator targets. Therefore, the Government made a good faith effort to set realistic targets at the design stage, but once implementation started, it was clear that these were too ambitious given the complexity of the project. The project was restructured to set more realistic targets, based on a better understanding of the environment and the context for delivery of these services. In addition to setting targets that more appropriately captured what could be expected for the sector, the changes to the PDO indicators clarified what the indicators aimed to measure (namely the definition of ‘employment/work experience’ under PDO indicator 1 and the timeframe for measuring employment under PDO indicator 2). This in no way changed the substance of the PDO indicator but did make it clearer.

b. Although the targets for two indicators were reduced, the overall scope of the project (coverage, financing, activities) remained the same.

Efficiency and Rating

26. The overall efficiency of the Grant is substantial.27 Investments in the skills development component made up 59 percent of the total Grant, while spending on the ESP made up about 25 percent of the total Grant. The remainder of the US$2.9 million went toward monitoring and evaluation and Grant management. Team argues that the overall implementation efficiency is what would be expected in the SP sector. That said, the implementation of a Grant that explicitly uses the Recipient’s CCT program (PATH) to target and benefit poor persons with disabilities is the first of its kind in Jamaica. As such, many of the interventions and activities supported under the Grant had to be implemented in an innovative way, and, with any innovation and the testing of new practices there are opportunities that yield positive net gains and also unforeseen challenges. These are discussed below.

27. Under the first intended outcome, increased skills development and employability of persons with disabilities, implementation efficiency was deemed substantial, that is, meeting the efficiency expectations of the sector. An initial hurdle to implementing the skills training for PWDs was identifying the institution(s) most qualified and able to work with PWDs. This caused a minor implementation delay, but eventually led to increased efficiency in the delivery of the skills training. Prior to the Grant’s effectiveness (but after it was approved internally by the World Bank), in mid-July 2013, the GOJ brought

27 This efficiency assessment is focused on implementation efficiency. As this is a small RETF, an economic analysis was not done ex-ante, and at the Grant’s closure, the team was unable to carry out an ex-post cost-benefit analysis, due to data constraints and comparable evidence for similar activities for the sector.


Page 12 of 28

together over 250 individuals from all over Jamaica for a Labour Inclusion Conference. This conference was a defining moment for the Grant and for Jamaica as various stakeholders from the disability sector, human resource practitioners, and other Government and Non-Governmental Organizations (NGOs) gathered for the first time in many years, acknowledging the experience and services that existed and defining an action plan around PWDs in the workplace.28 For this objective, the Grant had originally envisioned a single organization to deliver the training and skills development across the island, namely the HEART Trust/ National Training Agency (NTA),29 Jamaica’s primary training and skills development organization. HEART/NTA had been providing skills development and entrepreneurial training to PATH beneficiaries under Steps-to-Work for a few years nationwide and understood the importance of tailoring some of its curriculum to meet beneficiaries where they were in terms of education levels and experience. However, after initial conversations, HEART/NTA noted that they did not have the physical capacity to work with PWDs across the island nor the technical expertise to deliver courses on all the thematic areas defined under the Component.30 Subsequently, the Government decided to split the procurement into multiple contracts and put out a call for proposals for each of the thematic areas. The result was the contracting of various NGOs and other public entities who had a track record of training and coaching persons with disabilities (although small in scope) and linking them with employment. Among them included: the Abilities Foundation, Development Options, the Jamaica Association on Intellectual Disabilities (JAID- located in Kingston, St. Ann, and Manchester), Portmore Self Help Disabilities Organization, and the Jamaica Society for the Blind were natural fits.

28. The contracting of various NGOs and public entities already working with PWDs was innovative for the sector and contributed to improvements in the institutions’ capacity to operate. Never before had the GOJ contracted NGOs or public entities to deliver training or skills development services to PWDs. This posed financial and procurement challenges in the beginning, with each organization operating under different rules. In the end, individual contracts with each of the organizations selected to carry out the training combined with financial management training improved the day-to-day operational capacity of these organizations.31 Additionally, HEART/NTA worked with the various NGOs and public entities to ensure that they became accredited training institutions. The accreditation not only formalized the organization’s role in training of PWD in Jamaica, but it gave prominence to the certificate of completion for PWDs who would later present the certificate to potential employers.

29. Two of the institutions carrying out the training and skills development hired a job coach early on as a means of more effectively linking trainees with employment opportunities. Although not foreseen in the original design, the Grant recognized early on in carrying out the skills development that there was a need for a job coach or job counselor - someone who could serve as an advocate for the trained individual, link them with employers seeking the relevant skills, and carry out coaching to help them prepare for employment opportunities. Under the Ministry of Labour and Social Security, the

28 Benjamin, Chris. 2013. “Labour Market for Inclusion of Persons with Disabilities.” Kingston, Jamaica. The five areas the action plan focused on were: Strengthening Support Services for Persons with Disabilities; How to get better participation from Employers; Increasing workplace participation of PWDs; Increasing accessibility for PWDs; and, Strategies to facilitate PWDs in the workforce. 29 The HEART Trust/NTA is active in shaping a vibrant and dynamic workforce through the support, maintenance and development of Technical Vocational Education and Training institutions in Jamaica. https://www1.heart-nta.org/ 30 These included Skills Training, Job Coaching and Employment Readiness Training, Apprenticeship and on the Job Training, and Organizational Strengthening. 31 At one point, there was an attempt to create a consortium to ensure that all would operate under similar guidelines; however, there were challenges in getting all NGOs on board.


Page 13 of 28

Electronic Labour Exchange (ELE), the Ministry’s public employment service, carries out this role for general job seekers; however, it has never focused explicitly on linking PWDs with employers. The Grant, through the job coaches, filled this void for some trained PWDs. Qualitative evidence during the ICRR mission showed that this had a positive impact. The Grant Manager in Jamaica noted that many of the PWDs still employed at the end of the Grant likely graduated from the NGOs or public institutions with job coaches. Perhaps with additional resources, the impact and benefit of the job coaches could have been evaluated.

30. The second intended outcome, improved service delivery for poor children (aged 0-6 years) with special education needs, was deemed to be modestly efficient, just below expectations for the sector. Delays and challenges in the procurement of assistive aids for poor children with disabilities proved to be an impediment to meeting the intermediate result and disbursing the remaining funds for component 2. At the start of the Grant, the procurement process for assistive aids was delayed for fear of not having the storage space once purchased; however, this no longer became an issue once the process got started and the MLSS realized that the procurement of the aids would be much more complex. Much of the equipment could not be found locally; and, if it was found locally, there were few vendors and the items were expensive. As a result, the MLSS, in some cases, used international contracting to purchase the assistive aids needed for some of the children, prolonging the time needed to identify a competitive vendor, purchase the goods, and eventually receive them. This needed to be done for the purchase of an additional 165 assistive aids identified for poor children with disabilities enrolled in ESP.32 Procurement processes in Jamaica, in general, are arduous and slow due to the highly bureaucratic approval process which often makes the time for processing many contracts extremely lengthy. Subsequently, the need to carry out an international bidding for the assistive aids combined with Jamaica’s procurement processes caused delays in the purchase and provision of the aids. After a lengthy bidding process for the aforementioned 165 assistive aids (the bidding process concluded in September 2017), the vendor was unable to deliver the goods resulting in the Grant Management team in Jamaica renegotiating the contract with the same vendor weeks prior to the Grant’s closing, eventually having to cancel the contract and leaving undisbursed funds under Component 2. Arguably, some of the procurement delays could have been avoided by exploring alternative options for addressing the delays well before the closing date.

31. Enhancements in the methodology and timing of the educational workshops for parents of children with disabilities, though, made the implementation more accessible and impactful. One of the intermediate outputs of investment in the ESP was to work with parents with children with disabilities and to provide them with tools to improve their engagement with their children at home and better prepare them for the transition to school.33 The Grant-funded ESP Institutional Review showed there was a need to improve support for parents of children with disabilities. As such, the Grant invested in improving the methodology and coverage of the workshops. One of the challenges at the start of the Grant was that many PATH parents registered for the course and participated in at least the first module but were unable to complete it due to transportation costs (the course provided meals and child care services) and the timing of the course. Shortly after this feedback, the ESP workshop administrators

32 At the Grant’s closure, the Grant had only procured 81 assistive aids. The procurement of these aids had been done through various vendors on an on-demand basis which proved time consuming and worked well in instances where the aid was available locally and easy to procure. The 165 aids in the last procurement would have been through a single vendor as these assistive aids were not available on the local market, and would have been in addition to the 81 aids. 33 In Jamaica, except for the School for the Deaf and School for the Blind, there are not special programs or schools for children with extreme disabilities.


Page 14 of 28

worked to ensure that transportation costs were covered for some and that the timing of the workshops was done when it worked best for parents, some of whom were working and others who were caretakers of other children or elderly. Qualitative evidence gathered during the ICRR mission through individual and group meetings with parents participating in the workshops, suggested that the workshops were implemented well and will have lasting effects on the way parents communicate with their children and engage with them on a day-to-day basis.

32. Initial delays in the finding and hiring qualified personnel for the delivery of supportive services for the ESP, in the end, did not hinder the impact of the services provided. Jamaica has a limited number of persons in the country with qualifications to carry out the supportive services needed in the ESP (speech therapist, physical therapist, and interpreters). During the Grant’s implementation, only five speech therapists were identified in the country, and many had their own private practice. As a result, for the Grant’s implementation, the GOJ had to creatively find ways to employ personnel to provide the necessary ESP services. Initially, this caused some delays for the implementation of the services; however, the Government was able to creatively contract the specialists a few days a week, allowing them to maintain their private practices and still implement the services required to orchestrate the ESP. For many PATH households who had children with disabilities, these services ended up being life-changing. The Grant gave children with disabilities benefitting from the PATH access and the opportunity to enhance their physical and mental developmental skills. At the Grant’s closure, the Government recognized the impact of these services and agreed to continue to contract these providers so that all children with disabilities are able to access these services.

Justification of Overall Outcome Rating

33. Overall, the Grant’s outcome was satisfactory, demonstrating a high relevance to the CPF, almost fully achieving all of its intended outcomes (substantial efficacy), and for the most part, substantially efficient for the sector.34 The Grant also financed several activities that were not planned, which contributed to its satisfactory rating. At the close of the Grant, two of the three PDO indicators and five of the six intermediate indicators were met. The PDO indicator for work experience exceeded its end target, enabling PWDs participating in training to gain valuable hands-on experience in jobs related to their areas of skills training. The PDO indicator, percentage of trained PWDs who were employed within twelve months of training, was not achieved largely due to a number of factors beyond the scope or objective of the Grant (discussed in Section III). It is worth noting, though, that of the number of PWDs employed within twelve months, 51 percent were poor PATH PWDs between 18-36 years of age. This opportunity for these men and women, who traditionally have relied only on welfare their whole lives, sets them on a path to be more independent and actively engaged in society. Further, it gives them and all PWDs who completed the training course dignity and a sense of worth. Although this was not measured in terms of outcomes, in and of itself, qualitative evidence from the ICRR mission deemed this to be a significant achievement. Subsequently, the investment of this Grant in training and, for some, the link with employment likely yielded outcomes that could contribute to breaking the inter-generational cycle of poverty for PWDs. Similarly, the investment in the ESP gave poor children with disabilities enhanced mental and physical capacity which could likely translate to better earnings later in life.35

34 Due to reasons noted previously in this report, split ratings were not applied to separately assess the original and revised objectives and associated outcomes of the Grant. 35 Denboba, Amina D., Rebecca K. Sayre, Quentin T. Wodon, Leslie K. Elder, Laura B. Rawlings, and Joan Lombardi. 2014. “Stepping up Early Childhood Development: Investing in Young Children for High Returns.” World Bank. Washington, DC. And WHO and


Page 15 of 28

Other Outcomes and Impacts

34. The Grant had a positive effect on the institutional capacity of the MLSS and other institutions. In particular, the operation:

Strengthened the day-to-day operations of the NGOs providing the training and skills development for PWDs, positioning them to be more effective and prepared to scale-up in the future. Financial management and general management training were important investments for the NGOs providing services to PWDs.

Improved the capacity of the JCPD, strengthening its ability to advocate for persons with disabilities and make better use of the data available to inform policy and program development.

Furthered development of personnel of the ESP, boosting their qualifications and abilities to work with some of Jamaica’s most vulnerable populations. The manager of the ESP noted during the implementation completion and results report (ICRR) mission an increase in morale among staff.

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME

Key Factors During Preparation

35. The MLSS was the most appropriate stakeholder to manage the Grant activities given its mandate and overwhelming commitment to social protection; however, there was a missed opportunity to engage NGOs and private providers already working with PWDs during the Grant’s design which could have given more clarity to the actual implementation of the activities. Given the scope and coverage of the MLSS’s social protection programs and services (PATH, social care services, Steps-to-Work, and the National Insurance Scheme, among others) and its directive to support the Jamaica Council for Persons with Disabilities (JCPD) and the Early Stimulation Programme, the MLSS was deemed the most qualified for managing the activities under this Grant. During the design of the Grant, though, assumptions were made on who would implement the services to be provided under the Grant. While the MLSS did engage key stakeholders in the sector, this was not done until a month prior to the Grant’s effectiveness. Had this been carried out earlier, the timing in the implementation of key activities may not have been delayed and more realistic expectations could have been in place to avoid some of the challenges in the international procurement of assistive aids.

36. The inclusion of the monitoring and evaluation component at the design phase was important, but highly ambitious given the challenges Jamaica faced in carrying out assessments and creating management information systems under the World Bank funded Social Protection Project (SPP) (P105024). At the time of the Grant’s preparation, the MLSS was also implementing the SPP, which had a number of assessments and evaluations as well as enhancements to the BMIS. Many of the foreseen assessments and evaluations under the SPP were never completed and some were subsequently removed due to the lack of capacity to complete them or the inability to contract the right firm to carry out the evaluations. While monitoring and evaluation is critical to the management and understanding the impact of any project, the inclusion of multiple assessments and the implementation of a management information system in the Grant’s design was overly optimistic. In the end, the number of assessments was reduced and the Grant procured an off-the-shelf MIS, rather than creating one from scratch.

UNICEF. 2012. “Early Childhood Development and Disability.” Geneva, Switzerland.


Page 16 of 28

Key Factors During Implementation

37. The Government’s unwavering political and financial commitment to socially and economically engaging PWDs was critical for the successful implementation of the Grant’s activities. The support of the GOJ throughout the entire Grant was critical for the achievement of the outcomes. Politically, the Government signed the Disabilities Act of 2014 during the implementation of the Grant, and used the Grant to begin to explore what the operationalization of the Act would look like. Additionally, the GOJ engaged with and established a positive working relationship with NGOs and private sector entities already working in the sector to enhance service delivery to PWDs. At the Grant’s closure, the GOJ recognized the overwhelming need to continue to invest in PWDs and remained committed to financing and possibly scaling up some of the activities in the future.

38. The lack of detailed information about working aged PWDs in the PATH’s Beneficiary Management System (BMIS) during implementation made the identification of poor persons aged 18-36 years with disabilities for the skills development challenging. The Grant had originally committed to using the PATH’s Beneficiary Management Information System (BMIS) to identify working age PWDs; however, as recognized early on, the type of disability is not registered in the BMIS, thus making it very challenging to identify an appropriate training regime for eligible beneficiaries with disabilities. Furthermore, the training needs assessment of PWDs on PATH identified that many of the individuals had low levels of educational achievement and skills development.36 While many of them had attended primary or secondary school, few had actually completed school.37 Additionally, and consistent with the population trends, more females than males attended and completed schools.38 Acknowledging these challenges, the first round of the training and skills development ended up benefitting a large number of PWDs, with very few of them also being beneficiaries of the PATH, as had been set out at the start of the Grant and denoted in the PDO. This was a point of contention early on as the NGOs argued that the need and demand for training for PWDs in Jamaica was equally as great as the working age PWDs on PATH. The MLSS recognized this concern, noting though, that poor persons with disabilities often face greater financial and logistical obstacles to accessing these services. Subsequently, with each round of training that was carried out under Component 1, the number of PWDs on PATH increased, while the number of non-poor PWDs decreased. Social workers in the MLSS were employed to proactively reach out to PATH beneficiaries with disabilities aged 18-36 years to refer potential trainees and work with households to ensure that there were no barriers to the individual’s participation.39 Upon completion of the last round of training, PATH beneficiary households represented 53 percent of the total number of PWDs trained in all four rounds, largely due to these efforts to increase outreach to PATH households. 40

36 Moncieffe, Joy M. 2014. “Training Needs Assessment of Path Beneficiaries with Disabilities and Capacity Assessment of Non-Governmental Organizations Providing Skills Training for Persons with Disabilities.” Institute for Applied and Social Research. Kingston, Jamaica. Only 11.4 percent of the sample had attended skills training programs while a lower 6.2 percent (44 of 715 respondents) had completed the program. 37 Ibid. Of the females, 38.9 percent had last attended primary school and 32.6 percent had attended secondary. Only 2.5 percent had progressed to HEART/NTA Level 1. Of the males, 45.3 percent had completed primary school and 36.1 percent attended secondary school. 38 Ibid. 39 As such, the Grant provided a stipend to trainees, transportation, lunch, and a uniform. 40 In the third round of training, PATH beneficiaries with disabilities represented 78 percent of the total number of persons trained in the third round of training. This was up from the second round, where 50 percent was PWDs on the PATH and the first round, where 42 percent were PWDs on the PATH.


Page 17 of 28

39. As previously alluded to, the outcome of increased skill development which aimed to contribute to increased employability of PWDs may have been impacted by a number of factors beyond the scope or objective of the Grant. Increased employability under the Grant was achieved through improvements in job readiness (skills development and training, and use of job coaches), and for some, employment, with the latter facing greater challenges in achieving. In general, employment retention was ambitious. First, it is important to acknowledge that countries supporting similar types of short-term skills training for able-bodied poor, unemployed, and/or vulnerable youth often struggle to link their beneficiaries with employment. A meta-analysis of short term skills training programs in developing countries showed that these interventions can pay off, although many programs fail to deliver long-term positive impacts on employment.41 Second, improving the employability of PWDs in Jamaica had never been done before this Grant. Subsequently, it was difficult to determine what type of impact there would be.

40. Additionally, there are aspects outside the Government’s and Grant’s sphere of influence that likely affected the overall outcome of permanent employment of PWDs. At the start of the Grant, Jamaica’s economy was crawling back from one of the worst financial crises in history. The global financial crisis of 2008-2009 devastated Jamaica’s economy, which contracted sharply,42 sending poverty levels soaring to 17.6 percent in 2010 (from 9.9 percent in 2007)43, eroding previous progress and disparities in the well-being of households and communities across the country increased, particularly between rural and urban areas. The fall out of the crisis contributed to high levels of unemployment in Jamaica. At the start of the Grant, Jamaica’s unemployment rate was at 15.25 percent, the highest it had been in a decade.44 While the unemployment rate declined during the lifetime of the Grant and the number of persons trained increased, the lack of employment opportunities posed a challenge for all individuals seeking employment, but particularly for PWDs. Evidence has shown that impacts of short-term skills training tend to do better in developing countries when there is economic dynamism, with low unemployment, employment opportunities, and high growth rates.45 Finally, a last challenge to gaining employment was demonstrating to Jamaican employers the potential of PWDs in the workforce. The Grant financed communications campaign was an important start; however, the implementation period was too short to see the impact of these efforts.

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME

Quality of Monitoring and Evaluation (M&E)

41. M&E Design: Overall, the results indicators were well-defined; though, like the objectives, a few of them were ambitious. As discussed in the outcomes section, the percentage of able-bodied persons who are permanently employed upon completion of short-term skills training is less than 20 percent in some countries in the region. Thus, anticipating that 80 percent of PWDs (later revised to 40 percent in the restructuring) who completed the short-term skills training would be permanently employed was aspirational.

41 Arias, Omar. 2017. After 1-2 years of the skills training, only 12 percent are usually employed. 42 Real GDP declined by 1.6 percent in FY2008/09, with economic conditions deteriorating sharply in the second half of the year and contracted further in 2009/10, registering a decline of 3 percent just in the latter half of 2009. Bauxite production and exports fell by about 60 percent, while remittances—a traditional source of support to households and the balance of payments—declined by 33 percent. Data from CPS FY2014-17. 43 Statistical Institute of Jamaica. 2007. Jamaica Survey of Living conditions. Kingston, Jamaica. 44 Statista. 2018. 45 Arias, Omar. 2017.


Page 18 of 28

42. M&E Implementation: Some elements of what was originally projected to be carried out under M&E began late or were not carried out at all. Initially, the Project had aimed to carry out baseline and endline surveys for Components 1 and 2 and process and outcome evaluations (all of this to be financed under Component 3). Indicators were supposed to be defined early on, with baseline data collected and regularly monitoring and process and outcome evaluations carried out as a means of improving the training intervention and ESP as the Grant progressed. In the end, the Grant had created an Excel database to capture information on beneficiaries and had carried out one (1) impact assessment of Component 2. The assessment began very late (nearly in the last year of the Grant) and was completed just prior to the Grant’s closing. An impact assessment of Component 1 was initiated, but not yet completed by the Grant’s closing but MLSS assured it would supervise its completion and sharing of results. Many of the foreseen M&E activities were not completed due to the competing demands for project staff focused on implementation of Component 1 and 2 activities, and M&E specialists in the MLSS, lack of available data and challenges in finding consultants to carry out the larger process evaluation.

43. The Grant also aimed to develop an MIS to capture data from Components 1 and 2; in the end, recognizing the challenges and cost associated with creating an MIS from the start, the MLSS opted to purchase off-the-shelf software to manage information for the ESP. One of the many areas identified in the Grant-financed Needs Assessment and Institutional Review of the ESP carried out in 2015 was the urgent need for an information system that captured relevant data to support intake, diagnosis and monitoring of interventions and development. The Project originally intended to develop an MIS from scratch to support these objectives, however there were a number of issues preventing them from doing so including: staffing constraints with the MIS Unit at MLSS; competing priorities and little time available to support the MIS design process; prohibitive costs for developing a system from scratch; time constraints associated with finding an option that could be adapted to the needs of ESP; identifying the appropriate software; the need to add ‘Goods’ as an eligible expenditure under Component 3 of the Grant Agreement to allow for the purchase of hardware to operate the MIS (causing a restructuring); procurement processes, and the set-up of a system. In the end, the Grant decided to purchase an off-the-shelf software MIS that would allow the ESP staff to better monitor, evaluate and manage the day-to-day operations of the ESP. This off-the-shelf option was also moderately customized to ensure it could support the daily operations required by ESP staff. By the Grant’s closure, the MIS was functional, albeit basic, and the MLSS was working with the firm to ensure that the system continued to improve its functionality to allow for improved day-to-day management of the ESP. Beyond the closure of the Grant, the MLSS will continue to have ownership of the system and will be responsible for the system’s maintenance after the warranty period ends.

44. The GOJ also regularly produced bi-annual progress reports on the Grant’s implementation and carried out a number of assessments. The progress reports presented a concise summary of the implementation of the different components as well as an updated results framework. Data for the results framework were compiled from Excel datasheets kept on each of the interventions and would inform the World Bank’s ISRs. Moreover, some of the staff managing the Grant mentioned during the ICRR mission that the progress reports served as an important accountability tool, providing information on where there may be possible bottlenecks and identifying areas for possible improvements. The Grant also financed assessments of the training needs of PATH beneficiaries with disabilities aged 18-36, a Needs Assessment of ESP (previously mentioned) and a rapid assessment of the implementation of the skills development intervention for PWDs.


Page 19 of 28

45. M&E Utilization: Although the MIS for the ESP was not implemented until the end of the Project, qualitative evidence from the Grant’s closing mission indicated that the ESP was already being used to analyze data for children with disabilities, the types of services they were receiving and their trajectory of improvement. Moreover, for an area that had been rarely assessed in Jamaica, the findings from the various assessments marked an inflection point in the work being done with PWDs in Jamaica. To some degree, the findings supported the reshaping of the national dialogue around inclusivity and gave institutions leverage to seek additional support in the future.

Justification of Overall M&E Rating

46. Throughout implementation, monitoring and evaluation of the Grant was modest. Given the innovative design of this Grant, the MLSS committed to monitoring and evaluating the interventions under Components 1 and 2 as a means of deriving lessons for future scale-up. A concentrated investment in poor persons with disabilities was new for Jamaica and much of Latin America and the Caribbean. Although there were regular progress reports on the Project’s activities and outcomes, the delays in completing the Grant’s evaluation activities and timing for completing the ESP MIS close to the Project’s closing date, contributed to this modest rating.

Environmental, Social and Fiduciary Compliance

47. Environmental and Social Safeguard policies were not triggered.

48. Fiduciary. In general, there were established financial management arrangements which the Bank deemed to be functioning. For the duration of the Grant, the ISRs have noted financial management to be moderately satisfactory. On reporting, the submission of the financial reports was at times delayed, missing pertinent pieces of information, or inaccurately prepared. That said, all financial audits were found to be satisfactory.

Bank Performance

Quality at Entry

49. At entry, the World Bank leveraged existing knowledge on early childhood development interventions for children with disabilities and short-term training program and employment of persons with disabilities to support the design of a timely Grant that was highly relevant for the Jamaican context. Just prior to the Grant, the World Bank had published the “World Report on Disability.”46 For the design of the Grant, the Bank leveraged data from this report as well as expertise and experiences to better inform what to finance and areas for further development. In addition, experts working with adults and children with disabilities and in formulating inclusion policies in various countries in the region (Ecuador and Chile) participated in the early design of the Grant as a means of sharing lessons learned from other countries in the region. Additionally, the PDOs’ focus on the social and economic inclusion of PWDs was highly relevant to the GOJ’s commitment to promoting human capital development and economic participation of the poor and vulnerable, central to the National Social Protection Strategy being finalized parallel to the start of the Grant.

50. The Grant was seen as part of a larger coordinated effort of financial and technical assistance requested by the GOJ from the donor community to support the further development of services for this highly vulnerable population in Jamaica. The Grant was designed to complement ongoing

46 World Bank. 2011. World report on disability: Main report.” Washington, DC: World Bank


Page 20 of 28

interventions including the larger World Bank Social Protection Project (loans 7555-JM and 8329-JM), which primarily financed the institutional strengthening (targeting, payment, MIS, etc.) and expansion of the PATH CCT, and the IDB financed rehabilitation of the buildings housing the ESP.

51. Prior to the start of the Grant, the World Bank could have worked more closely with the GOJ to better define the targeting and implementation arrangements of the skills training component. As previously mentioned, the Grant had anticipated that the PATH’s Beneficiary Management Information System (BMIS) would identify work-abled PWDs and that these individuals were fully capable of participating in the existing training modules. However, an assessment under the Grant would later show that the learning levels of PATH beneficiaries with disabilities were much lower than non-PATH beneficiaries with disabilities. The Bank could have done more to identify this issue ahead of time by possibly carrying out the training needs assessment of PATH beneficiaries with disabilities using the BMIS prior to the Grant’s implementation. The Grant had also anticipated having HEART/NTA spearhead the skills development and training of PATH beneficiaries with disabilities aged 18-36 years given that the MLSS was already using them as the primary agent to implement the national training program for PATH beneficiaries, Steps-to-Work. However, an initial institutional analysis of HEART/NTA’s capacity to work with PWDs was not carried out. This was later resolved with a national workshop bringing together all NGOs and public entities working with PWDs on training and skills development; but, this would not be done until the Grant was nearly effective and already designed.

52. At entry, the Bank could have done more analysis on the GOJ’s capacity and availability of data to carry out the monitoring and evaluation of such an innovative project. Targeted and well-thought investments in both the ESP and the skills development and training programs had never been done before in Jamaica. While there was a component on monitoring and evaluation, many of the anticipated assessments were not carried out in the end. Although this is the result of a number of factors (lack of consultants, available data, etc.), the Bank could have done more due diligence on where the country was in terms of capacity and existing data to better define what evaluations, assessments and information systems (such as the off-the-shelf software) were possible for carrying out during the life of the Grant.

Quality of Supervision

53. The implementation cycle had three (3) task team leaders (TTLs) for the Grant, allowing for consistent support throughout the life of the Grant. The first TTL designed the Grant and saw it through the Grant’s effectiveness. After the Grant became effective, co-TTLs took over the management of the Grant until the Grant’s completion. This allowed for regular supervision of the Grant, including an MTR mission in 2015, communication and a dependable means of problem-solving when challenges arose.

54. The Bank supported three restructurings which, arguably, improved the efficacy of the Grant. Apprised by the results from the 2015 MTR mission and subsequent supervision missions, the Grant sustained three restructurings. The Bank worked diligently to inform these restructurings with results from other skills training projects in the region and on early childhood development for children with disabilities. The Bank also used the restructurings to address challenging procurement issues. This is particularly evident with regard to the development of a new management information system, which was later replaced with the procurement of an off-the-shelf MIS. The Bank’s flexibility and creative resolve to address the challenge allowed for the addition of Goods under the M&E component and the purchase and use of off-the-shelf software to manage and administer the ESP.

55. The Bank routinely facilitated review and inputs from experts in the disabilities and SP sector to support Grant implementation. Notably, the Bank facilitated the review of plans for Component 1


Page 21 of 28

training and Component 2 interventions through a visit by a leading disabilities expert from Chile in October 2014. Additionally, the terms of reference and scope for the Codes of Practice activity received inputs from Bank staff with expertise on disabilities rights and development. The options for the off-the-shelf ESP MIS also received expert review from a leading MIS expert within the Bank, providing MLSS with feedback on the parameters and utility of the potential options that were explored before a final software was selected.

Rating of Bank’s Performance

56. Overall, while there were some minor challenges at entry, the Bank’s performance is assessed as satisfactory. The challenges in the design of the Grant are challenges that were not uncommon at the time the Grant was being developed. The World Bank’s operational knowledge and experience on the inclusion of persons with disabilities was limited to only a few countries and this Grant was very much seen as an innovative opportunity to pilot such interventions. As such, during supervision, the Bank team worked with the GOJ to ensure that interventions remained relevant to the country’s priorities and responded to the needs of the population while meeting the outcomes set out at the start of the Grant.

Risk to Development Outcome

57. The risk to the sustainability of the development outcomes is substantial. The Grant played an important role in financing services and interventions that had not previously been a part of Government services. As a result, there are factors that will contribute to sustaining the development objectives and outcomes beyond closing, including increased capacity of organizations providing training to PWDs; sustained ESP services, including parenting training; and improved monitoring of ESP services through the Management Information System financed by the Grant. Despite this, there is a considerable risk that without additional funding, many of these interventions may not be financed in the future, or at least not at the scale they had been implemented under the Grant.

Loss of key personnel to connect PWDs with employment: While many of the NGOs will continue to provide skills training to PWDs, without the financial investment, coverage will decrease and “job counselors” will disappear. This will have a profound impact on establishing and maintaining the link between the PWDs trained and employers, often a barrier for PWDs.

Increase of the cost of training and skills development for poor PWDs. Under the Grant, PWDs also benefitting from PATH did not have to pay to access and take courses. Without the Grant, poor PWDs will have to pay out of pocket, likely hindering poor PWDs from receiving training and skill development.

Retention of key personnel to provide services in ESP: Many of the technical staff hired under the Project will not be able to continue without the financial resources to hire them. The Government, though, has budgeted for the inclusion of two therapists for ESP to support future operation.

58. Technical and financial capacity of NGOs and other public entities providing training and skills development could potentially lead to future funding for their efforts. While the training and skills development courses may not operate at the level they did under the Grant, there is little risk that these organizations would not continue with the training and administration of skills development courses for PWDs. After all, even before the Grant, they were providing these services. Additionally, capacity building provided under the Grant to the organizations improves their potential for possible funding from other development agencies or international donors, reducing risk to their sustainability.


Page 22 of 28

59. For Project-related services delivered by the Government, these will continue to be sustained and supported. Most notably, the ESP falls under the direct responsibility of MLSS. Through the Project-supported MIS; commitment to financing continued therapeutic services after Project completion; and capacity building of ESP staff to deliver parenting education, these services will continue to be offered in a more comprehensive way than they were provided before the Project was initiated. Additionally, JCPD also falls directly under the ambit of MLSS and following Project close, the Council will assume responsibility for the dialogue and support to the sector that was being offered through the Project. This will ensure a sustained avenue for support to PWDs on the policy agendas.

V. LESSONS LEARNED AND RECOMMENDATIONS

Lessons Learned

58. The Government of Jamaica’s strong support was central to engaging NGOs, employers, parents of children with disabilities, and adults with disabilities. Historically, support for PWDs had largely been ad-hoc with some support through the PATH and services run through various private and civil society organizations under the direction and leadership of the JCPD. From the start of the Grant, there was a notable change in GOJ’s commitment and recognition that a different approach needed to be taken to engage persons with disabilities. The 15-17 percent of the population with disabilities were no longer viewed as a vulnerable group that needed just assistance, but also as a critical part of the greater population with the potential to be actively engaged in employment, skill development, and education-related activities. This change in perception radically altered the way in which the GOJ interacted with the NGOs and civil society organizations implementing the training and skills development.

59. Jamaica is a fervent stalwart of the social and economic inclusion of PWDs and legislatively has made critical advances that demonstrate this. Important work remains, though, in the development of the remaining Codes of Practice and the operationalization of the Disabilities Act. The Grant was able to support initial efforts for the completion of the Codes of Practice on Education, and, Training and Employment. Completion of these Codes and the other five Codes will be central to ensuring that the Disabilities Act passed in 2014 is effective. The JCPD will also establish a Disability Rights Tribunal and evolve into its own corporate body to better support the efforts, policy and legislation in the sector. Continuing to invest in the JCPD and efforts to operationalize the Disability Act will ultimately have a more effective and sustainable impact on PWDs in Jamaica.

60. Ensuring services for PWDs throughout all of Jamaica is critical for the economic and social inclusion of all PWDs and opportunities remain for investing in additional expansion of services. Currently, the bulk of services for PWDs in Jamaica are concentrated in Kingston and St. Andrew, causing PWDs to have to travel long distances to reach services, which, physically and financially, is not always possible. Services for PWDs need to be expanded outside of urban areas to other less accessible towns. This will require critical, though necessary, investments in infrastructure, human resources, and equipment for PWDs.

Recommendations

61. The PHRD RETF was the correct tool for testing this innovative model and, where the supporting environment and capacity exist, it can and should be replicated in other places. Prior to the Grant, investments in PWDs and programs supporting PWDs in Jamaica were limited and sporadic, and even more so for poor households who didn’t have the financial means to access the few services that did exist. Within this context, the PHRD Grant allowed the GOJ to creatively pilot various strategies and


Page 23 of 28

interventions for the social and economic inclusion of PWDs that may not have been feasible with a large lending operation. With the Grant, the MLSS was able to leverage the targeting tool for the PATH to identify some of Jamaica’s most poor and vulnerable households with PWDs as potential beneficiaries and link them with the services supported under this Grant. Moreover, the Grant caused Jamaica to gain a better understanding of what services existed and the capacity of these services to work with PWDs coming from impoverished households. This allowed for the innovative contracting of NGOs and other public entities for the delivery of training and skills development services and the opportunity to increase their day-to-day operational capacity. For a country that adamantly supports early childhood development for all children, it also profoundly affected the approach to ECD for children with disabilities, providing assistive aids for some and therapeutic services (speech therapy, physiotherapy, etc.) for all, which were not being offered before the Grant's execution. There is potential to duplicate this kind of work in other countries where the supporting environment and capacity exist and to scale it up in Jamaica.

62. Projects aimed at improving employment outcomes, particularly for hard-to-serve47 populations, should include strong demand-side activities with employers. Under the design of this Grant, Component 1 activities centered on training, with the labor market study being the principal activity focused on employers. The Grant was able to carry out additional activities with employers, namely the communications campaign and two sensitization workshops. In addition to this, training providers engaged in recurring intermediation with employers to facilitate job attachments and to advocate for employment. Despite this, the results from the labor market study and full-time employment outcomes revealed that more systematic sensitization, training, and intermediation with employers may have been useful. As with all activation programs, both demand-side and supply-side interventions are critical. For hard-to-serve populations such as PWDs, impediments to employment go beyond employer preferences and non-cognitive skills, to issues related to stigma and additional costs for adapting work environments. As Jamaica works on finalizing its Codes of Practices for Employment and Education and Training, strong input, engagement, sensitization, and feedback mechanisms from the private sector will be essential. These lessons are important for other Projects with similar target populations and objectives.

63. Sustained public awareness campaigns are critical for debunking misinformation and changing perceptions and attitudes. To date, there public information campaigns have been sporadic. The lack of sustained communications and public awareness has contributed to misinformation among the public, which sometimes reinforces discriminatory beliefs and prejudices. While the Grant did support two employer sensitization workshops and other communication campaign-related efforts, there needs to be a sustained public awareness campaign to both the general public and employers all over the country to change the misconceptions and outlooks individuals have toward PWDs. Financing communications activities would therefore be extremely beneficial to any future operation serving this population.

64. Institutions providing training and skills development to PWDs will require ongoing investment in infrastructure and human resources. The Grant provided essential support to many NGOs and other public entities providing training and skill development to PWDs; however, many educational facilities in Jamaica remain ill-equipped to serve PWDs, and also require complete system changes to serve certain

47 “Hard-to-serve” implies that the needs of some populations may be beyond the scope of services that are typically available in welfare or welfare-to-work offices. The term is also often used interchangeably with the term “hard to employ,” as these populations often have characteristics that impede their ability to find and keep jobs. University of Michigan, Barriers to employment and the “hard to serve”: Implications for services, sanctions, and time limits. 2002.


Page 24 of 28

PWD populations (for instance hearing impaired persons). These enhancements will allow for the ongoing inclusion of PWDs in Jamaica.

65. Concentrating the scarce resources of multiple development partners in an area that is highly underinvested can have a profound impact. For improvement of the ESP, the IDB and World Bank worked together to finance complementary investments to ensure that the Program would thrive. The IDB financed the upgrade of the ESP facilities while the Grant supported improvements in the staff’s capacity, an MIS and the introduction of therapeutic services and parental training. Parents of, and children with, disabilities now receive expanded services in a more comfortable learning environment.

66. Investing in monitoring and evaluation is important for advocating for changes in legislation, funding, programming, and policies for the inclusion of persons with disabilities. Evidence has shown that critical investments in monitoring and evaluation benefit future programming, funding, policies and legislation that promote the social and economic inclusion of persons with disabilities.48 Ongoing maintenance of the MIS for the ESP will be important to ensure that it continues to serve as a management tool for improving the Program. Moreover, the regular collection of information and assessment of interventions is critical to for future investments.

.

48 Cappa, Claudia, Nicole Petrowski, and Janet Njelesani. 2015. “Navigating the landscape of child disability measurement: A review of available data collection instruments.” European Journal of Disability Research: 9 (2015) 317-330.


Page 25 of 28

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS

A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increase the employability and skills development of poor persons with disabilities (ages of 18-36 years).

Indicator Name Unit of Measure

Baseline Original Target Formally Revised

Target

Actual Achieved at Completion

Persons with disabilities who receive employment (work experience) after completing classroom training

Number 0.00 300.00 300.00 384.00

23-Aug-2013 30-Dec-2016 14-Aug-2018 01-Aug-2018

Comments (achievements against targets):



Target


Persons with disabilities trained under the project who have gained employment within twelve (12) months after completion of the program

Percentage 0.00 80.00 40.00 12.48

23-Aug-2013 30-Dec-2016 14-Aug-2018 01-Aug-2018

Comments (achievements against targets): By the Project's closing, 601 beneficiaries completed Component 1 training, of whom 75 have


Page 26 of 28

received full time employment following training completion. Objective/Outcome: Improve the service delivery of special education needs to poor children with disabilities between the ages of 0- 6 years.



Target


Poor children (0-6 years) with disabilities in Early Stimulation Program show improved readiness for school

Percentage 0.00 50.00 40.00 47.27

23-Aug-2013 30-Dec-2016 14-Aug-2018 01-Aug-2018

Comments (achievements against targets): The end target for this indicator was 40 percent of a total of 880 poor children with disabilities. By the Project's closing, the current rate was 47.27 percent which represents 416 children benefitting from physiotherapy (215), speech therapy (120), and adaptive aids (81).

A.2 Intermediate Results Indicators

Component: Component 1: Skills Development of Persons with Disabilities



Target


Labor market study completed Text 0.00 1.00 1.00 1.00

23-Aug-2013 30-Dec-2016 30-Dec-2016 01-Aug-2018

Comments (achievements against targets): This target was met in December, 2016.


Page 27 of 28



Target


Completion of the Assessment of Training Capacity

Text 0.00 1.00 1.00 1.00

23-Aug-2013 31-Dec-2014 31-Dec-2014 01-Aug-2018

Comments (achievements against targets): This target was met in October, 2014.



Target


PWD completing skill/entrepreneurial training activities upon enrolment

Percentage 0.00 80.00 80.00 95.55

23-Aug-2013 11-Aug-2017 14-Aug-2018 01-Aug-2018

Comments (achievements against targets): This target was exceeded by Project completion. At the Project's completion, 629 beneficiaries were enrolled in 4 rounds (351 for Rounds 1 and 2; 161 for Round 3; and 117 for Round 4). Of those enrolled, 601 have completed training (336 for Rounds 1 and 2; 153 for Round 3; and 112 for training under Round 4).

Component: Component 2: Strengthening the Service Delivery of Early Stimulation Programme



Target


Increase in rehabilitative sessions received

Percentage 52.00 10.00 10.00 7650.00

23-Aug-2013 11-Aug-2017 14-Aug-2018 13-Nov-2017

Comments (achievements against targets): This target was exceeded by Project completion, with 2,700 30-minute speech therapy sessions and


Page 28 of 28

1,278 physiotherapy sessions delivered to 137 children ranging from 45 minutes to 1.5 hours per session.



Target


Parents of PATH beneficiaries trained on how to cope with children with disabilities

Number 0.00 250.00 250.00 259.00

23-Aug-2013 11-Aug-2017 14-Aug-2018 01-Aug-2018

Comments (achievements against targets): This target was exceeded by Project completion, with 139 parents trained in Round 1 and 120 parents trained in Round 2.



Target


PATH children in ESP who receive adaptive aids

Percentage 0.00 60.00 60.00 14.72

23-Aug-2013 11-Aug-2017 14-Aug-2018 13-Nov-2017

Comments (achievements against targets): This target was not met by Project completion, as 81 children have been provided adaptive aids of 550 children targeted.


Page 29 of 28

B. ORGANIZATION OF THE ASSESSMENT OF THE PDO

Objective/Outcome 1: Increase the employability and skills development of poor persons with disabilities (ages of 18-36 years).

Outcome Indicators

1. Number of persons with disabilities who receive employment (work experience) after completing classroom training 2. Percentage of persons with disabilities trained under the project who have gained employment within twelve (12) months after completion of the program

Intermediate Results Indicators

1. Labor Market study completed 2. Completion of Assessment of Training Capacity 3. Percentage of PWD completing skill/entrepreneurial training activities upon enrolment

Key Outputs by Component (linked to the achievement of the Objective/Outcome 1)

1. Increase in the number of PWDs who receive work experience after completing classroom training 2. Increase in the percentage of PWDs trained under the project who have gained employment within 12 months after completion of the training program 3. Increase in the percentage of PWDs completing the skill/entrepreneurial training activities upon enrolment

Objective/Outcome 2: Improve the service delivery of special education needs to poor children with disabilities between the ages of 0- 6 years.

Outcome Indicators 1. Poor children (0-6 years) with disabilities in Early Stimulation Program show improved readiness for school

Intermediate Results Indicators

1. Percentage increase in rehabilitative sessions received 2. Number of parents of PATH beneficiaries trained on how to cope with children with disabilities 3. Percentage of PATH children in ESP who receive adaptive aids


Page 30 of 28

Key Outputs by Component (linked to the achievement of the Objective/Outcome 2)

1. Increase in the percentage of poor children (aged 0-6 years) with disabilities who participated in the Early Stimulation Program who have shown improved readiness for school 2. Increase in the percentage of rehabilitative sessions received 3. Increase in the number of parents of PATH beneficiaries trained on how to cope with children with disabilities 4. Increase in the percentage of PATH children in ESP who receive adaptive aids.


Page 31 of 28

. .

ANNEX 2. PROJECT COST BY COMPONENT

Components Amount at Approval

(US$M)

Amount after Restructuring

(US$M)

Total Amount Disbursed (US$M)49

Percentage of Approval (US$M)

Component 1: Skills Development of Persons with Disabilities

1.70 1.70 1.38

Component 2: Strengthen the service delivery of special education needs to children with disabilities

0.72 0.72 0.53

Component 3: Monitoring and Evaluation

0.27 0.19 0.06

Component 4: Project Administration and Management

0.20 0.28 0.24

Total 2.89 2.89 2.21 0.00

49 These figures are as of September 18, 2018. A final financial report is still pending, as the Recipient has not completed processing of transactions incurred prior to the Grant’s closing date. The table will be updated once the final report is received.


Page 32 of 28

ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS

Implementation Completion Report Social & Economic Inclusion Project

Government Response

1.0 Introduction

The Government of Jamaica received financing for the Social and Economic Inclusion of Persons with Disabilities

(PwDs) Project in the sum of USD $2,997,790.00. The project was supported by funding received under the Japan

Policy Human Resources Development (JPHRD) Grant implemented under four components as follows:

Component 1: Skills Development of Persons with Disabilities between the ages 18-36 under PATH

Component 2: Strengthening the service delivery of special education to children with disabilities

Component 3: Monitoring and Evaluation

Component 4: Project Administration and Management

The Project was originally scheduled to be implemented over a four year period: August 2013 – August 2017. A one

year extension was granted to facilitate the full realization of the objectives of the project. The Project ended officially

on August 14, 2018. The project was restructured during Mid Term Review in December 2015 to revise some of the

indicators. Overall, the Social and Economic Inclusion of Persons with Disabilities Project along with Jamaica’s policy

and legislative commitments for the disability sector has directly contributed to human capital development needs for

persons with disabilities.

The Government has reviewed the Banks Implementation Report in respect to the Social & Economic Inclusion Project

dated October 15, 2018. We acknowledge the overall outcome rating of the Project has being satisfactory. The


Page 33 of 28

Government in its response presents its comments on the Implementation of the Project – achievements, challenges

and the way forward.

2.0 Summary of achievements

1. Six Hundred and One (601) PwDs completed skills training activities upon enrolment. 629 were

enrolled over the life of the project.

2. Three Hundred and Eighty Four (384) Persons with Disabilities (PwDs) received work experience

after completing classroom training programmes.

3. Seventy Five (75) trained under the project gained permanent employment within 12 months after

completion of training programme. Of this number 38 were PwDs on PATH.

4. Capacity Building workshops held with Non- Governmental organizations who provided training to

PwDs. The workshops provided technical assistance to the organizations in aligning sub projects

to the achievement of the overall objectives of the Social and Economic Inclusion of persons with

disabilities project. Activities included a review of the funding, budgeting, targeting, procurement and

monitoring and evaluation processes required for successful implementation of sub project activities.

At the end of project implementation, there was an improved capacity among the participating

organizations to implement similar projects, seek and secure donor funding and also play a greater

role in the advocacy for and the placement of trainees in employment opportunities.

5. Forty Seven Percent (47%) of poor children with disabilities in the Early Stimulation Programme

showed improved readiness for school. This represents a total of 416 children benefitting from

Physiotherapy (215), Speech Therapy (120) and adaptive aids (81)

6. Completion of Labour Market Study


Page 34 of 28

7. Completion of Training Needs Assessment of PwDs on PATH and Assessment of Non-

Governmental organizations providing training to PwDs

8. Increase in number of Physiotherapy and Speech Therapy sessions

9. Two Hundred and Fifty Nine (259) parents of children with disabilities receive parenting skills training.

10. Implementation of a Management Information System to track services offered and assessment of

children of the Early Stimulation Programme.

11. Developed and implemented a comprehensive communication campaign that increased employer

and general public awareness of the employability assets of PWDs and services offered by the Early

Stimulation Programme for Children with Disabilities.

3.0 Challenges Experienced

The challenges experienced during the implementation of the Project were mainly related to the

implementation of the skills training and employment activities and service delivery to children with

disabilities.

1. The success of the project relied on the support of other organizations and institutions.

Based on the interactions with these organizations, the achievement of some project outputs were

delayed due to institutional capacity issues inclusive of inadequate staffing and technical capacity,

funding and equipment.

2. Employment access and retention by PwDs – The project had limited control over the provision

of employment opportunities for PwDs across the interventions that were implemented. Employment

was largely driven by the economy and the availability of jobs. This impacted the achievement of

the indicator related to permanent employment of PwDs at the completion of training.

3. Negative Public perception of PwDs and accessibility (accommodation) barriers – Some

employers still have a negative perception of the capacity of PwDs to contribute positively to the

workforce and view hiring PwDs as an act of social responsibility rather than a Human Resource


Page 35 of 28

Development practice. There are also the actual physical barriers to entry where there has been little

investment in both the built and technical environment to enable access of PwDs. This includes

investments in ramps, wheelchair accessible spaces and information communication technologies

used by PwDs.

4. Functional literacy and interest levels of PwDs on PATH – Majority of the PwDs targeted were

functioning at literacy levels that would not have allowed them to matriculate into certificate training

programmes. This impacted on the duration of interventions (held for longer periods) and a literacy

component had to be introduced to facilitate matriculation and enhancement of literacy levels. In

addition, many targeted PwDs were merely not interested in participating in the interventions that

were being made available through the project. This further led to difficulties in mobilizing entire

cohorts of PATH beneficiaries for training. This would have also proven difficult for the Non-

Governmental Organizations as they would have to neglect other PwDs who required the assistance

of the organization for training.

5. Absence of or limited self-Employment options – The design of the project did not include the

provision of grants to participants who would have opted for self-employment at the end of training

programmes. The issue of finances was a huge constraint for many who were interested in this

avenue.

As it relates to services for children with disabilities:

1. There was a delay in the delivery of Speech Therapy and Physiotherapy services as there was some

difficulty experienced with the procurement of these Consultants. These activities are expected to

continue at the ESP funded by the Government of Jamaica.

2. Limited local suppliers with the capacity to provide required assistive aids to beneficiaries in a timely

manner.

3. Difficulties mobilizing Parents for attendance at Parenting Workshops. It was a challenge for some

parents to attend workshops because of financial issues. Many were unemployed and also had no

one to care for their children in their absence.


Page 36 of 28

Despite the challenges experienced, quite a bit of creativity/innovation was employed to ensure that the

Project Objectives were met. For e.g. The methodology for the implementation of training activities for the

component had to be revised based on the realities of the disability sector within Jamaica and the context of

the readiness of institutions to provide training to persons with disabilities. Based on the Procurement Plan

that was submitted for the JPHRD Grant, it was anticipated that one firm would have been contracted to

provide services for the training and employment of PwDs. However, based on an internal review, the project

was informed that it would have been difficult to identify one firm that would be able to deliver all the services

across the island and that there was a paucity of resources in the PWD sector and the capacity of the firms

was not adequate to meet the needs of the drafted Terms of Reference (TOR). It was then agreed that the

TOR would be separated and a call for proposal done by specific intervention areas by the 14 Parishes

(geographic regions) across the island. This led to a reduction in the capacity load and reduced the

requirement for multiple technical expertises within the selected firm.

During implementation; the following strengths and lessons were also realized:

4.0 Strengths/Lessons Learnt

1. Non-Governmental Organization led implementation - implementation through contracted

organizations already working with PwDs facilitated the delivery of training programmes. These

entities have forged partnerships with the HEART National Training Agency and the National Youth

Service For the certification and employment placement of PwDs. Some institutions also engaged

the Private Sector Companies and other organizations that provided both temporary and permanent

employment opportunities for PwDs.

2. Mobilization of PATH beneficiaries for Training – A major challenge encountered during the

project was the recruitment of a full training cohort (100 %) of PATH beneficiaries. This proved

difficult for the implementing organizations as they had other PwDs to treat with who required this

kind of intervention. Going forward, it is therefore necessary to ensure that appropriate and practical

objectives are set regarding mobilization of specific cohorts of persons for training.

3. Capacity strengthening of contracted partners – over the life of the project, the actual process of

implementation served to strengthen the capacity of partners to implement projects of a similar nature

in the future.


Page 37 of 28

4. Provision of key enablers during training - stipend, transportation, lunch, uniforms - This ensured

that the attrition rate among participants was kept at a minimum and failure of PwDs to complete the

interventions was mainly due to issues outside of the projects control, for example, behavioural

problems or illness.

5. Skills training and employment combination – before the project, many of the organizations

placed more emphasis on training and had a lesser focus on ensuring that participants had access

to employment opportunities. Based on the design of the project and temporary employment of

participants being mandatory a greater effort was made to ensure at least temporary employment.

6. Utilization of Job coaching to facilitate labour activation and inclusion of PwDs in the workforce –

The use of this methodology acted as a buffer between the employer and the PwD being employed

facilitating a much smoother transition for the PwD in the work environment.

7. Multi Stakeholder approach to implementation – The MLSS forged partnerships with Non-

Governmental organizations, Public Sector Bodies, Non State Actors and the Private Sector. This

augured well for the achievement of project objectives and highlighted how opportunities were

created through strong partnerships.

5.0 Efforts to Address Sustainability/Forward Look

The Social & Economic Inclusion Project implemented by the Government of Jamaica was the first in the

English speaking Caribbean and a unique innovation to pattern going forward. The gains under the Project

will be sustained particularly as it relates to the provision of training to persons with disabilities. The

experience of the Project would have strengthened the capacity of the organizations to deliver training

programmes to PwDs. Although training offered may not be as wide in scope as with funding for the project,

these activities will still be completed as they form a core function of the organization. Additionally, the

organizations will now be able to add different components to these interventions which will auger well for

better training outcomes.

Regarding service delivery to children: the MLSS has committed to fund Speech Therapy activities going

forward and this has been budgeted to continue in the current and upcoming financial years. In addition, the


Page 38 of 28

Early Stimulation Programme will continue to deliver Parenting Workshops utilizing Curriculum that was

designed under the project

6.0 Monitoring & Evaluation

The M&E Framework captured adequate indicators. However, in some instances the targets were overly

ambitious. For example the target for employment was too high and was deemed by the Government at the

design phase to be unachievable as the Management of the Project would have no control over the factors

impacting the economy and job availability. However, after review of the grant submission, measurement of

this indicator was found to be necessary and the target set (as advised) could be reviewed at mid term.

Additionally, although evaluations are necessary for assessment of these Projects, having had no previous

experience or best practice in implementing specific Projects for PwDs (training, employment, services); the

evaluation methodologies were too many and not specifically detailed in terms of expectation. Nevertheless,

a number of evaluations were undertaken that provided results that guided implementation as well as future

expansion. Specifically, there was a needs assessment of PWDs, labour market study and two end impact

evaluations for Components 1& 2.

During the life of the Project, information on the beneficiaries was maintained albeit in an excel spreadsheet.

Progress reports were prepared using information from the database to monitor and track implementation in

a timely manner. An insightful addition to the M&E framework was the development of a Client Management

& Patient Information System for the monitoring of service delivery and children assessment of the Early

Stimulation Programme. This will provide for the long term sustainable monitoring of this Programme that

was not envisioned in the M&E Framework at the design phase. Going forward: The MLSS has committed

to:

1. Provide assistance to the Early Stimulation Programme for the maintenance of the Client

Management System through ongoing Technical Support from the MLSS’ MIS Department.

2. Provide assistance to the Early Stimulation Programme with the Data Entry of paper based client

files that would have been completed prior to the implementation of Patient Management System.


Page 39 of 28

7.0 Bank Support

Overall the Banks support was deemed satisfactory. However, the design phase could have benefitted more

from expertise in the Bank on Disability issues. This expertise came somewhat later after implementation

started. It was also felt that more accommodation could have been given to facilitating automation of the ESP

although not included in the original M&E Framework (lengthy process to facilitate inclusion of the activity)

and facilitating suggestions regarding the purchase of assistive aids (albeit near to Project close), recognizing

the challenges experienced in sourcing these aides during the life of the Project. There was Bank support

overall for the general implementation of the Project and effort in securing a one year Project extension.


Page 40 of 28

ANNEX 4. SUPPORTING DOCUMENTS (IF ANY)

Documents

IMPLEMENTATION COMPLETION AND RESULTS …...The World Bank Jamaica Social and Economic Inclusion of Persons with Disabilities (P144263) 03 08-Dec-2015 Moderately Satisfactory Moderately