2014 Nuru Kenya Education Program Impact Assessment

Results of the 2014 Education Uwezo Survey

December 2014

Authored by

Bethany Paris

With contributions from Kim Do

Kristin Lindell Veronica Olazabal

2

Executive Summary The Nuru Kenya (NK) Education Program aims to increase child literacy to the Standard Two (Grade 2) level among rural primary school children through student-centered teaching and literacy-focused interventions. As part of a larger assessment strategy, the Nuru Kenya (NK) Monitoring and Evaluation (M&E) Program team supports this goal by addressing the following evaluation question in this report: What is the impact of the Nuru Kenya Education Program on progress toward English literacy and a student’s ability to read at least at the paragraph level? In order to capture progress toward Standard Two literacy, a quasi-experimental methodology was implemented where NK M&E took a census of NK Education public school children in Grades 2-7 in one school 1 in Kuria West, Kenya. NK M&E compared the change in literacy during a pre-intervention year (2013) with its first intervention year (2014). This approach essentially treats the students tested in 2013 as a “control” group and the students tested in 2014 as a “treatment” group while aiming to determine if, and by how much, NK Education caused shifts in literacy proficiency. Being more robust than a sample, a census design helps to ensure that sampling error does not cause differences observed between the two groups. Results of this study indicate that NK Education positively impacted its two core indicators: 1) average literacy proficiency level; and 2) percentage of children who can read at least at the paragraph level. Average literacy proficiency level is based on a spectrum where: 0 = None, 1 = Letter, 2 = Word, 3 = Paragraph, 4 = Story, 5 = Full Literacy. Prior to receiving NK Education Outreach, public school children who were tested averaged 2.56 along this spectrum, whereas after the intervention, average literacy proficiency rose to 2.66, an increase of 4 percent. Similarly, the number of children who could read at least at the paragraph level prior to NK Outreach rose by 5 percent during the pre-intervention year; however after receiving one year of NK Outreach, this difference increased by more than double, to 13 percent. Further analyses show that the driver of the increase in literacy proficiency is the progress of children in Grades 2-5. For example, average literacy proficiency for public school children in Grades 2-5 increased by 6 percent after participating in NK Outreach, whereas public school children in Grades 6-7 dropped by 1 percent during this same time period. Similarly, the number of public school children who scored at the paragraph level or higher in Grades 2-5 rose 15 percentage points during the post-intervention year (2014) compared to only 9 percentage points during the pre-intervention year (2013). For Grades 6-7, scores increased by 2 percentage points in comparison with no change during the pre-intervention year. In conclusion, the evidence demonstrates that NK Outreach had a positive impact on literacy proficiency and ability to read at the paragraph level for public school children in Grades 2-5. This impact can be attributed to the NK Education Program; moreover, the census study design helps to ensure that differences observed between pre and post intervention years are not due to sampling error. Recommendations for NK M&E and Education to consider as a result of these findings are as follows:

1 The specific name of the school Nuru Kenya surveyed is not included in the report to preserve the anonymity of the study.

3

1. Focus outreach on Grades 2-5 rather than Grades 2-7, as the younger public school children

demonstrate increased rates of progression towards literacy proficiencies. 2. Group public school children by average literacy proficiency in order to better work with

lower performing public school children (as based on the beginning of year scores).

4

Table of Contents

Introduction 5  

The Integrated Nuru Model 5  

Nuru Kenya (NK) Education Program 5  About Uwezo Literacy Assessment 6  

Methodology 7  Sampling Frame 8  Data Collection 9  

Analysis 9  

Results 9  Results of Progress Toward Full Literacy 10  Results of Achievement of Paragraph Level and Above 11  

Limitations 13  

Conclusions and Recommendations 13  

Appendix 14  

5

Introduction The Nuru Monitoring and Evaluation (M&E) Program produces useful and relevant information that can contribute to key decision-making about Nuru’s programs (e.g., whether to continue, replicate and/or scale an intervention). With this focus on utilization at the center of Nuru’s M&E strategy, the M&E team works to objectively monitor and evaluate the performance and impact of Nuru’s four impact programs—Agriculture, Financial Inclusion, Healthcare, and Education. In service to this approach, the Nuru Kenya (NK) M&E team continuously assesses NK Education to answer the question: What is the impact of the Nuru Kenya Education Program on progress toward English literacy and a student’s ability to read at least at the paragraph level? This paper presents the program rationale, goals, and progress toward those goals to answer this question.

The Integrated Nuru Model Nuru International is on a mission to end extreme poverty in remote, rural areas. Communities facing extreme poverty deal with fundamental challenges around hunger; an inability to cope with economic shocks; averting preventable disease and death; and illiteracy. Nuru has proven its ability to deliver lasting impact in these four areas and is currently positioning its model for global scale. As a catalyst for sustainable development, Nuru’s role is to identify nationals to raise up as servant leaders and nation builders; remove barriers preventing them from realizing their full potential; equip them with skills, resources, and attitudes to end extreme poverty in their region; and build social enterprises to provide a reliable, market-based source of capital. By establishing community development organizations that are locally led and also launching social enterprises to fund the work, Nuru enables nationals to lift an entire region out of extreme poverty within seven years.

Nuru Kenya (NK) Education Program The Nuru Kenya (NK) Education Program aims to increase child literacy to the Standard Two (Grade 2) level among rural primary school children through student-centered teaching and literacy-focused interventions. NK Education works with the Ministry of Education to supplement existing classroom curriculum in rural primary public schools located in communities where Nuru farmers live. NK Education Outreach facilitates student workshops that focus on the five main components of literacy development: phonemic awareness, phonics, fluency, vocabulary, and comprehension. NK coordinators aim to build student confidence and increase student participation in literacy development through games, reading activities, and creative projects.

6

Since 2012, NK Education has been working with schools in Kuria West2. School selection is based on the Ministry of Education’s recommendation in terms of school-level need. In 2014, NK Outreach was offered to public school children in Pre-Unit (Kindergarten) through Grade 7 at a total of 17 schools across two divisions: Isibania (9 schools) and Mabera (8 schools). NK Education will expand into the Kehancha division in 2015. Figure 1 provides maps of Kuria West district and its divisions. All participating schools received similar outreach across the 2012, 2013, and 2014 academic years3.

Figure 1: Maps of Kuria District and Divisions in Kenya

Workshops conducted for up to four hours of classroom time per month were held in all Nuru schools in both the 2013 and 2014 academic years. Periodically, workshops were cancelled due to school exams or events that were not communicated by teachers. In May of 2014, NK Education cancelled two weeks of workshops because of the funeral and community events related to the death of Nuru Kenya Chairman Philip Mohochi. Based on monitoring data collected at every workshop, the average child to facilitator ratio in 2014 was 9 to 1, with slightly more than 5,700 children participating in NK Outreach across the first three quarters of 2014. In 2013, the average child to facilitator ratio was 8 to 1, and slightly more than 4,100 children participated in NK Outreach.

About Uwezo Literacy Assessment

The Uwezo Literacy Assessment is a tool developed by a regional initiative between Kenya, Tanzania, and Uganda called Uwezo, which strives to improve the competencies in numeracy and literacy among primary school aged children in Grades 2-74. Uwezo administers the Uwezo Literacy Assessment to children in Grades 1-8 across Kenya.

2 The World Bank cites Kuria West district as being >70% below the Kenyan rural poverty line ($1.25/day). 3 The standard school year in Kenya consists of three terms. Approximate dates for each term are: January-March; April-July; August-November. 4 For more information on Uwezo, please visit www.uwezo.net.

7

As NK Education’s goal is focused on increasing literacy for primary school-aged children in rural communities, it implements the Uwezo tool to assess progress toward Standard Two literacy with the English portions of the assessment5. The Uwezo tool can assess progress toward literacy simply and frequently; therefore, NK Education uses the tool in a select number of schools to both monitor and evaluate NK Outreach at the beginning and end of the year6. NK Education’s Uwezo assessment measures a student’s English literacy at the letter, word, paragraph, and story levels. Testing begins at the paragraph level to conserve time. For example, if the student struggles to read the paragraph during testing, he or she is directed to the word level and instructed to read any five words from a word list. If he or she struggles at the word level, the student will be directed to read letters. If the student displays successful comprehension at the letter level, he or she will advance back up by level of proficiency until it is clear that the child cannot advance any further. On the other hand, if the student first reads at the paragraph level successfully, he or she will be directed to the story level and is then asked two questions to assess comprehension (see Appendix 1 for a complete write-up). In addition to recording the student’s literacy level, the assessment records each student’s grade, name, age, and gender.

Methodology Nuru implements a multi-tiered approach to assess progress toward impact. These three tiers include a longitudinal study with comparison to the formal Uwezo national assessment (Tier 1); a quasi-experiment approach aimed to mimic a treatment and control by comparing literacy proficiency during a pre-intervention (control) year and a post-intervention (treatment) year (Tier 2); and the monitoring of other select schools (Tier 3). These approaches are more fully explained in the “Nuru Kenya Education Impact Assessment: Results of the 2013 Education Surveys.” The key evaluation questions and indicators for all assessments are listed below in Table 1. This paper is limited to the results of the Tier 2 evaluation strategy through a study at one school that received its first year of NK Outreach in 2014. Moving forward, this paper will refer to 2013 as the pre-intervention year and 2014 as the post-intervention year.

Table 1: Education Evaluation Indicators

Education Indicator Evaluation Question % of average literacy proficiency What is the impact of NK Education on

progress toward English Literacy? % of children who can read at least at the paragraph level

What is the impact of NK Education on a student’s ability to read at least at the paragraph level?

5 Kenya’s official languages are English and Kiswahili. 6For more detail, see “2013 Nuru Kenya Education Impact Assessment” for full details of Nuru’s strategy for evaluating NK Education during previous iterations.

8

For the Tier 2 evaluation strategy, Nuru compared the change in literacy progression across one school7 during the pre-intervention year (2013) with the change in literacy progression during its first intervention year (2014). This approach aims to determine if the net annual change in literacy (between the beginning and end of each year) is greater for the post-intervention year than the pre-intervention year. Moreover, NK M&E captures shifts in the number of certified teachers, school infrastructure, and materials available to public school children from year to year in each school. These data help M&E understand whether there were changes in the school environment that might prevent a comparison of annual outcomes in literacy proficiency.

Sampling Frame

NK M&E administered the Uwezo literacy assessment to 100 percent of public school children in Grades 2-7 in attendance on the given survey dates for both 2013 and 2014. Surveying 100 percent of public school children (effectively a school census) versus a sample ensures NK M&E is able to decrease its margin of error in order to provide more precise estimates of data at each grade level in addition to at the aggregate school level. Table 2 (below) displays the number of public school children surveyed by each grade at the beginning of year (BOY) and end of year (EOY) for both the pre-intervention and post-intervention years. Note the variation in number of public school children from beginning of the year to the end of the year. Even though NK Outreach is conducted in Pre-Unit (Kindergarten) through Grade 7, assessment begins at Grade 2, consistent with Uwezo’s assessment methodology. When comparing data over multiple years, note that Grade 7 students matriculate out of primary school and a new group of Grade 2 students are assessed as they enter into the Uwezo testing range.

Table 2: Number of Public School Children Surveyed by Grade and Year, 2013 -2014

2013 (Pre-Intervention Year) 2014 (Post-Invention Year)

BOY EOY BOY EOY

Grade 2 36 39 52 55 Grade 3 45 49 34 40 Grade 4 33 32 47 53 Grade 5 44 39 29 34 Grade 6 43 42 36 41 Grade 7 33 27 43 50 TOTAL 234 228 241 273

7 The specific name of the school Nuru Kenya surveyed is not included in the report to preserve the anonymity of the study.

9

Data Collection

NK M&E enumerators collected data by coordinating available dates and times with both the Ministry of Education and NK Education. The survey collected each child’s name, age, and gender as well as the literacy score for letter, word, paragraph, story, and comprehension levels8. Prior to survey implementation, two NK M&E field managers trained enumerators and selected thirteen to conduct the Uwezo assessment9. The enumerators collected BOY literacy data in January and February 2014; and collected EOY literacy data in mid-October for both 2013 and 2014. All children present in Grades 2-7 were assessed individually in the sample school.

Analysis Data were analyzed at the school and grade levels in the selected school for approximately 240 public school children in both 2013 and 2014, as described previously in this paper, to assess pre- and post-intervention changes in literacy proficiency. First, scores for public school children’s progress toward Standard Two literacy were assessed at the aggregate level and then by grade level for 2013 and 2014. Progress toward Standard Two literacy at the BOY and EOY time points were assessed via aggregate scores created by M&E: 0 = None, 1 = Letter, 2 = Word, 3 = Paragraph, 4 = Story, 5 = Literate. These scores were then averaged to determine the mean proficiency level for each year. Public school children scoring at least a 3 on the assessment were classified as “paragraph or above”; public school children scoring a 5 were classified as Standard Two literate. Examples of the assessment are included in Appendix 1. In addition to average literacy proficiency, achievement of paragraph proficiency and above was also assessed for all public school children for 2013 and 2014. Finally, the overall progression of public school children in each literacy category across 2013 and 2014 within two time periods, BOY and EOY, was assessed across grade level.

Results The following results section is comprised of two sub-analyses: results of progress toward full literacy and results of achievement of paragraph level and above for the selected school. Overall, results show that NK Education demonstrated progress towards impact. Average literacy rates for public school children attending NK Outreach increased by 4 percent from the pre-intervention year. Additionally, public school children in the Grades 2-5 grouping progressed at a greater rate towards Standard Two literacy and in achieving paragraph or above literacy over time when compared with the public school children surveyed in 2013 (pre-intervention year).

8 Literacy refers to Standard Two literacy, or the level at which a child should be able to read in the second grade. 9 NK M&E trained more than 13 enumerators. However, based on performance during the training, only 13 were selected to collect field data.

10

Results of Progress Toward Full Literacy

Between the pre-intervention year (2013) and the post-intervention year (2014), average literacy proficiency increased for all public school children from 2.56 to 2.66, respectively. This increase indicates that, pre-intervention, literacy progression started closer to the word level and, post-intervention, shifted closer to the paragraph level as shown in Table 3 below. In order to further explore this change, all grades surveyed (2-7) were separated into two groupings: Grades 2-5 and Grades 6-7. As would be expected, the younger public school children (Grades 2-5), on average, were at the lower end of the spectrum (closer to the letter level) while the older public school children (Grades 6-7) were at the higher end of the spectrum (story level and above) as outlined in Table 3 below.

Table 3: Average Literacy Proficiency Level Across 2013-2014

Progress toward Standard Two Literacy10

All Grades Grades 2-5 Grades 6-7

2013 (Pre-Intervention Year) 2.56

(n = 462)

1.84

(n = 317)

4.13

(n = 145) 2014 (Post-Intervention Year) 2.66

(n = 514)

1.95

(n = 344)

4.08

(n = 170)

Literacy Scores: 0 = None, 1 = Letter, 2 = Word, 3 = Paragraph, 4 = Story, 5 = Full Literacy Interestingly, when reviewing results by grade groupings, it was clear that the driver of the increase in literacy proficiency in 2014 was the progress of children in Grades 2-5. As seen in Figure 2 below, the public school children in Grades 2-5 showed a change in literacy proficiency across years of 6 percent11. More specifically, between the beginning and end of 2013 (pre-intervention year) the average literacy proficiency change was 1.84, while during the post-intervention year the average was 1.95. This is in contrast to public school children in Grades 6-7 who showed a drop of 1 percent in literacy proficiency, from 4.13 during the pre-intervention year to 4.08 during the post-intervention year.

10 The number of observations (n) in Table 3 include the total number of individual test scores from BOY and EOY that were averaged for the specified year. 11 Of the six other schools assessed through M&E’s Tier 3 monitoring strategy, Grades 2-5 showed greater improvements in average literacy proficiency compared with Grades 6-7. However, the improvements in paragraph level and above were only seen for the Tier 2 school assessed in this paper and one additional school. It is likely that the shift in strategy to group students by performance influenced these outcomes in the Tier 2 school.

11

Figure 2: Percent Change in Average Literacy Proficiency by Grade, 2013 (Pre-Intervention Year) to 2014 (Post-Intervention Year)

Results of Achievement of Paragraph Level and Above

Another lens that can be used to measure the impact of NK Outreach is assessing public school children performance around higher literacy proficiency levels such as achievement of paragraph, story, and full literacy. Table 4 (below) outlines the percentage of public school children who achieved proficiency in reading at paragraph level and above, which is inclusive of the story and Standard Two Literacy levels. At this level, a clear increase is seen between the pre-intervention year of 5 percentage points (from 48 percent to 53 percent) and the post-intervention year of more than double, 12 percentage points (from 46 percent to 58 percent). See Figure 3 below.

Figure 3: Changes in Paragraph Level and Above Across the 2013 (Pre-Intervention Year) and 2014 (Post-Intervention Year)

48% 53%

46%

58%

0%

10%

20%

30%

40%

50%

60%

70%

Beginning of Year (BOY) End of Year (EOY)

2013 (Pre-Intervention Year) 2014 (Post-Intervention Year)

12

As in the earlier assessment, the same grade cut points were utilized in assessing the selected public school children’s ability to read at the paragraph or above level to determine the drivers of the change, shown in Table 4 (below). Overall, public school children in the lower grades (2-5) increased their paragraph proficiency in both pre-intervention (2013) and post-intervention (2014) years.

Table 4: Differences in Paragraph Level and Above, Within and Across 2013 (Pre-Intervention Year) and 2014 (Post-Intervention Year)

Achievement of Paragraph level and above

BOY EOY (inclusive of story level and full Standard Two literacy) Pre-Intervention Year (2013) 48% 53% Grades 2-5 27% 36%

Grades 6-7 91% 91%

Post-Intervention Year (2014) 46% 59% Grades 2-5 27% 42%

Grades 6-7 86% 88%

However, the average score of public school children at the paragraph level or higher in Grades 2-5 rose 15 percentage points in the post-intervention year (2014) compared to only 9 percentage points during the pre-intervention year (2013). For Grades 6-7, paragraph and above proficiency did not change during the pre-intervention year (2013). Within the 2014 assessment, there was a 2 percentage point increase in scores for public school children in Grades 6-7. Results for these frequencies are displayed below in Figure 4.

Figure 4: Percentage Point Change in Paragraph Level and Above, 2013 (Pre-Intervention Year) Compared to 2014 (Post-Intervention Year)

2013 2014

Grades 2-5 9% 15%

Grades 6-7 0% 2%

0%

2%

4%

6%

8%

10%

12%

14%

16%

Freq

uenc

y of

Stu

dent

s Te

stin

g at

Pa

ragr

aph

and

Abo

ve (%

)

13

Limitations This study includes various limitations. In 2013, M&E proposed to track public school children across multiple years in order to create a time-series assessment of NK Education. However, this turned out to not be possible as the number of public school children fluctuates throughout the academic year for a variety of reasons (i.e. at the beginning of the 2014 academic year, 241 public school children were tested; at the end of the year, 273 public school children were tested). Therefore, NK M&E determined that the best course of action for future analyses was to generate an aggregate score at each time point (BOY and EOY).

Conclusions and Recommendations Compared with scores from the pre-intervention year (2013), the assessment presented in this report shows NK Education had a positive impact on average literacy proficiency for public school children in Grades 2-5. Similarly, the data reveal much greater increases for children in Grades 2-5 after participating in NK Outreach compared with public school children in Grades 6-7 when examining changes in the achievement of paragraph level and above. Furthermore, because of the study design, these differences are not caused by sampling error and the positive impact is attributable to the NK Education Program. Recommendations for the NK M&E and Education teams to consider as a result of these findings:

1. Focus outreach on Grades 2-5 rather than Grades 2-7, as the younger public school children demonstrate increased rates of progression towards literacy proficiencies.

2. Group public school children by average literacy proficiency in order to better work with lower performing public school children (based on the beginning of year scores). This would ensure that the outreach provided is at an optimal level for each student.

14

Appendix Appendix 1: Example of Uwezo Assessment

Figure 1: Example of “Letters” and “Words” literacy level as

presented to public school children during testing.

15

Figure 2: Example of “Paragraph” and “Story” literacy level as presented to public school children during testing.