MPrep Impact Report March 2013 FINAL - Eneza Educationenezaeducation.com/.../10/MPrep-Impact-Report-March-2013-FINAL.pdfChandaria has approximately 180 students in Class 7 and 150

http://mprep.it Contact: [email protected]

2012 Impact Study Report

Updated: 15 March 2013

The image part with relationship ID rId2 was not found in the file.

INTRODUCTION The following paper outlines the impact findings of MPrep’s study tool for students, as determined by a pilot with 30 Class 7 and Class 8 students selected randomly from M.M. Chandaria Primary School in the Baba Dogo region of Nairobi, Kenya. This study began 1 February 2012 and went until the end of the 2012 school year in mid-November. (Kenya follows the calendar year for its school year calendar.) End of year results from the national Kenyan Certificate of Primary Education (KCPE) exam became available in January 2013. Impact is defined as student achievement results. We are specifically looking at how students who had access to MPrep are different from their peers in the same school with the same teachers. This is measured by summative academic exams, given by the school to all students. We are one of the only educational technology ventures that has ever controlled for access to the technology itself. It is not our aim, however, to prove that technology alone solves educational problems. We also measured student behavioral changes, and observational differences in classroom behavior in this study. DESCRIPTION OF PILOT During this 9-month pilot, 30 students participated in study sessions with the MPrep student study tool. This supplemental study tool consists of: • Content-aligned quizzes in every subject on the Kenyan National Syllabus for Class 7-8 • Individualized feedback for right and wrong answer choices • Ranking and self-assessment messages • Broadcasts of praise – messages that congratulate students according to uncertain incentives. These

include: o Praise for highest students in school o Most improved students in school o Most active students in school

Congratulations to Mary Akinyi of MM Chandaria. She was the most active user this week! Text 8512 for more chances to review with MPrep.

Broadcasts of Praise





Broadcasts of Praise We hypothesized that by using MPrep for only 30 minutes per week over a 9-month period, students will improve their academic scores more than the norm. In addition, we also propose that MPrep causes students’ study habits to change, wherein, students that utilize MPrep frequently study more than before and enjoy school more than before. METHODOLOGY Background Information on Chandaria Primary We chose 30 students at random – 20 from Class 7 and 10 from Class 8 – from a typical public school in Nairobi, M.M. Chandaria Primary known as “Chandaria.” Chandaria has approximately 180 students in Class 7 and 150 students in Class 8. The median income level is below the poverty line at around $4-5 USD per day. Many of the students feed from the nearby Korogocho Slum area. This slum is renowned for its high prevalence of crime and drug use. It is considered “the most dangerous slum in Nairobi.” For information about the population around Chandaria, see http://www.aphrc.org/insidepage/?articleid=470 Phase 1 of Pilot: Early Achievement and Student Behavior Findings On February 1st, students began using MPrep for 1/2 hour – 1 hour per week in what we call “study sessions.” From February to April, we studied the students’ usage and academic scores while using MPrep during these study sessions. Students were grouped into same-gender groups of 2-3 students per phone, and they were encouraged to work collaboratively on study sessions that we recommended for them to take. At each study session, we took observational notes about the children, looking closely at

• Interest level • Engagement • Collaborative habits with other students • MPrep scores

During the first 6 weeks of MPrep engagement, these were our findings:

1. Student achievement increases over time

Students began with quizzes that were less difficult on 14 February 2012, averaging 62%. As the difficulty of MPrep progressed, however, students’ scores began to decrease, averaging at 56% on March 13. The difficulty of MPrep continued to increase from March 13 onward, but the students’ achievement averages began to increase from here, reaching 66% by the end of the 6-week study on March 27.

MPrep affects student achievement over the course of a short period, not only with its content, but also with its ability to change study behavior.





This demonstrates that students began to adapt to the study sessions, utilizing more focus and effort over time. We observed this happening within the classroom setting, mirroring typical student behavior. Once the students were given clear ideas of their data, they made efforts to immediately improve. Thus, MPrep affects student achievement over the course of a short period, not only with its content, but also with its ability to change study behavior. Students learn concepts through MPrep, and they also learn how to adapt behaviors in response to the feedback they receive from MPrep. From this information, we decided to engage in a full study from baseline to end of year exams. (See next section.)

2. Student usage peaks when broadcast praise messages are utilized Although we experimented with using airtime as a reward on February 28, there was minimal impact. Unsurprisingly to any teacher, student usage peaks when MPrep begins to use praise messages. These messages praise students based on active usage, highest scores, and most improved. As soon as we began using these praise messages on March 20, we observed a surge in interest in MPrep. In order to keep this enthusiasm elevated, we instated a common classroom and psychological principle noted as “uncertain incentives.” Because students were unsure of what they would be awarded for and whether they would be awarded as a group or as an individual, they continued their engagement over time.

3. Students study more after using MPrep and the school community feels positive about MPrep At the end of Term 1, we gave the students a survey to measure:

• Attitudes toward MPrep • Attitudes toward school • Ways to improve MPrep

(See Appendix for survey questions) As shown by the “How You Feel About MPrep” chart, 100% of students reported positive feelings toward MPrep, 85% of which reported strong positive feelings. In addition, 80% of students reported studying more after MPrep was introduced into their lives. From anecdotal conversations with students, we found that students went home and paid to utilize MPrep on their own after feeling comfortable with using it. One

50%

55%

60%

65%

70%

Chandaria Impact Study: Mean Score

Average Score

85%

15% How You Feel About MPrep

Really Like It It's ok

Don't like





student, Linet, reported getting 40 Kenyan shillings each week from her mother to study using MPrep. Almost every student reported using MPrep at home, outside of our regulated study sessions. When asked whether their teachers and parents approved of MPrep, an astounding 95% of parents and 80% of teachers had strong positive feelings about MPrep.

Phase 2: 6-Month Student Achievement and Behavior Data To more thoroughly measure our impact, MPrep began a 6-month impact study beginning with a baseline assessment given on March 30, 2012. This assessment was not created by MPrep; rather, it consisted of the typical end of term assessment given to students. Since students are heavily tested in Kenya, we decided that it was unethical to provide the students with an additional test, especially when we could utilize this one for measurement. Then, we could compare the class averages to the MPrep Student Average and see how each measurement progressed throughout the year. At the end of the year, all students are given the KCPE national exam, which we will compare to the termly measurements stated here. BASELINE RESULTS According to the baseline exam, which consisted of a summative assessment of all materials taught during Term 1, MPrep students on average scored less than the class average in both Class 7 and Class 8. For Class 7, students scored lower than the average in every subject. In Class 8, students scored lower than the average in every subject except Kiswahili and CRE. Class 7 – Raw Scores for 30 March 2012 Baseline

English Kiswahili Math Science Social Studies

CRE TOTAL

MPrep 46 48 39 45 26 19 234 AVG 47 51 42 51 27 20 243 Class 8 – Raw Scores for 30 March 2012 Baseline


CRE TOTAL

MPrep 45 56 41 55 29 19 250 AVG 47 53 44 55 31 17 252

80%

15%

5% What Your Teachers Think of MPrep

Like it

Think it's ok Don't like it 95%

5%

What Your Parents Think of MPrep

Like it

Don't know about it Think it's ok

Don't like it





QUANTITATIVE RESULTS Our quantitative results are from the end of year tests, given in November 2012. For Class 7, these tests were summative exams that included the entire curriculum for the school year. For Class 8, the results from the Kenyan Certificate of Primary Education (KCPE), a national exam given as an exit exam/qualifying exam for secondary school. The end of term 2 (EOT 2) exam was given to all students in Class 7 and Class 8 on July 31, 2012 as what we consider a “benchmark exam.” For Class 7 students, this exam consisted of a summative exam that spanned all topics covered by the teacher during the course of the term. A small note on rigor: It is difficult to compare one test to the other, as the Term 2 exam is often much more difficult than the Term 1, and the KCPE or end of year exam is usually much more difficult than the Term 2 exam. This material is usually more difficult and rigorous than the material from Term 1. Despite this, we are specifically looking for how MPrep students compare to the average Chandaria student, so the amount of rigor does not matter. Our quantitative findings are as follows:

1. Overall, MPrep students increase their academic scores greater than the average

a. MPrep Class 7 students increased their scores 9 points more than the average Chandaria student.

b. In accordance, the average Class 8 MPrep, who started 2 points below the class average, scored 2 points above the average on the KCPE exam.

2. Class 7 MPrep students exhibited greater growth in every subject, when

compared to the average student.

3. Class 8 MPrep students exhibited greater growth in every subject except for Kiswahili and Science, when compared to the average student.

4. MPrep students in both Class 7 and Class 8 either caught up to, or surpassed their

peers after starting with marks below them.

Overall, MPrep students increase their academic scores greater than the average.





Class 7 Baseline Results (30 March 2012)


CRE TOTAL

MPrep 46 48 39 45 26 19 234 AVG 47 51 42 51 27 20 243 Class 7 End of Term 2 Results (31 July 2012)


CRE TOTAL

MPrep 49 51 43 48 22 17 235 Change +3 +3 +4 +3 -4 -2 +1 AVG 52 51 46 47 22 18 237 Change +5 0 +4 -4 -5 -2 -6 Class 7 End of Year Results (15 November 2012)

English Kiswahili Math Science Social Studies/CRE combined

CRE TOTAL

MPrep 45.1 53.5 48.2 41.8 50.2 18.4 239 Change -‐0.9 +5.5 +9.2 -‐3.2 +5.2 -‐0.6 +5 AVG 46.1 53.2 48.3 42.1 49.4 18.2 239 Change -‐0.9 +2.5 +6.3 -‐8.9 +2.4 -‐1.8 -‐4 Class 8 Baseline Results (30 March 2012)


CRE TOTAL

MPrep 45 56 41 55 29 19 250 AVG 47 53 44 55 31 17 252 Class 8 Mock KCPE Results (31 July 2012)


CRE TOTAL

MPrep N/A N/A N/A N/A N/A N/A 225 Change -25 AVG N/A N/A N/A N/A N/A N/A 224 Change -28





Class 8 KCPE Results (5 December 2012)


CRE TOTAL

MPrep 50.1 48.2 37.2 39.3 44.44 - 219.20 Change +5.1 -8.2 -3.8 -15.7 -3.56 -30.8 AVG 49.46 45.94 38.53 39.8 43.53 - 217.26 Change +2.46 -7.06 -5.47 -15.2 -4.47 -34.74

SURVEY RESULTS The following results are from our 2012 end of year survey:

1. Study time From this study we found that MPrep positively affects the study behavior. Students who are on the MPrep platform studied more hours outside School than those who are not on the MPrep platform. 67% of MPrep students study more than 4 hours after school per day compared to 55% of students who are not on MPrep. When students stay focused and study for some extra hours after school, they increase their chances of success.

2. Parent communication Overall, parent communication happens once per term. Most of the parents come to school at least once a term to discuss the student performance with the teachers. We also found out that although the parents come at least once per term, they always discuss school matters with their children at home. We see MPrep as being a potential communication tool for parents, giving them automatized reports about their students, and limiting frequent travel to stay heavily involved.





3. Level of Education

Most of the students expect to attain a degree or Post graduate diploma. This is a good indication that the parents, the teachers and other stakeholders need to do more to help these children realize their dreams in life What these findings means for us: As MPrep these findings are important for us. They helps us to serve the kids better and to better placed to advise the parents, teachers and the education stakeholders.

OTHER FINDINGS Most Active MPrep Schools 2011 KCPE 2012 KCPE % Increase Malanga Primary 210 234 11.4% Muthurwa Primary 199 230 15.6% Some of our most active schools on MPrep also self-reported their results. They stated that they “used MPrep in order to have more individualized teaching instruction and figure out the gaps” in their classes. Within the Impact Study Group there was also the first student ever to receive a scholarship to a National Secondary School (Alliance Boys). OBSERVATIONAL From 9-months of observational notes and mid-year surveys our qualitative findings are as follows:

1. Students enjoy using MPrep Through survey evidence and anecdotal evidence from classrooms around Kenya, we know that students enjoy using MPrep. Our group of students even showed up to attend our study group while schools were not in session due to a nationwide teachers’ strike. Here they are (out of uniform!)





2. Students are more aware of what they know and don’t know after using MPrep

MPrep engages our students in goal-making sessions. During these session, our students have become clear of what they do and do not need to study.

3. Students collaborate while using MPrep

Students prefer to work in groups to study, even when each of the students has their own phone.

4. Students compete with each other while using MPrep After MPrep sessions, students are often overheard comparing their marks and ranks. Most students have their rank for the school and region memorized.

5. Students engage in competition with peers outside the study group to use MPrep Most students have begun to tell friends at other schools about MPrep, encouraging them to participate and compare ranks.

6. Students study more outside of their “MPrep time”

80% of students report using MPrep at home, despite the cost.

7. Students say their parents and teachers are overtly positive about them using MPrep Teachers and parents have also called us, texted us and given feedback in person. All feedback about the product has been positive.

CONCLUSIONS

1. MPrep positively impacts student academic outcomes 2. MPrep positively impacts student behaviors toward education 3. MPrep engages students in collaboration and competition with their peers in school and

outside of school 4. MPrep is well-received by the school environment including teachers and parents

FURTHER RESEARCH We realize this study is limited in scope, time-length, and statistical numbers. In addition, most of our evidence comes from student use of the study tool, whereas our project engages the entire educational community – teachers, head teachers, parents, and students. In the future, we aim to measure these additional findings:

1. How MPrep impacts parental engagement in school 2. How MPrep impacts teacher behaviors, including teacher attendance, student evaluation of teachers,

and head teacher evaluation of teachers 3. How students’ academic outcomes change over a longer period of time (2-3 years of using MPrep).

Does our tool put students on a more positive academic course? 4. How does MPrep affect students academic outcomes at other schools?

With these findings we hope to prove that MPrep, and its mission of uniting educational stakeholders around student achievement has the ability to change educational outcomes in Kenya and can serve as a model for the rest of the developing world.






ADDITIONAL INFORMATION You can visit the following to gain more information about MPrep: Site: http://mprep.it Twitter: http://twitter.com/MPrepKenya Facebook: http://facebook.com/MPrep Blog: http://mprep.wordpress.com For additional information about MPrep’s impact findings, please contact: Isaac Kosgei – Community Relations Director P.O. Box 29128 – 00100 GPO Nairobi, Kenya [email protected]

Documents

MPrep Impact Report March 2013 FINAL - Eneza Educationenezaeducation.com/.../10/MPrep-Impact-Report-March-2013-FINAL.pdfChandaria has approximately 180 students in Class 7 and 150