Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)

EffectivenessofInteractiveDistanceInstruction:ExperimentalEvidencefromGhanaianPrimarySchools

JAM I E J O HN S TON – S TAN FO RD UN I V E R S I T Y

CHR I S TO PH E R K SO L L – MATH EMAT I C A P O L I C Y R E S E A RCH

GhanaEducationEvidenceSummit2017

PhotoCredit:VarkeyFoundation

Howcanwedeliverhigh-qualityteachingtoruralstudents?

• Studentsinruralareassignificantlyunderperform(OECD2013,GES2012,WorldBank2012)

• Inremoteruralareas,shortageoftrainedprimaryteachers(WorldBank2012)

• Teachingqualitymattersforstudentlearning(Glewwe etal.2011,McEwan2015)

• Structuredpedagogyinterventionsappeartohavelargest,mostconsistentpositiveeffectsonlearning(Ganimian &Murnane 2016,Snilstveitetal2015,Kremeretal.,2013)

• Littleknownabouteffectivenessofinteractivedistanceinstructionmodels

VideoCredit:VarkeyFoundation

MGCubedSatelliteClassProgram• Varkey FoundationMGCubedProgram

• Equipped72schoolswithtechnologypackage

• Broadcastslivelessonsdailytoremoteclassrooms§ 6trainedstudioteachers,eachbroadcastingto12classesatatime§ 1hourmathand1hourEnglish,5daysaweek

• In-personteacherstrainedas“facilitators”

• Targetsmostmarginalizedstudentswithinschools§ Grades2-5§ Girlsandboys§ Identifiedaccordingtocriteria:distancetoschool,numberofsiblings,truancy,age

PhotoCredit:VarkeyFoundation

Girls’AfterSchoolProgram• Providesafter-schoolsessionstoin-schoolgirlsandout-of-schoolgirlsincommunity

(whohaveleftschool)

• 1 hourperday,4daysaweek

• Engagegirlsintopicsincluding§ Earlypregnancy§ Earlymarriage§ Girls’rights§ Financialliteracy

• Providesaccesstoadultfemalerolemodels

• Boys’monthlyafter-schoolprogramstartedinresponsetodemandforsimilarprogramPhotoCredit:VarkeyFoundation

TheoryofChangeImprovequantityandqualityofprimaryschooleducation

Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence

Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity

Improvecurricularcontent:§ Moreengaging§ Moreinteractivity

In-Schoo

l

Gainsinnumeracyandliteracy

ResearchQuestions:

1) Istheprogrameffectiveatincreasingskills?

2) Whatarepotentialmechanismsofchange?

EvaluationApproach

cohortreceivesintervention

ClusterRandomizedControlledTrial(RCT)§ 70treatmentand77controlschools,assignedatbaseline§ Followedcohortofstudents(grades2-4atbaseline)fortwofullyears

Interventionbegins Year1 Year2

Jan2014baseline

Jun-Jul2014monitoring1

Year0

Feb-Mar2015monitoring2

Jun-Jul2016endline

Jan-Feb2017postendline

Jun-Jul2015midline

Mar-Apr2016monitoring3

Cohort:Grade2 Grade3 Grade4

Cohort:Grade3 Grade4 Grade5

Cohort:Grade4 Grade5

Cohort:Grade5(excludedfromRCT)

datacollections

2yearimpact

Grade6

1yearimpact

cohortnotreceivingintervention

Data1) Skillsassessments

§ Baseline,midline,andendline§ Literacy:EarlyGradeReadingAssessment(EGRA)§ Numeracy:EarlyGradeMathematicsAssessment(EGMA)

2) Studentandteacherdata§ Studentsurveysatbaseline,midlineandendline§ TeachersurveyinYear2§ Attendancemeasures(recordsandspotchecks)inYears1and2§ Qualitativeinterviewsandfocusgroupdiscussionswithstudents,teachersandadministrators

3) Classroomobservations§ Year2§ AdaptedStallingsObservationProtocol§ Snapshotsyieldingproportionofclasstimeon(1)activeinstruction,(2)passiveinstruction,(3)classroom

management,and(4)offtaskactivities

MeasuringLearning:EGRA&EGMA• EarlyGradeReadingAssessment(EGRA)

§ Oraltestmeasuringsubtasks:§ Lettersperminute§ Wordsperminute(WPM)<- focalmeasure§ Inventedwordsperminute§ Listeningcomprehension§ Oralvocabulary§ Readingcomprehension

§ BaselineWPM:11.5§ Baselinestandarddeviation:21.2

• EarlyGradeMathematicsAssessment(EGMA)§ Oraltest,measuringsubtasks:

1. Numberidentification(20%)2. Quantitydiscrimination(20%)3. Missingnumberpatterns(20%)4. Addition/subtraction/wordproblems(40%)

§ Baselinemeanscore:46.0percentcorrect§ Baselinestandarddeviation:20.1

focalmeasure:combinedtotal

SOURCE:RTIInternational,2009

EGRAWPMExamplePassage:

• N=5485atbaseline- 60%girls- Averageage=10.9- ~8householdmembers- 50%mothersattendedschool- 60%fathersattendedschool

• Treatmentandcontrolbalancechecks- Balanceonobservablesatbaseline,midline,andendline- Midlineattrition=13%- Endlineattrition=16%- Nodifferenceinattritionbetweentreatmentandcontrolgroups

SampleCharacteristics

MainResultsWhatistheeffectofsatelliteprogramonliteracyandnumeracyskills?

◦Classessignificantlyimprovednumeracy◦ 0.22standarddeviationimprovementinyear1◦ 0.26standarddeviationimprovementinyear2

◦Gainsinmathscoreslasted oneyearaftertheprogramended◦ 0.23standarddeviationgainsforstudentstestedoneyearafterprogramended

◦Aftertwoyears,marginallysignificantimprovementinliteracy◦ 0.07standarddeviationgainsaftertwoyears

GainsinNumeracyandLiteracy

◦ After2yearsofclasses,largeandsignificantimprovementinfoundationalskills◦ 0.82standarddeviationgaininlettersperminute◦ 0.17standarddeviationgainininventedwordsperminute◦ 0.17standarddeviationgaininoralvocabulary

LiteracySubtasks

0.07

0.82

0.17 0.17

0.04 0.000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Words per minute Letters per minute Invented words per minute

Oral vocabulary Listening comprehension

Reading comprehension

Stan

dard

dev

iatio

n ga

ins

Two years of classes***p<0.01,**p<0.05,*p<0.1

***

****

**

◦ Afteroneyearofclasses,significantimprovementinallnumeracysubtasks◦ Aftertwoyearsofclasses,greatergainsinnumberidentityand missingnumbertasks

NumeracySubtasks

0.2

0.150.17

0.19

0.34

0.12

0.250.21

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Number identification Quantity discrimination Missing number Addition/subtraction

Stan

dard

dev

iatio

n ga

ins

One year of classes Two years of classes***p<0.01,**p<0.05,*p<0.1

***

******

***

***

**

******

*** *

MechanismsWhatarethepotentialdriversoflearninggains?





In-Schoo

l


§ ->Noevidenceofeffectsonattendance§ ->Noevidenceofeffectsontime-on-task§ ->BUT:In-classroomfacilitatorsengagedinon-task

activities(includingactiveinstruction)morethanhalfofthetimethatthevideowasbeingshown,essentiallyincreasingclassrooms’teacher-studentratio





In-Schoo

l


§ ->Noquantitativeevidenceofimprovedteachingmethods§ ->Qualitativeevidencesuggestsimprovements

Mechanisms:TeachingPedagogy• Studentfocusgroups/interviewssuggestimprovementsinlearningexperience

“Sometimesteachersleaveyoubehindifyoudon’tunderstandandevencaneus.MGCubedteacherstrytheirbesttomakeusunderstandlessons.”

• Facilitatorsseenaseffectiveteachingintermediaries“ThefacilitatorintheMGCubedclassexplainswhatthestudioteacherteachesifwearefindingitdifficulttounderstandthestudioteacher.”

• Newteachingmethodsandapproacheslearnedaspartoffacilitatorrole“Andtheytaughtuswayswecanusetoteachthem...Thoughweweretaught‘thinkpairshare’atthetrainingcollegeIwasnotusingitsowhenwehadtheprogrammerecentlyIwentbackandamnowusingit.”





In-Schoo

l


§ ->Stillinvestigating

PolicyImplicationsandNextSteps• Interactivedistancemodeleffectiveatincreasingliteracyandnumeracyskills

• Increasedinstructorpresenceinformofdistanceand in-personinstructoranimportantlevertoexplore– aretherewaystomakerolesmorecomplementary?

• Forthcomingisacosteffectivenessanalysistodeterminehowinterventioncomparestootherheavier-handedstructuredpedagogyprograms

• Relevanceofinteractivemodelincreasingascostsofinternetanddistanceinstructiondecreasing andinstitutionsincreaseuseofonline/blendedinstruction

LinktoWorkingPapercepa.stanford.edu/sites/default/files/wp17-08-v201708.pdf

ThankYouQuestionsandfeedbackwelcome!

[email protected]

Documents

Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)