53
Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riaz

Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riaz

Page 2: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Motivation School based management (SBM) has become a popular reform area in the past decade:

◦ There is redistribution of decision-making authority to the school council.

◦ School committees/councils are established and receive annual grants for school improvement.

What do we expect?

School grants serve as catalysts for communities to participate the in school management process (1)

◦ Informal Parent-Teacher Associations (PTA) will evolve into formal structures (school councils) over time.

◦ Informal networks will provide avenues for community-level collective action to fix what is wrong with the school.

Community engagement induced by school grants, in turn, will(2): ◦ Result in better mapping of community-identified needs (allocative efficiency)

◦ Lead to improvements in resource-use efficiency at school level (technical efficiency)

What happens in practice?

Limited awareness of community members about their rights and roles in school councils (3)

◦ Capture of school grants by Head Teacher/sub-district officials (4)

◦ Leakages/underutilization of grants(5)

Even when they are aware, community participation in school councils remains low due to: ◦ Absence of mechanisms to facilitate dialogue between communities and schools

◦ Political/administrative interferences discouraging effective participation to take root

◦ Lack of ownership of school based management policies at the community level

1 Pradhan et al., 2011; 2 Bruns et al., 2011 3 Pradhan et al., 2011; Banerjee et al., 2008; Pandey et al., 2009; 4 Gazdar,2000; 5 Implementation completion report: Sindh

Education Sector Reform Program (SERP)

Page 3: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

What is needed?

Carefully designed mechanisms for communities to operationalize school-based management policies:

◦ Facilitated meetings at the village level

◦ Participatory development of tools to enable members of school committees to perform expected roles

◦ Providing adequate flexibility to school councils to respond to community-identified needs

◦ Convergence of Information, Communication & Technology (ICT) platforms with grassroots institutions

Growing evidence of successful interventions that empower communities to improve school outcomes:

◦ Village-level meetings and information campaigns in three states in India(6)

◦ Community developed monitoring tool in Uganda (7)

◦ Text messages to create parental awareness in Indonesia (8)

◦ Strengthened linkages supported by elections of school councils in Indonesia (9)

6 Pandey et al., 2009;

7 Barr et al., 2012;

8 Cerdan-Infantes & Filmer, 2015;

9 Pradhan et al., 2011

Page 4: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

What we assess? 1. How important are mechanisms designed to foster community awareness, dialogue and SMC elections in improving

school outcomes? - Why?

o Decentralization is expected to strengthen local participation by (10)

improving access to information

fostering deliberative decision making

instituting regular elections

2. To what extent do the impacts vary between externally administered face-to-face deliberation and an ICT platform that anonymizes exchange of information between key stakeholders? - Why?

o Anonymous community dialogue platform:

reduces social costs of participation by side-stepping feudal power structures

requires mobile-phone literacy for active participation

serves as a preference-aggregation tool

o Externally administered face-to-face deliberations:

have higher social/time costs to participation

do not require literacy/education to participate

provide equal access to men and women to engage in community dialogue

3. Do these mechanisms work better when new Executive Councils are instituted and provided participatory training to develop a school improvement plan? - Why?

(1) and (2) might not work when SMCs are not appropriately constituted.

(1) and (2) might take time as attitudes and behaviors of existing members are hard to change

10 Mansuri and Rao, 2013

Page 5: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Sindh: Second largest province of Pakistan

Sindh has a population of 42.4 million - oughl one ua te of Pakistan’s total population ◦ The net enrolment rate at the primary level for ages 6-10 is 61% in all of Sindh.

◦ 73% in urban areas (compared to 77% in all of Urban Pakistan)

◦ 52% in rural areas in Sindh (compared to 63% in all of Rural Pakistan)

EMIS: Annual School ◦ There are 46,071 government schools in Sindh.

◦ 41,721 are primary schools

◦ 2,326 are middle/elementary schools; 1,729 are secondary schools; 295 are higher secondary schools.

◦ Nearly 15% of rural schools are closed for six months or have reported zero enrollment and/or teachers.

◦ 57% of functional Government primary schools are one-teacher schools.

◦ Quarter of functional schools lack access to at least one of the four basic infrastructure facilities: ◦ Boundary wall, toilet, drinking water supply, and/electricity.

Household and School Census 2011-12 (Study districts only) ◦ Only 43% of school age children (ages 5-16) are enrolled in school with the lowest proportion of 28.7% in Mirpurkhas.

◦ 1,727 schools were mapped in 300 villages. ◦ 1,644 are public primary schools, 19 are middle schools, 21 are secondary schools.

◦ Roughly 50% of the schools were found functional on an unannounced visit at the village census.

◦ Majority of schools are one-teacher schools.

◦ In sharp contrast to Punjab, the private sector is virtually absent in rural Sindh ◦ Only 14 private schools captured in the village-level census.

Page 6: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level
Page 7: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

What we did? The interventions were designed to elicit meaningful and sustained participation by communities to manage local schools.

Common Elements across all Treatments: ◦ Village-wide community mobilization (door-to-door campaign, posters, jingles, mosque announcements)

◦ Scripted introduction by two facilitators at the start of each meeting to explain purpose of the meeting.

◦ Participants fill attendance sheet with a mobile number accessible in their household.

◦ Short student performance to encourage participation

◦ Audio-clip 1: 10 minute dramatized story to highlight the importance of education and introduction to School committee with key messages reinforced by the facilitator through Flipcharts. ◦ See intervention materials here

T1: Village-level Meeting Only [INFO-MEET]

◦ Audio Clip 2: 20 minute discussion on the rights, roles and responsibilities of SMC members highlighting specific actions they could take to improve education outcomes in community schools. Information was also provided on ways they could contact SMC representatives.

◦ Key messages reinforced by a Flipchart.

◦ Moderated and inclusive discussion around Audio 1 and 2 (external facilitators only)

◦ Introduction to committee members present at the meeting

◦ Take-home brochure provided contact numbers and key messages in visual template.

Page 8: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

T2: Village-Level Meeting [INFO-SMS] ◦ Audio Clip 3: Introduces the Concept and Purpose of Community Dialogue Platform (CDP).

◦ Hands-on demonstration of CDP: Key features of the platform were highlighted: ◦ Summary messages to capture majority of the opinions

◦ Feed a k as ui kl su a ized hile p ese i g se de s’ i fo atio a o it

◦ Opt-Out Design ◦ Mobile phones captured when attendance sheets were filled.

Design features to improve participation:

◦ Nominal credit towards air-time (start and mid-way)

◦ Mobile-literate community volunteers

Post meeting Engagement:

◦ Informational messages

◦ Weekly summary messages

◦ Interactive Voice Response calls for real-time feedback

T3: INFO-MEET-SUPPORT ◦ All activities in T1, plus:

◦ Taluka Education officer (TEO) participated in the meeting to hold elections and transfer accountability to the newly elected member

◦ Hands-on training was provided to freshly elected members in three-structured meetings conducted over a three-week period.

T4: INFO-SMS-SUPPORT ◦ All activities in T2, plus T3

◦ Only difference between T3 and T4 is that TEO was NOT present when discussion took place over the Community Dialogue Platform.

What we did?

Page 9: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Was treatment received?

Page 10: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

*The o d loud e ges so e of the atego ies o igi all used fo so ti g. SMC i ludes SMC , SMC Fu d , E e uti e Co ittee , Pa e t , Mo e , S.I.P. a d SMC Offi e . Othe s i ludes Stipe d , Budget a d Edu atio Depa t e t .

CDP Messages: Preference Aggregated (dialogue period)*

Legend – Weights per Item Mentioned in CDP

Items Weights

Books & Stationary 3%

Flood 1%

Infrastructure & Facilities 39%

Learning 8%

SMC 10%

Student 12%

Teacher 23%

Other 4%

Page 11: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Planned Expenditure Estimates (School Improvement Plan)*

*"Buildings" includes floors, walls, ceilings, windows, doors and paint. "Boundary" includes boundary walls as well as main gates. "Water" includes water pipes, taps, motor pumps, hand pumps and water coolers. "Electric" includes lights, fans and electrical wiring. "Classroom" involves paint, wall, ceiling and floor, where an SIP specifically states that these changes are made within a classroom. "Furniture" includes chairs, stools, benches, tables and cupboards. "Miscellaneous" includes mostly labor wages.

Legend – Weights per Expenditure in SIP

Expenditure Weights

Blackboard 1%

Boundary Wall 3%

Building 24%

Classroom 4%

Cleanliness 6%

Electric 12%

Furniture 24%

Textbooks 4%

Toilets 6%

Water 9%

Miscellaneous 8%

Page 12: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

2011

Census-

level

village

mapping

April

2012

January

2013

Baseline Survey

of Households

and Schools

March

2013

January

2015

Endline

School

Surveys

March

2015

Qualitative

Data

Collection/

Focus

Group

Discussions

Village Level

Meeting

September

2015

Timeline

June

2013

Post-

meeting

Activities

Page 13: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Cluster-Randomized Trial

CONTROL

[57 V,

97 schools]

INFO-MEET-SUPPORT

[57 V,

92 Schools]

INFO-MEET

[59 V,

103 Schools]

INFO-SMS-Support

[57 V,

104 Schools]

INFO-SMS

[57 V,

91 Schools]

Page 14: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level
Page 15: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Data Census Survey

◦ Community level questionnaire to inform the design of interventions (300 focus groups)

◦ School mapping tool to obtain basic information on all schools as identified by the community.

◦ Household listing tool to obtain basic household information for all households in the village.

Baseline and Endline Data Collection ◦ Observational data collected from a surprise visit:

◦ School functionality; student/teacher attendance

◦ Announced visits

◦ School infrastructure profile; Student/Teacher enrollments and profiles from records; Income/Expenditure records

◦ Student tests (20 students grades 3-5) for all schools at endline

◦ Student tests conducted in sub-sample of schools at baseline

◦ Competencies tested: English, Math

Page 16: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Final Sample

Page 17: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Balance – Outcome Variables

Page 18: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Balance – School & Teacher Characteristics

Page 19: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Balance – School & Teacher Characteristics

Page 20: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Randomization Worked

◦No significant differences in key school, teacher and student characteristics except for a slight difference in whether the school has a boundary wall in SMS-Meet compared to the controls.

◦ In the analysis we control for school characteristics that we lack balance on as well as control for school size.

◦We also control for whether the school was impacted by floods in 2010 and heavy rains in 2011.

Page 21: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Summary Statistics

Note: Standard Deviations are given in parentheses. For Panel A,

school functionality variable includes all schools covered during the

unannounced round at the baseline and endline. All other descriptives

cover schools that were open at the announced round of visit at the

baseline and endline and that completed all modules of the survey.

Number of observations in Panel A fluctuates for teacher test scores

because it is calculated at the announced visit at the endline and

subject to attempting at least one question in the respective subject

section. Panel B provides student level descriptives at the endline for

students in grade 3, 4 and 5 that attempted at least one question in

the respective subject. See Annex II for more detailed summary

statistics.

Page 22: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Regression Specification ANCOVA (Analysis of Covariance) ◦ For school i in village v:

◦ Where 1 denotes data collected at endline and 0 denotes data collected at baseline:

𝑖𝑣 = 𝛽𝑇𝑣 + 𝑓 𝑖𝑣 + 𝑢𝑣 + 𝜀𝑖𝑣

◦ 𝑖𝑣𝑖 = School Outcome Indicator at endline

◦ 𝑇𝑣 = Treatment indicator

◦ 𝑖𝑣 = Vector of control variables at baseline

◦ 𝑢𝑣 = District fixed effects

◦ Standard errors at clustered at the village level

◦ ANCOVA is the preferred specification(9)

(8) Difference-in-Difference regressions are available upon request. The magnitude of effect sizes remain largely similar to ANCOVA results.

(9) [McKenzie, 2012]

Page 23: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Impact of Treatment on Number of Teachers?

Page 24: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Average Treatment Effects on School Functionality and Teachers

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors

clustered at village level. Unannounced visit at the baseline represents the census round of survey. Sample for school is open on unannounced visit covers

479 schools (all schools covered in baseline and endline). Sample for teacher indicators covers 387 schools that were open on the day of visit (announced

round) and that completed all modules of the survey at both baseline and endline. Baseline control for school is open includes dummy for flood and for

teacher indicators includes school enrollment (Grade 1 to 5). Results on school functionality are also validated through multinomial logit (see Annex III) .

Elaborated results for teachers can be found in Annex III .

Page 25: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

*Place Improvement to Infrastructure Index Here

Infrastructure Treatment Effects

Page 26: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

The most recurrent expenses include

◦ Buying or repairing of furniture

◦ Building new rooms

◦ Repair of Boundary wall

◦ Installation of water motor/tank or hand pump

In areas susceptible to heavy rainfalls or floods, school's expenses also include leveling the school ground

Infrastructure Expenditure – Qualitative Results

Page 27: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Average Treatment Effects on Improvements to Infrastructure

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with

standard errors clustered at village level. Sample covers 387 schools that were open on the day of visit (announced round) and that completed all

modules of the survey at both baseline and endline. Baseline controls include flood, teachers registered and school size. See Annex III for

difference and difference specifications and controls.

Page 28: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Impact of Treatment on Early Grade Enrollment?

Boys Enrollment: Katchi to Grade 1 Boys Enrollment: Katchi to Grade 5

Page 29: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Average Treatment Effects on Student Enrollment and Test Scores Results

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at

village level. Sample covers 387 schools that were open on the day of visit (announced round) and that completed all modules of the survey at both baseline and

endline. Baseline control for early grade indicators include infrastructure (measured as an index 0-4 to reflect boundary wall condition, electricity (in use), functional

toilets and drinking water (equal weights)) and for test scores include student age, gender, grade and teacher test score averaged at school level. See Annex III for

expanded enrollment results. For the test scores sample, only tracked students are covered. Students in grade 3 to 5 that attempted at least one question in the

respective subject at both baseline and endline are considered. Scores are normalized against the control group. 24 questions for English and 23 questions for

mathematics were considered for this analysis in line with baseline test. See Annex III for expanded results for student test scores.

Page 30: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Confirmation of Stable Unit Treatment Value (SUTVA) assumption

◦ Dropped all schools located less than one kilometer from the control schools.

◦ Results remain consistent with results from full sample (See Annex IV)

◦ Rules out contamination of controls from treatment.

Sensitivity check for factors driving higher rates of teacher absenteeism

◦ Limit the sample to only one-teacher schools at endline

◦ Find no significant differences in proportion of teachers absent in limited sample (See Annex III)

◦ Suggestive evidence of teachers slacking off when there is more than one-teacher in the school(11)

Sensitivity Analyses

11 [Duflo et. al, 2015]

Page 31: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

40 School level Case Studies conducted through comprehensive Focus Group Discussions (FGDs) ◦ Schools selected using Purposive Sampling Strategy from treatment bins

◦ Each FGD was held with the respective SMC Executive Body members and teachers of the school

◦ See Annex V for results.

Findings

Increase in teachers registered: ◦ In 72% of the Case Study schools that experienced increases in teachers registered, this change was attributed to

efforts by SMC members, Villagers and the Head Teacher

Infrastructural Improvements: Schools experience improvement in infrastructure through two channels: a) SMC funds and b) externally raised funds

◦ Sources of external funding include Village s’ contributions, NGO donations and MPA/MNA funds

◦ 50% of Case Study schools that experienced an infrastructure improvement funded it through the SMC Fund only, further 40% funded it through a combination of SMC & external funds and 10% funded it with external funds only

Student Learning Outcomes: Number of teachers registered, Teacher attendance and Commitment of teachers to the school are the most important factors in determining student learning outcomes

◦ Teacher attendance is highly dependent upon tea he ’s residency status

◦ 89% of the Case Study schools discussed teacher performance in regards to student learning outcomes

Election Vs. Non Election Villages ◦ Initially, communities resented newly elected Executive Body (EB) members.

◦ Newly elected EB members in the absence of continuous nudges from the community (CDP) lost interest after the elections.

What are the mechanisms?

Page 32: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Project Cost Distribution (Cost Effectiveness)

Design

25%

Operational

75%

TOTAL COSTS

Info-Meet SMS-Meet

Capacity Building &

Election Support

Design (USD) $27,083 $26,737 $24,891

Operational (USD) $101,976 $111,632 $24,294

Total (USD) $129,059 $138,369 $49,185

Info-Meet

43%

SMS-Meet

47%

Capacity

Building &

Election

Support

10%

OPERATIONAL COSTS

Info-Meet

34%

SMS-Meet

34%

Capacity

Building &

Election

Support

32%

DESIGN COSTS

Page 33: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

◦ Decentralization policy with transfer of resources and authority at local level may not yield expected results

◦ School accounts exists

◦ SMC members exist

◦ Money in bank accounts exists

◦ What is la king a e the institutions to ena le stakeholde s to pla thei e pe ted ole. ◦ Participatory design and local ownership of institutions can change outcomes of policies that otherwise seem not to work.

◦ Information Communication Technologies (ICTs) customized to local contexts

◦ Strengthening and deepening of grass root institutions through convergence?

◦ Limited understanding of the role trust, identity and power politics:

◦ Institutional memory of distrust?

◦ Resentment/resistance towards interference in local power dynamics?

◦ Constraints in making technology inclusive?

Discussion

Page 34: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Discussion Cont. What has this field experiment changed for the Government of Sindh?

◦ Re-thinking the role communities in service provision: ◦ Non-responsive (bad) communities Recognizing communities can play a role if they are properly engaged.

◦ No results of School Grants program were equated with evidence of failure of the reform: ◦ Re-orientation of School grants program to meet pre-conditions to quality for grants

◦ Emphasis on Elections and what the communities do with money instead of just giving them resources.

◦ Introduction of innovative ways to get feedback from stakeholders: ◦ Education Secretary started an SMS based initiative to link students, teachers and parents (ILMI)

Directions for future Research

◦ Unpack role of power holders in the village and pathways to impact

◦ Interventions designed to change the mix of preferences at community level ◦ Infrastructure learning

◦ How communities can monitor what happens inside the school? ◦ Teacher attendance, quality of teaching, student learning.

Page 35: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

References

Barr, A., Mugisha, F., Serneels, P. and Zeitlin, A. "Information and collective action in community-based monitoring of schools: Field and lab experimental evidence from Uganda." Unpublished paper (2012).

Banerjee, A., Banerji, R., Duflo, E., Glennerster, R. and Khemani, S. Pitfalls of pa ti ipato p og a s: E ide e f o a a do ized e aluatio i I dia. National Bureau of Economic Research. No. w14311. (2008)

Bruns, B., Filmer, D., and Patrinos, H. Making schools work: New evidence on accountability reforms. World Bank Publications. (2011).

Cerdan-Infantes, P., and Filmer, D.. "Information, knowledge and behavior: evaluating alternative methods of delivering school information to parents." World Bank Policy Research Working Paper 7233 (2015).

Channa, A. and Faguet, J. De e t alizatio of health a d edu atio i de elopi g ou t ies: a ualit -adjusted e ie of the e pi i al lite atu e . Economic organisation and public policy discussion papers, EOPP 38. STICERD, London, UK. (2012).

Duflo, E., Dupas, P., a d K e e , M. S hool go e a e, tea he i e ti es, a d pupil-teacher ratios: Experimental evidence from Kenyan primary schools. Journal of Public Economics. 123, 92-110. (2015)

Gazdar, H. Pu li Failu e, Politi al Co st ai ts a d Politi al Resou es -- Basi Edu atio i Pakista . Working Paper. Asia Research Centre, London School of Economics. May 2000.

Mansuri, G. a d Rao, V. Lo alizi g De elop e t: Does Pa ti ipatio Wo k? The Wo ld Ba k.

M Ke zie, D. Be o d Baseli e a d Follo -up: The Case fo Mo e T i E pe i e ts. The Wo ld Ba k.

Pandey, P., Goyal, S. and Sundararaman, V. Co u it pa ti ipatio i pu li s hools: i pa t of i fo atio a paig s i th ee I dia states. Edu atio Economics 17, no. 3. 355-375. (2009)

Pradhan, M, Suryadarma, D., Beatty, A., Wong, M., Gaduh, A., Alisjahbana, A. and Artha, R. "Improving educational quality through enhancing community participation: results from a randomized field experiment in Indonesia." American Economic Journal: Applied Economics 6.2: 105-126. (2014)

Pritchet, L., Woolcock, M., a d A d e s, M. Capa ilit T aps? The Me ha is s of Pe siste t I ple e tatio Failu e . Ce te fo Glo al De elop e t. (2010)

The Wo ld Ba k. P oje t App aisal Do u e t fo the Se o d Si dh Edu atio Refo P oje t [SERP II]. Washi gto , DC.

Page 36: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

ANNEX

Page 37: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Annex I: Intervention Materials

Return to the main presentation.

Page 38: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Annex 2: Summary Statistics

Page 39: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Summary Statistics – Village Characteristics

Standard Deviations are given in parentheses. Number of observations in Panel A fluctuates for teacher test

scores because it is calculated at the announced visit at the endline and subject to attempting at least one

question in the respective subject section. Return to main presentation.

Page 40: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Note: Standard Deviations are given in parentheses. For Panel B,

descriptives are given for schools that were open in the announced round

of visit at the baseline and endline and that completed all modules of the

survey. Number of observations in Panel A fluctuates for teacher test scores

because it is calculated at the announced visit at the endline and subject to

attempting at least one question in the respective subject section. Panel C

provides student level descriptives at the endline for students in grade 3, 4

and 5 that attempted at least one question in the respective subject.

Return to main presentation.

Summary Statistics – School & Student Characteristics

Page 41: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Annex 3: Results

Page 42: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Impact on School Functionality

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at village level. Unannounced

visit at the baseline represents the census survey. Sample includes all schools covered by the enumerator during the unannounced visit at the baseline and endline. Control variable includes

dummy for flood. Return to the main presentation.

Page 43: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Impact on Teachers

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at village level. Sample covers all schools that were

open on the day of visit (announced round) and that completed all modules of the survey at both baseline and endline. Control variable includes school enrollment (Grades 1 to 5). Return to main results table.

Return to sensitivity analysis slide.

Page 44: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Impact on Infrastructure

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at

village level. Sample includes all schools that were open on the day of visit (announced visit) and that completed all modules of the survey. Control variables include

flood, total teachers and school size. Return to the main presentation.

Page 45: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Impact on Early Grade Enrollment

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at

village level. Sample covers all schools that were open on the day of visit (announced visit) and that completed all modules of the survey. Control variable includes

Infrastructure (an index on a scale of 0-4 to capture boundary wall condition, electricity (in use), functional toilets and drinking water (equal weights)). Return to the

main presentation.

Page 46: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Impact on Student Test Scores

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at village level. Sample includes all

387 schools that were open on the day of visit, that completed all modules of the survey and had the test conducted. Controls include student age, grade, and teacher test scores averaged at the

school level. Student test scores in each subject are conditional on having attempted at least one question in the respective subject causing some fluctuation in observations across English and

Mathematics. Scores are normalized against the control group and have been averaged at the school level. Sample includes students of grades 3, 4, and 5 only. Some schools did not have any student

from these grades present on the day of survey, resulting in a different sample size. 24 questions for English and 23 questions for Mathematics. Return to main presentation.

Page 47: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Annex 4: SUTVA Analysis

Page 48: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Functionality and Teacher Performance – SUTVA Analysis

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at village level. Unannounced visit at the

baseline represents the census survey. Sample for functionality indicator includes all schools covered at the unannounced visit at the baseline and endline. Sample for teacher indicators includes all schools that

were open on the day of visit (announced round) and that completed all modules of the survey at both baseline and endline. Control variables for functionality include dummy for flood and for teacher indicators

include school enrollment (Grades 1 to 5). Treatment and control schools within 1 km of each other were removed from this analysis. Return to the main presentation.

Page 49: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Infrastructure – SUTVA Analysis

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at village level. Sample includes all schools

that were open on the day of visit (announced visit) and that completed all modules of the survey. Control variables include flood, total teachers and school size. Treatment and control schools within 1 km of each

other were removed from the analysis. Return to the main presentation.

Page 50: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Early Grade Enrollment and Student Test Scores – SUTVA Analysis

Note: Parameter estimates statistically different than zero at 99% (***), 95% (**), and 90% (*) confidence. District fixed effects added with standard errors clustered at village level. Sample includes all schools

that were open on the day of visit (announced visit) and that completed all modules of the survey. Control variable for enrollment includes Infrastructure (an index on a scale of 0-4 to capture boundary wall

condition, electricity (in use), functional toilets and drinking water (equal weights)). For student test scores, we limit our sample to students in grades 3 to 5 that attempted at least one question in the

respective subject at both baseline and endline. Control variables for test scores include student age, gender, grade and teacher test score. Scores are normalized against the control group and have been

averaged at school level. 24 questions for English and 23 questions for mathematics were considered for this analysis in line with baseline test. Return to the main presentation.

Page 51: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Annex 5: Qualitative Results

Page 52: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Return to the main presentation.

Page 53: Salman Asim Dhushyanth Raju Nazmul Chaudhury Amina Riazpubdocs.worldbank.org/en/325371508179084973/07-TZ-IE-CaseStudy-Sindh.pdf · ¸Standard errors at clustered at the village level

Return to the main presentation.