The Ishraq Program in Egypt

Ghada BarsoumPopulation Council

PEP Meeting June 2008

Ishraq: Basic Facts• Targets out-of-school adolescents (12-

15) • In rural Upper Egypt: Lowest human

development indicators: 26% of girls receive no schooling (ELMPS06); unpaid family enterprise workers; early marriage and childbearing; FGM

• Intervention lasts for 20 months• Girls meet 4 times weekly for 3 hours

Ishraq: Basic Facts– Four Int’l NGOs: Save the Children;

CEDPA (the Center for Development and Population Activities); Caritas; Population Council

– Government partner with an increasing role: National Council of Youth (NCY) to allocate specific hours in youth centers to be used as “safe spaces” for girls

– Local NGOs implementing, adapting the model and mobilizing communities on the village level

Ishraq: Basic FactsThree Components:

1) Literacy (Learn to be Free, Caritas): Girls sit for the literacy exam and are given the opportunity to be mainstreamed into the formal educational system (grade 7) – Program provides help in issuing birth certificates (required for school enrollment)

2) Life-Skills (RH, Basic: livelihood, nutrition, girls rights .. (CEDPA)

3) Sports and Recreation (PC)• Monitoring and evaluation (PC)

Intervention so far ..• Piloted in 2001 in 4 villages in Minia

(N=275)• Expanded in 2004 to 6 villages in Minia

(2nd rounds in 2 villages from the pilot stage) (N=336)

• Expanded in 2006 to 5 villages in Beni Suef (N=289)

• A new phase is beginning September 2008 with a new scale-up and –out plan

A New Phase of Vertical and Horizontal Scale up

• Transfer of Program ownership to National Council of Youth (NCY) – the longstanding government partner

• 50 villages (2500 girls) – new components (graduation funds, financial literacy, livelihood)

• The role of the resource team envisioned : transfer of the tested innovative design; training (NGOs and NCY); resource mobilization; research and evaluation; advocacy dissemination;

• THEN: Within 5 years, NCY would incorporate Ishraq in its budget and independently operate it.

Hence, the importance of rigorous analysis and

thorough documentation of this phase

Analysis & Documentation of Earlier Phase (mainly descriptive)

• 92% of those who sat for the literacy exam passed

• Over half achieved an “Excellent” score• 66% (re-)entered the formal education system

– as compared to a national average of 6% for those who pass literacy exam nationally

Monitoring and follow-up• 98 graduates enrolled in Sept’06 in grade 7• 22 graduates are currently in technical secondary

education

Current proposal

LITERATURE REVIEW

• To estimate the impact of a program we need to know the difference between:(1) the outcomes for participants if they

participated in the program and (2) the outcomes had they not participated

• The latter is not observed Primary problem in identifying treatment

effects is the estimation of that counterfactual

Selection Problem• A common proxy for the counterfactual: average

outcomes of untreated units (e.g., a comparison or control group)

• Works out in random social experiments• Problem: violation of non-randomness of

participation frequently arises when the (to-be) treated and the untreated differ in (often unobservable) characteristics selection issue

• Selection bias problem results in mixing-up the treatment effect with pre-program differences.

Non-experimental (econometric) methods

• Use non-experimental (econometric) methods where a comparison group with “comparable” characteristics is constructed.

• Whole body of literature has grown and incorporated econometric advancements such as non-parametric techniques. Many attempts to compare the performance of different econometric estimates (e.g., Lalonde 1986 and the response it triggered).

• Different methods rely on different assumptions : they would produce the same results only if there is no selection issue.

Techniques Used

• Selection on observables:– Linear regression– Matching methods

• Selection on unobservables– Instrumental variables– Longitudinal data analysis

1. Selection on Observables:• Assumption: participation is determined by

characteristics that are fully observable to the evaluator (X). by statistically conditioning on these characteristics, the selection issue is resolved.

• Linked to linear regression• Matching methods

– Outcomes of an individual are matched with the outcomes of an individual in the comparison group that is most similar using non-parametric techniques.

– Not impose linearity

2. Selection on Unobservables• Instrumental variables

– Affects participation but not outcomes– Ex: natural experiments, RDD, randomized

experiment design.• Longitudinal data analysis

– Difference-in-differences (fixed-effects): difference out the unobserved differences between the treated and the untreated by comparing the pre-post change of treated units with that of the untreated units.

• Assumption: unobserved differences are time-invariant

3. Combining longitudinal methods with other methods• For example, control variables can be

added to the difference-in-differences estimator.

• Difference-in-differences can be combined with matching methods by differencing the data first and then applying the matching method

Dropout ProblemTwo ways are employed in the literature 1.Re-interpret as the impact of the offer of the

treatment rather than its receipt.2.Adopt an estimate adjustment

• Introduced by Bloom (1984). Requires availability of follow-up data for dropouts, full-participants and control group members

• Bloom (1984) focused on “no-show” dropouts and a “partial-show” case in the context of multiple-treatment design.

• Heckman, Smith and Taber (1998) focus on alternative treatment impact parameters in the case of partial treatment. They use an instrumental variable approach.

• Heckman, Hohmann, Smith and Khoo (2000) analyze dropout cases where participants dropout to pursue better alternatives to the program (they also looked at the substitution problem where control group members seek other programs). They provide an estimate adjustment to deal with dropout and substitution.

DATA COLLECTION PLAN• Start with about 6 clusters of pre-selected villages

that fulfill certain basic criteria. • Each cluster will initially consist of about 12

villages• 5 will be selected as intervention villages and 2 as

controls• Target group: 12-15 out-of-school girls.• Mapping & listing

– Intervention villages: all households in the intervention villages to identify households containing out-school adolescent girls in the 12-15 age range.

– Control villages: depending on size, either the entire village or randomly selected segments.

DATA COLLECTION PLAN- CONTD• Surveys:

– Baseline– Endline– 5 years after completion (?).

• To sum up: info on eligible out-of-school girls in:– intervention villages (participants and non-

participants)– control villages

• Info on their households and “gatekeepers”.

DATA COLLECTION PLAN- CONTD• Examples of information to be collected

– Household characteristics• Parental education, work• Owned assets

– Individual characteristics: • Age• Schooling, including grade attainment, and

repetition.• Participation in literacy programs.

– Variables we plan to use as instruments

Data Collection Quality

• Based on experience from the pilot phase and other surveys in Egypt, we expect:– High response rates– High follow-up rates.– Low attrition rates in control villages

• Double-check measure: compare population figures from our survey to corresponding age-group population figures from the 2006 Census.

Recruitment• Two classes of 25 girls in each intervention village • Participation is voluntary• First-come first-served basis. We foresee that

participation randomization will not be not feasible– Based on experience from the pilot stage, applicants

arrived at different points in time.– Different size of pool of applicants in different villages Randomizing the selection of intervention and control villages. Not randomizing participation within intervention villages Self-selection issue

Examined Outcomes

• Literacy– level of success in passing government-

sponsored literacy tests and rates at which girls are mainstreamed back into preparatory (middle) schools.

• Attitudes about marriage and childbearing:– Ideal age at marriage – Views about decision-making regarding

marriage partners and timing of marriage. – Ideal family size and fertility intentions.

Examined Outcomes- CONTD

• Knowledge about nutrition, hygiene, and reproductive health.

• Attitudes about harmful traditional practices (e.g., FGM)

• Social isolation, peer networks, and participation in group or community activities

• Gender norms index

Foreseen Challenges

1.Self-selection into participation

2.Non-randomness of drop-out 3.Effect on attitudes

Self-Selection Sources

1. Project recruitment process– Household visits by promoters (pilot

phase)• Geographical proximity• Friends and relatives

2. Girls that self-select into the program are more likely to have parents who are more motivated and committed to girls’ education

Non-random Dropout• Dropout more likely in the following cases

– Early marriage – Child work– Domestic chores– Lower commitment to female education

PROPOSED METHODOLOGY

• Adopt strategies corresponding to the different stages of the project implementation– Recruitment / Pre-program activities– Baseline data collection– Estimation strategies

I - Recruitment Activities• Improved Ishraq recruitment strategy

– Visual advertisements – More training for promoters– Ishraq village committees – Including a local NGO for implementation– Ishraq workshop/registration event

• Broader outreach• More information less dropout• Keep track of who registered and form.

– Pre-program trial period (?)

II- Randomization Strategies

• Randomized assignment from the pool of registered girls may not work

• Alternative randomization procedures we thought of– Randomly choose girls from baseline

data– “Randomized encouragement”

procedure

III – Data Collection

• Measures to ensure data quality• Collection of variables to serve as

instruments

IV – Estimation Strategies

• Preliminary steps– Compare the baseline characteristics of

control village and intervention village members

– Compare characteristics of participants, waiting list members and non-participants.

– Probit models for participation into Ishraq

Estimation Strategies - CONTD

• Plan to use several models: – intent-to-treat– instrumental variable – difference-in-differences– Other specification / models / Specification

tests are also discussed.

Intent-to-treat• outcomes of control village members are

compared to intervention village members whether they were treated or not

• the treatment explanatory variable = 0 if the girl resides in a control village = 1 if she resides in an intervention village

• Advantage: no selection issue.• Disadvantage: effect of the treatment will be

diluted as the model will include untreated girls when estimating the average treatment effect

Instrumental variable• Instrument that affects participation but not

directly affect outcomes. • Distance between home and youth center

(proxy for costs of participation). + Distance to the village center as an

explanatory variable in both the participation and outcome equations to capture access to other services and the effect of the centrality of the household location.

• Collect information on distance (or time) based on respondents’ answers and on GPS coordinates.

Difference- in-differences• Using a group dummy, period dummy (pre- or

post- program) and an interaction of the group and period dummies.

• Combine the differences- in differences with other methods in additional specifications: a “regression-adjusted” specification by adding observable characteristics.

• Considering combining difference-in-differences with the propensity score method

Other Models/Specifications• Estimate the difference between the outcomes of

treated girls and that of waiting list girls. – Good comparison group: equivalent in terms of

motivation and valuation of girls’ education• Heckman and Hotz (1989) pre-program specification

test – Rationale: prior to treatment, treatment dummies should

not affect outcomes for the control or the to-be treated. – Null hypothesis: treatment variable coefficient is equal

to zero when using baseline data. – Limitation: assume that if a model successfully adjusts

for pre-program differences, treated and untreated units will also do so in a post-program period

– Not widely used.• RDD on age (?)

Human subjects concerns• IRB: an oversight committee, whose

mandate is to protect the rights and safety of the subjects of the Council’s research.

• Confidentiality, minimal risk to participants and informed consent.

Timeline

• Data collection in August 2008• Intervention Sept 08 – April 10• Endline data collection (expedited to fit

schedule, using MCAPI): February 2010 • Draft report: June 2010