+

Measuring Investment in Human Capital Formation: An Experimental Analysis of Early Life Outcomes

Orla Doyle, Colm Harmon, James Heckman, Caitriona Logue, Seong MoonUCL, 25th-26th June 2012

+ Preparing for Life Programme

One of the first experimental early childhood intervention in Ireland

Funded by Irish Government (DCYA) & The Atlantic Philanthropies

Community-led initiative operated by Northside Partnership in Dublin ~ 6,400 inhabitants 33% dependent on social welfare 60% live in social housing (>3 times national average) 47% lone mothers (29% national average) 16% unemployed (3 times national average) 66% early school leavers (38% national average) 5% third level education (29% national average)

Source: Census (2006)- CSO

Belief within the community that children were lagging behind their peers

+ Design of PFL

Aim: Improve levels of school readiness by assisting parents in developing skills to prepare their children for school

Bottom-up approach: community initiative involving 28 community groups, service providers, & local representatives

Theoretical Framework: Grounded in several psychological theories of development including the theory of human attachment, socio-ecological theory of development and social-learning theory

Evaluation: Randomised Control Trial design

+Design of Preparing for Life

+ PFL EvaluationImpact Evaluation Data collection: Pre-intervention (baseline), 3 mnts (WASI), 6mths,

12mths, 18mths, 24mths, 3yrs, 3.5yrs, 4yrs

Informant: Mother is the primary informant, but also fathers, child, other independent data sources (birth records)

Implementation Evaluation Aim: Delve into the blackbox of programme effectiveness & evaluate fidelity

to the PFL model

Data collection:1. Implementation data on the Database Management System 2. Focus groups with participants3. Semi-structured interviews with mentors/IO

+ Recruitment

Eligibility Criteria: Cohort of pregnant women residing in PFL catchment area between Jan

2008-August 2010 (32 months) Includes preparious and non-preparious women

Recruitment: Maternity hospital at first booking visit (b/w 12-26 weeks) Within the local community

Population-based recruitment rate, based on all live births during the recruitment phase, was 52%

+Computerized Randomisation Procedure

Unconditional probability randomisation strategy

After informed consent obtained, mother ask to press the key to allocate her assignment condition

Evaluation & Implementation team automatically received an email with the assigned PFL ID number and treatment condition

Preserves the integrity of the procedure as no opportunity for recruiter to intentionally influence assignment

115 allocated to High treatment group 118 allocated to Low treatment group

+ Baseline Data Collection

Conducted post randomisation, pre intervention Baseline data wave completed in August, 2010

123 measures: Demographics & SES, health behaviour & pregnancy, parenting, social support, psychological assessments (maternal well-bring, personality traits, self-esteem, self efficacy, attachment, time preferences)

Interviews conducted PFL High treatment group: 104 PFL Low treatment group: 101

Used to determine the effectiveness of the randomisation procedure

+ Methodology

Permutation based hypothesis testing (Heckman et al. 2010) Classical hypothesis tests unreliable when the sample size is small and the

data are not normally distributed Permutation test are distribution free, thus suitable in small samples Based on the assumption of exchangeability between treatment

conditions under the null hypothesis

Stepdown procedure (Romano & Wolf, 2005) Ignoring the multiplicity of tests may lead to the rejection of “too many”

null hypotheses Test multiple hypotheses simultaneously by controlling overall error rates

for vectors of hypotheses using the family-wise error rate (FWER) as a criterion

Less conservative & more powerful than other methods as takes account of statistical dependencies between tests

+ Summary of Permutation Tests Examining Differences at Baseline

Proportion of Measures Not Significantly Different at Baseline

Topic PFL Low – PFL High

Family Socio-demographics 33/33 (0%)

Maternal Well-being 24/24 (0%)

Maternal Health & Pregnancy 35/35 (0%)

Parenting & Childcare 10/13 (74%)

Social Support 17/18 (94%)

Total NOT Statistically Different 119/123 (97%)

Randomisation worked!

+ Permutation Tests Comparing Baseline Differences in Selected Family Socio-Demographics

Low Treatment – High Treatment

InstrumentN

(nLOW/ nHIGH)

MLOW

(SD)MHIGH

(SD)p Effect Size

(d)

Mother’s Age 205(101/104)

25.30(5.99)

25.46(5.85)

ns .03

First-time Mothers 205(101/104)

0.50(0.50)

0.54(0.50)

ns .09

Mother Married 205(101/104)

0.18(0.38)

0.14(0.35)

ns .09

Mothers with Junior Certificate Qualification or Lower

205(101/104)

0.40(0.49)

0.34(0.47)

ns .12

Mothers with Primary Degree 205(101/104)

0.03(0.17)

0.03(0.17)

ns .01

Mothers Unemployed 205(101/104)

0.41(0.49)

0.43(0.50)

ns .05

Annual Income of Working Mothers (in Euros)

75(38/37)

19,602(8,093)

19,224(9,851)

ns .04

Fathers Unemployed 198(97/101)

0.31(0.46)

0.43(0.50)

ns .24

Residing in Social Housing 204(101/103)

0.55(0.50)

0.55(0.50)

ns .00

In Possession of a Medical Card 205(101/104)

0.66(0.47)

0.60(0.49)

ns .14

+ Analysis of 6 Month Outcome Data

Test for treatment effects across 8 main domains including 160 measures Child development Child health Parenting Home environment Maternal health Social support Childcare & service use Household factors & SES

25 Step-down categories defined

258 interviews conducted PFL High treatment group: 84 PFL Low treatment group: 90

12

+ Attrition & Disengagement up to 6 months

Official dropout between baseline - 6 months: High treatment – 13% Low Treatment – 6%

Disengagement between baseline - 6 months : High treatment – 9% Low Treatment – 10%

Baseline Characteristics Associated with Attrition

VariablesHigh Treatment

Group Low Treatment

Group p p

Weeks in preg. at programme entry 0.651 0.563

Mother's age 0.505 0.063

Partnered 0.410 0.503

Married 0.541 0.699

Living with parent(s) 0.477 0.556

First time mother 0.629 0.073

Low education 0.121 0.131

Mother employed 0.010** 0.772

Saves regularly 0.337 0.545

Social housing 0.639 0.759

Cognitive Resources (WASI) at 3MO 0.570 0.097

Physical Health Condition 1.000 0.511

Mental Health Condition 0.760 0.197

Smoking during pregnancy 1.000 0.367

Drinking during pregnancy 0.439 0.308

Drug ever used 0.159 0.675

Vulnerable attachment (VASQ) 0.324 0.622

Positive parenting attitudes (AAPI) 0.565 0.757

Self efficacy (Pearlin) 0.263 0.389

Self esteem (Rosenberg) 0.096 0.717

Knowledge of infant development 0.731 0.003**

+

ASQ Scores & Difficult Temperament

N(nHIGH/ nLOW)

MHIGH

(SD)MLOW

(SD)Individual

Test p1

Step Down Testp2

Effect Sized

ASQ Gross Motor Score 173(83/90)

40.78(11.93)

38.50(12.99)

ns ns 0.18

ASQ Communication Score 173(83/90)

53.07(7.84)

51.78(8.49)

ns ns 0.16

*Difficult Temperament 173(83/90)

11.70(5.71)

12.21(5.50)

ns ns 0.09

ASQ Personal Social Score 172(82/90)

46.52(12.09)

45.94(13.57)

ns ns 0.05

*ASQ Social-Emotional 173(83/90)

14.76(10.68)

15.17(13.75)

ns ns 0.03

ASQ Fine Motor Score 173(83/90)

50.84(9.46)

51.39(10.17)

ns ns 0.06

ASQ Problem Solving 173(83/90)

51.87(9.39)

52.56(9.92)

ns ns 0.07

Notes: 1 one-tailed (right-sided) p value from an individual permutation test with 1000 replications. 2 one-tailed (right-sided) p value from a Step Down permutation test with 1000 replications. * indicates the variable was reverse coded for the testing procedure. ‘ns’ indicates the variable is not statistically significant.

RESULTS FOR HIGH & LOW TREATMENT GROUPS: CHILD DEVELOPMENT

+ Summary of Six Month Results

Proportion of Measures Significantly Different at Six Months

Domain PFL Low – PFL High Step Down categories

Child Development 0% (0/13) 0/2

Child Health 10% (3/30) 0/3

Parenting 23% (5/22) 1/5 (PSI)

Home Environment & Safety 36% (8/22) 1/2 (HOME)

Maternal Health 5% (1/20) 1/4 (Health)

Social Support 38% (5/13) 0/2

Childcare & Service Use 7% (1/14) 0/2

Household Factors & SES 0% (0/26) 0/5

Total Statistically Different 14% (23/160) 3/25 categories

+ Summary of 6 month results

Majority of the results for High V’s Low are in hypothesized direction

A few significant differences identified (14%)

In line with other home visiting programmes

+ Additional Analysis

Interaction & Subgroup analysis Gender, lone parents v partnered parents, first time v non first time

parents, high versus low IQ mothers, high v low family risk

Programme appears to benefit high IQ mothers, families with multiple children & families with multiple risks

Analysis of engagement (treatment intensity) On average, high treatment group received 14 home visits of ~1 hr in

duration= total treatment = 14 hours

Higher maternal IQ & vulnerable attachment style & joined programme earlier associated with higher engagement, while smoking during pregnancy associated with lower engagement

+Contamination in RCTs

Contamination occurs when the control groups either actively or passively receive the intervention intended for the treatment group (Cook & Campbell, 1979)

Potential for contamination is high in PFL Members of the treatment and control groups may be friends, neighbours,

colleagues, same family!

Community comparison group used to safeguard against contamination

Included a range of direct (‘blue-dye’) in each survey

+ Contamination in PFL

Heard the phrase MHIGH

(SD)MLOW

(SD)MLFP

(SD)Low – High

pHigh –

Comparisonp

Low – Comparison

p

Mutual Gaze0.59(0.49)

0.08(0.27)

0.04(0.19) <0.01 <0.01 ns

Circle of Security0.49(0.50)

0.12(0.33)

0.05(0.21) <0.01 <0.01 ns

If contamination DID NOT take place: • High & Low treatment group will differ in their responses • Low treatment group will not differ from comparison group• High treatment group will differ from the comparison group

Contamination Questions in 6 month PFL survey

Little evidence of contamination among treatment groups Evidence of absorption of PFL knowledge among high treatment group

+ Update on PFL

Oldest PFL child is almost 4 years and old & youngest is 13 months

18, 24, & 36 month surveys are currently in the field

12 month results available during Summer

Over 1,500 interviews conducted as part of impact evaluation

Conducted four school readiness surveys with junior infant cohorts in 2008, 2009, 2010, 2011

Conducted focus groups with low & high treatment groups & semi-structured interviews with all PFL mentors

Evaluation will continue until all children start school

Programme website: www.preparingforlife.com

Evaluation website: http://geary.ucd.ie/preparingforlife/