Ema esrc surrey

Evaluation of Education Maintenance Allowance Pilots

Sue Middleton - CRSP

Carl Emmerson - IFS

The Policy Problem (1)

• Very large increases in participation in the later 80s and early 90s had levelled off by the mid-90s

020406080

per c

ent

16 year olds

17 year olds

18 year olds

The Policy Problem: Post-16 Participation and Socio-Economic Group

0

10

20

30

40

50

60

70

80

90

Parents' Occupation

per

cent

16 year olds in full-timeeducation 199816 year olds NEET 1998

Source: Youth Cohort Study, Statistical Bulletin 02/2000

The Policy Problem: Post-16 Participation and Gender

0

10

20

30

40

50

60

70

80

1989 1992 1994 1997 2000

per

cent Male 16 years

Female 16 years

Source: DfES Statistics 2001 http://www.dfes.gov.uk/statistics/DB/SFR/index.html

Reasons for Non-participation

• Little evidence about the reasons• Money is one factor among many

“...not surprisingly, money looms large in the accounts given by disaffected young people of their lives. They report one of the key barriers to further and higher education to be (lack of) money”. (Newburn, 1999)

The Policy Response

• Multiple policy responses• Education Maintenance Allowance

- to encourage participation, retention and achievement in post-16 education

- focusing on young people from low income families

Education Maintenance Allowance

• Household income < £13k

- weekly allowance up to £40 per week

• Household income £13k - £30k

- weekly allowance tapers to minimum £5 per week

• Retention and achievement bonuses

- available to ALL awarded EMA

• Receipt subject to compliance with Learning Agreement

• 4 Variants being piloted

EMA Variants Maximum Weekly Amount

Retention and Achievement Bonuses

Paid to:

Variant 1 £30 £50 Young person

Variant 2 £40 £50 Young person

Variant 3 £30 £50 Parent

Variant 4 £30 £80 (retention)

£140 (achievement)

Young Person

EMA Evaluations

• EMA Main

– 10 LEAs

• Leeds and London

– 5 LEAs

• EMA Transport

– 5 LEAs

• EMA Extensions

– 4 LEAs

Evaluation in the LEAs

ROUND TABLE DISCUSSIONS

• EMA Implementation

groups

DATA COLLECTION

• Labour market• Education profile• Take up of EMA• Socio-demographics

INTERVIEWS

• LEA administrators• Careers Service

representative• TEC representative• Employer organisations

AREA VISITS

Qualitative Interviews with Young People/Parents

• Year 1 – Participants/Non-participants

(young people and parents)

• Year 2 – Longitudinal interviews with young people and early leavers

WAVE 1 WAVE 2 WAVE 3 WAVE 4

Face to face

October 1999

Telephone

October 2000

Telephone

October 2001

Telephone

October 2002COHORT 1

COHORT 2

Quantitative Design

Face to face

October 2000

Telephone

October 2001

Telephone

October 2002

Telephone

October 2003

The data• Questionnaires have detailed information

on:- all components of family income

- household composition

- GCSE results

- mother’s and father’s education, occupation and work history

- early childhood circumstances

- current activities of young people

Matching approach

• Involves taking all EMA eligible young people in the pilot areas and matching them with a weighted sum of young people who look like them in control areas

• Difference in full-time education outcomes in pilot and control areas in this matched sample is the estimate of EMA effect

• Crucial assumption is that everything is observed that determines education participation

How is this done?• Don’t match on all X’s, but can instead match on

the propensity score (Rosenbaum and Rubin, 1983)

• Propensity score is simply the predicted probability of being in a pilot area given all the observables in the data

• Use kernel-based matching (Heckman, Ichimura & Todd, 1998)

• The matching is undertaken for each sub-group of interest

• Family background- household composition, housing status, ethnicity, early

childhood characteristics, older siblings’ education and parents’ age, education, work status and occupation

• Family income- current family income, whether on means-tested

benefits• Ability (GCSE results)• School variables• Indicators of ward level deprivation

Variables are matched on:

Analytic Strategy for EMA

Propensity Score Matching:

Measures the Impact of EMA

BUT

Requires Large Sample Sizes

Weighting Issues

Limited Disaggregation

Descriptive Analysis:

Provides Contextual Detail

Allows Disaggregation

Overcomes Weighting Issues

BUT

Cannot Measure Impact

The Impact of EMA on Participation

• EMA has increased participation by 5.9 percentage points

• EMA had a larger effect on young men than young women

0

1

2

3

4

5

6

7

8

Overall Urban Men Urban Women

Base: Eligible young people in Cohort 1 & 2

The Impact of EMA on Participation

• Draw is from both those who would have been in work and those who would have been NEET


-4

-2

0

2

4

6

8

per

cen

t

FTE

Work/Training

NEET

EMA and Retention at Year 13

• Slightly larger impact on participation in year 13

• Suggests that retention has not fallen


0

1

2

3

4

5

6

7

8

per

cen

t

Y12 effect Y13 effect

Participation and Retention by Variant

0

2

4

6

8

10

12

Participation Retention

per

cen

t

Variant 1 Variant 2 Variant 3 Variant 4

Conclusions (1)

• EMA effect around 6 percentage points

• Plays an important role in reducing gender differences in post-16 participation

• Important to control for local area effects- matching on ward level data important

Conclusions (2)

• More effective paying EMA to young person rather than parent

• Bigger retention bonuses have significantly larger effect on retention than other variants

• Increase in participation drawn from both work and other groups

What We Learn, And When: From PSM

Surveys Information on Best Impact Measure

Wave 1 Participation 2002 Report 2

Wave 2 Participation and Retention in Year 13

2003 Report 3

Wave 3 Retention in Year 14

Achievement

Higher Education Entry

2004 Report 4

What We Learn And When: From Descriptives (1)

Surveys Information on

Wave 1 Year 11 decision making

Awareness, Applications and Receipt

Year 12 Courses

Wave 2 Retention bonuses

Year 12 achievement

Destinations

What We Learn And When: From Descriptives (2)


Wave 3 (& 4) Achievement Bonuses

Year 13 Achievement

Higher Education:

- Courses and Institutions

- Financial Support

Destinations over time

What We Could Learn


All surveys Destinations over time including labour market and higher education entry (for sub-groups)

Part-time work, hours and earnings

Expenditure patterns and responsibility