The mathematics skills of school children: How does England compare to the high performing

East Asian jurisdictions?

PISA maths – England versus East Asia

EnglandSwedenPolandAustriaFrance

Slovak RepublicNorway

SloveniaDenmark

IcelandEstonia

GermanyAustraliaBelgium

New ZealandMacao-China

NetherlandsCanada

JapanSwitzerland

LiechtensteinFinland

Chinese TaipeiKorea

Hong Kong-ChinaSingapore

Shanghai-China

0.00 0.50 1.00 1.50 2.00 2.50 3.00

Years of schooling ahead of England

By age 15, children in East Asian counties are approximately 1.5 years of schooling ahead of children in England

Huge policy interest in this….These regions and nations – from Alberta to Singapore, Finland to Hong Kong, Harlem to South Korea - should be our inspiration’

(Secretary of State for Education in England, Michael Gove)

‘we must learn from high-performing nations like Japan’(Shadow Secretary of State for Education in England, Stephen Twigg)

‘We have to see this as a wake-up call….. We can quibble, or we can face the brutal truth that we’re being out-educated’

(US Secretary of Education, Arne Duncan)

WHY!?

....we don’t really know

Possible explanations• Biology / genetics

• Cultural differences- Commitment and highly value education (‘Tiger mums’)

• ‘Failing’ secondary schools

• Teaching methods

• Design of the curriculum

• Many more!!

Possible explanations• Biology / genetics - Can’t do anything about

• Cultural differences – Hard to change in short-run- Commitment and highly value education (‘Tiger mums’)

• ‘Failing’ secondary schools

• Teaching methods Things we can change

• Design of the curriculum

• Many more!!

...but we do not know the relative importance of each of these factors

Policy – hard to know what we need to do to catch up!

Current research

Lots of research looking at the ‘high-performing’ systems(particularly East Asia)

• E.g. Review of the curriculumhttps://www.education.gov.uk/publications/standard/publicationDetail/Page1/DFE-RR178

• E.g. Teaching observations in Shanghi

Pretty much all qualitative

This paper: Think what quantitative research can add to the debate

But there is a MAJOR problem for both qualitative and quantitative research

East Asian countries have a lot of East Asian children in them!

….Big confounding factor

Implications of this confoundingQualitative research

– We observe differences in curriculum– We observe differences in teaching methods– BUT we don’t know if this has causal impact– Really the reason why East Asian countries doing better?

Quantitative research– Can’t easily ‘control away’ cross-country confounders– Therefore also difficult to establish causal impacts– PISA can tell us how far we are behind……– …..But it can’t tell us why!

My paper…..What else can be done to inform the debate????

Starting point– How do we judge school performance?– Widely recognised it should not be on % 5 A*- C grades– Should be judged on ‘value-added’ (how much children

improve)

International application– Should think same way about countries / school systems– Don’t judge UK secondary schools by PISA test scores– Judge them by value-added – How much do UK children improve in secondary school

(relative to East Asian children)

We try to start thinking about this…..

(1) How do average test scores change between age 10, 14 and 16 in England compared to East Asia?

(2) How does the distribution of test scores change between 10, 14 and 16 in England and East Asia?

(3) How does the socio-economic gap in children’s test scores change over age in England compared to East Asia?

Data

Ideal data→ Longitudinal→ Follow same kids over time→ Test scores designed to be comparable over time and

across countries→ E.g. A longitudinal PISA

This type of very rich data does not exist

Therefore have to think about an alternative

Data we do have….→ TIMSS 2003 4th grade (age 9/10)→ TIMSS 2007 8th grade (age 13/14)→ PISA 2009 (age 15 year 3 months – 16 years 2 months)

– Think of this as ‘repeated cross-sectional data’

– Investigate change over time at the population level under certain assumptions:

• Samples drawn from same population at each time point• The same skill is being measured at each time point

Argument:Do these assumptions hold for TIMSS / PISA data?Argue they provide an OK approximation….

An aside – how similar are PISA / TIMSS?Often argued that PISA / TIMSS measure ‘different skills’

• PISA = ‘Functional ability’• TIMSS = ‘Curriculum’ based skills

Qualitative research:Analysis of questions suggests there are differences..…but unable to say of this has major impactOther statistical differencesItem-response model usedTarget populations etc

But really how different are PISA and TIMSS??

PISA 2009 vs TIMSS 2011 (maths)KR SGTW

HK

JP

RU

IL FIUSGB(E)HU AUSILTITNZKZ SE

NO

ROAE TRMY

GE THTNCL

QA JO

ID

350

400

450

500

550

600

TIM

SS

201

1

350 400 450 500 550 600PISA 2009

Strong correlation between x-national results (r = 0.88)

Actually tell us pretty similar things about performance

PISA 2003 vs Key stage 3 test scores (England)

Correlation PISA maths with Key Stage 3 average scores = 0.83

Correlation PISA maths with Key Stage 3 math scores = 0.85

Previous graphs suggest that key assumptions of the method used may approximately hold

20

International z-scoresNevertheless ……..

• PISA and TIMSS raw test scores are not directly comparable – based on a different array of countries.

• Convert into international z-scores.

• Each country’s mean test score (for each wave of the survey) is adjusted by subtracting the mean score achieved amongst all children in the countries for that particular survey and dividing by the standard deviation

• Estimates refer to English pupils’ test performance relative to other countries

21

Countries included in this study• Include all countries that took part in TIMSS 4th grade (2003),

TIMSS 8th grade (2007) and PISA 2009

• Leaves 13 countries:- England (Focus)- Asian Tigers (Hong Kong, Japan, Singapore, Taiwan)- Other developed (e.g. Italy, Norway, US)

22

Measures‘Average’ = Mean z-score

‘Inequality’ = Standard deviation of test scores

‘Low achievers’ = 10th percentile of test distribution

‘High achievers’ = 90th percentile of test distribution

‘Low SES’ = 0 – 25 books in the home

‘High SES’ = More than 200 books in home

(i) Measuring construct of interest?

(ii) Measurement error?

23

Methods‘Average’ = OLS regression (allowing for clustering). Two-sample

t-test for statistical significance

‘Low achievers’ = Quantile regression at 10th percentile. Bootstrapped standard errors to allow for clustering.

‘High achievers’ = Quantile regression at 90th percentile. Bootstrapped standard errors to allow for clustering.

SES differences in achievement = OLS regression. Maths test scores as response. Books in home as key explanatory variable. Controls for gender and immigrant status.

24

Results

…. we are just as far behind at age 9/10 as at 15/16 (on average)

Gap between England and East Asia does not widen between 9/10 and 15/16

(in terms of our relative position)

Age 9/10 Age 13/14 Age 15/16

-0.2

0

0.2

0.4

0.6

0.8

1

Hong KongJapanSingaporeEnglandTaiwan

(Ave

rage

test

scor

e (in

tern

atio

nal z

-sco

res)

….(some evidence) low achievers catch-up (age 10 – age 16)

-0.50

-0.30

-0.10

0.10

0.30

Taiwan

Hong Kong

Japan

Singapore

England

Low achievers in England may improve relative to low achievers in East Asian countries between primary and secondary school

…. But the high achievers may lose some ground

High achievers in England may fall back relative to those in East Asian countries between primary and secondary school

-0.3

-0.1

0.1

0.3

0.5

0.7

Taiwan Hong Kong Japan

Singapore

England

Cha

nge

(inte

rnat

iona

l sta

ndar

d de

viat

ion)

Age 9/10 Age 13/14 Age 15/160.2

0.4

0.6

0.8

1.0

1.2

EnglandSingaporeJapanTaiwanHong KongCross-country average

Soci

o-ec

onom

ic g

radi

ent i

n te

st sc

ores

SES gap in math test scores

Big gap in England….…seems to increase between 10 and 16

….but this holds true in all countries

Caution neededDifficulties with measurement of SES (and with books in the home)

29

Conclusions

Suggests (but does not conclusively prove) that....

• ‘Poor’ PISA ranking not about failing secondary schools...- Just as far behind East Asian countries before children each secondary school (on average)

• .......though still possible some changes are needed- E.g. Curriculum / incentives to stretch the highest achievers more?

• Focus should perhaps be earlier years???- Primary school? Early childhood?

• .... but still can’t rule out ‘culture’ as the dominant force?

- Likely to take hold early in life.- Hence consistent with the large cross-national differences at age 10.