OECD’s Leaning Tower of PISA: Approach with Caution, Use with Care

Leslie Rutkowski, PhD David Rutkowski, PhD

Indiana University

Ispra, Italy 2014

PISA - Briefly

I N D I A N A U N I V E R S I T Y 1

International assessment of achievement in math, science & reading.

• 15 year olds • Sample of 35 students (not class-based) • Not focused on curriculum • Major / minor domain

OECD says: “PISA reviews the extent to which students near the end of compulsory education have acquired some of the knowledge and skills that are essential for full participation in modern societies, particularly reading, mathematics and science.”

Where we are headed

• What the critics historically had to say • What the centrist critics say now

• Policy • Technical

• What can we do with PISA

I N D I A N A U N I V E R S I T Y 2

Historic criticisms (not comprehensive)

• Some scholars simply oppose large-scale testing of any kind

• Many have objected to league-table ranking

• Others criticize PISA in particular: • Promoting a free-market agenda in

education • A narrow, limited view of an educational

system • Heavy bias toward OECD countries (but still

recruiting many partner participants)

But these points are not our focus I N D I A N A U N I V E R S I T Y 3

A new type of critic – The Centrists

• Grew as assessments grew (from NAEP to TIMSS to PISA)

• Not anti-assessment • Rather, understanding International Large-

Scale Assessment (ILSA) methodology led to Appreciation for what ILSAs can do Growing concerns over what ILSAs are

being asked to do

I N D I A N A U N I V E R S I T Y 4

Centrists and the OECD Policy Agenda

In some ways we have a “perfect storm” • Increasingly, the OECD is using PISA to

make policy recommendations (not just or even mostly in education)

• Centrists argue that, methodologically, data are being pushed to and beyond reasonable boundaries – and they have the wherewithal to say so

I N D I A N A U N I V E R S I T Y 5

One Example: Schleicher (2013)

“…PISA shows what is possible in education, it takes away excuses from those who are complacent, and it helps countries see themselves in the mirror of the educational results and educational opportunities delivered by the world’s leaders in education.” “Success will go to those individuals, institutions and countries which are swift to adapt, slow to complain and open to change.” “PISA can help to make that happen.”

I N D I A N A U N I V E R S I T Y 6

http://oecdeducationtoday.blogspot.com/2013/12/are-chinese-cheating-in-pisa-or-are-we.html

Deconstructing a Bit “But PISA shows what is possible in education, it takes away excuses from those who are complacent…”

– How is this even possible for one test administered to 15-year olds every three years in three subjects (only one major domain) to tell us what is possible in education?

“and it helps countries see themselves in the mirror of the educational results and educational opportunities delivered by the world’s leaders in education.”

– Does a high PISA score equate to being a world leader in education?

“Success will go to those individuals, institutions and countries which are swift to adapt, slow to complain and open to change.”

– Are ILSA in general and the OECD/PISA in particular the vehicle to indicate this change? Moreover, should we be worried that the OECD claims that they have keys to the vehicle for success?

Centrists & PISA Technical Issues

Although the issues get little popular press attention, methodological criticisms run deep

• Achievement model • Sampling • Trending

I N D I A N A U N I V E R S I T Y 8

PISA Technical Issues - Comparability

Achievement model assumes homogeneous populations of examinees (Peru = Italy = Singapore = USA) Empirical Findings: This assumption is questionable and has consequences for rankings and comparing achievement scores across countries But! All is not lost…

• The OECD has implemented some improvements that acknowledge heterogeneity of testing populations

I N D I A N A U N I V E R S I T Y 9

Examining estimation methods

• We compared several methods for estimating achievement and found that the approach typically used in ILSA contexts has meaningful impacts on country rankings.

• This is especially true for “middle” performers.

I N D I A N A U N I V E R S I T Y 10

Results

I N D I A N A U N I V E R S I T Y 11

Country "True" Score "True" Rank

International Parameters

Rank

OECD Parameters

RankEasy Rank

Shanghai 0.56017 1 1 1Korea 0.40979 2 2 2

Singapore 0.23272 3 3 3Belgium 0.08174 4 4 4Poland 0.0512 5 5 5Norway 0.0323 6 6 6Estonia 0.02822 7 8 8Hungary -0.0063 8 15 15Sweden -0.01457 9 7 7Taiwan -0.01986 10 11 10Ireland -0.02129 11 10 11USA -0.02408 12 14 14

Germany -0.04188 13 9 9Great Britain -0.07415 14 12 12

Italy -0.08792 15 13 13Greece -0.13345 16 16 16Chile -0.46533 17 17 17 17

Romania -0.75886 18 18 18 18Indonesia -0.9755 19 19 19 19

Peru -1.27798 20 20 20 20Kyrgyzstan -1.81302 21 21 21 21

Summary

• From preliminary findings, the easy booklet approach doesn’t harm or help rankings for low performing systems.

• But it probably helps with telling what low performing systems’ kids know and can do.

• And heterogeneity in the “middle” should be incorporated in the models to get stable estimates for these educational systems.

I N D I A N A U N I V E R S I T Y 12

PISA Technical Issues - Sampling

The Shanghai sample • 79% of 15 year olds were captured in sampling

frame

• Loveless (2014) argues that there is a large swath of missing 15-year olds

• We compared top 75% of Massachusetts PISA participants to Shanghai The massive gaps aren’t so massive

I N D I A N A U N I V E R S I T Y 13

The Missing 15-Year Olds

I N D I A N A U N I V E R S I T Y 14

National POP (mil)

15 Year Olds Reported by PISA

15 Year Olds % of POP

Predicted Number of 15 year olds

Difference

PISA's 15 year olds as % of Predicted (C/E)

Malaysia 28.3 544,302 1.92333 364,283 180,019 149.42%Mexico 112.3 2,114,745 1.88312 1,445,546 669,199 146.29%Chinese Taipei 23.2 328,356 1.41533 298,635 29,721 109.95%Korea 49.4 687,104 1.3909 635,886 51,218 108.05%Ireland 4.5 59,296 1.31769 57,925 1,371 102.37%United States 309.3 3,985,714 1.28862 3,981,366 4,348 100.11%France 62.8 792,983 1.26271 808,373 -15,390 98.10%Lithuania 3.3 38,524 1.16739 42,478 -3,954 90.69%Switzerland 7.8 87,200 1.11795 100,403 -13,203 86.85%Hungary 10 111,761 1.11761 128,722 -16,961 86.82%Portugal 10.6 108,728 1.02574 136,445 -27,717 79.69%Italy 60.3 605,490 1.00413 776,193 -170,703 78.01%Greece 11.3 110,521 0.97806 145,456 -34,935 75.98%Germany 81.8 798,136 0.97572 1,052,945 -254,809 75.80%Japan 128.1 1,241,786 0.96939 1,648,927 -407,141 75.31%Spain 46 423,444 0.92053 592,120 -168,676 71.51%Shanghai-China 23 108,056 0.46981 296,060 -188,004 36.50%

PISA AVE 35.36 497,728 1.2872

Source: http://www.brookings.edu/blogs/brown-center-chalkboard/posts/2014/01/08-shanghai-pisa-loveless

Important note

OECD stance is that the sample is representative and that these problems don’t exist. http://oecdeducationtoday.blogspot.cz/2013/12/are-chinese-cheating-in-pisa-or-are-we.html

I N D I A N A U N I V E R S I T Y 15

Our Analysis

Original differences (SE): Adjusted differences:

I N D I A N A U N I V E R S I T Y 16

Sample Math Reading Science Shanghai-China 613 (3) 570 (3) 580 (3) Massachusetts 514 (6) 527 (6) 527 (6)

Sample Math Reading Science Shanghai-China 613 (3) 570 (3) 580 (3)

MA Top 75% of Schools 532 (7) 545 (7) 547 (7) MA Bottom 25% of Schools 453 (8) 467 (8) 461 (8)

Selective samples, closing gaps

Taking uncertainty into account: • Math: Shanghai as low as 606, MA as high as 546 • Reading: Shanghai as low as 564, MA as high as

559 • Science: Shanghai as low as 574, MA as high as

561 The point: the test is sensitive to the sample of examinees. Vietnam (56%), Albania (55%), and Turkey (68%) remain in the international rankings without caveat (PISA 2012 Results in Focus)

I N D I A N A U N I V E R S I T Y 17

Trends and Test Design

• Mazzeo & von Davier were commissioned to review the PISA design in 2008 and reported some concern in several areas

• Mixed design of booklets (math, science reading)

• Rotated booklet design • With major/minor domain, these factors lead

to context/booklet effects • Possibly too few trend items, especially in

minor domains

I N D I A N A U N I V E R S I T Y 18

Summary

Several policy and methodological issues urge caution* on the part of: - Researchers - Media - Policy makers *Note: caution does not mean prohibited!

I N D I A N A U N I V E R S I T Y 19

SO WHAT CAN ILSA’S DO?

I N D I A N A U N I V E R S I T Y 20

ILSAs Have a Lot to Offer

• They provide unique data to many countries (The Czech Republic as one example)

• They also work as training tools for national systems

• They create an important assessment infrastructure

• They offer one form of evidence on the performance in select areas of education

• They have moved the field of assessment forward (methodologically)

I N D I A N A U N I V E R S I T Y 21

Some ways they can inform policy

Improved and more academic research using the data

– PISA/TIMSS are used… – But in the scheme of things largely

untouched in academic research More in-depth learning from others:

– What are the “top” performers doing? – What are “peer” countries doing?

I N D I A N A U N I V E R S I T Y 22

A Policy Example

NCEE in the US has done qualitative work on the top performing PISA countries. http://www.ncee.org/programs-affiliates/center-on-international-education-benchmarking/top-performing-countries/

I N D I A N A U N I V E R S I T Y 23

NCEE

I N D I A N A U N I V E R S I T Y 24

And from the OECD

I N D I A N A U N I V E R S I T Y 25

And For Researchers

• When used carefully, PISA can provide a wealth of useful data

• But! • Avoid causal claims • Use as just one piece of evidence • Keep in mind that achievement is a fallible

measure • Draw on multiple cycles for replication –

stronger evidence • This is just good advice for any ILSA user

I N D I A N A U N I V E R S I T Y 26

In sum:

This isn’t the answer

I N D I A N A U N I V E R S I T Y 27

This is…

Appreciate PISA for what it is – an imperfect but valuable resource that should be used with caution and, from time to time, needs improvement

I N D I A N A U N I V E R S I T Y 28

Final Thoughts for Europe and ILSAs

Regional comparisons seem a useful way forward. Because all educational assessments are political and negotiated, Europe should consider how it would benefit from an EU assessment. We think there are many advantages:

– Focus on EU needs – Focus on curricular commonalities – A greater sense of what EU peers are doing – Background questionnaires that are focused towards common goals

and understandings – A greater EU voice on the development of the conceptual framework

and understanding of education – The US National Assessment for Educational Progress (NAEP) is a

good example

I N D I A N A U N I V E R S I T Y 29

Thank You!

Leslie Rutkowski: lrutkows@indiana.edu David Rutkowski: drutkows@indiana.edu

I N D I A N A U N I V E R S I T Y 30