Quality and equity in educational outcomes Seeing school systems through the prism of PISA Organisation for Economic Cooperation and Development (OECD)

1111

Quality and equity in educational outcomes

Seeing school systems through the prism of PISA

Organisation for Economic Cooperation and Development (OECD)

Campbell What Works seminar 9 November 2006

Dr. Karin ZimmerOECD / Directorate for Education

2222

3333 In the dark……all students, schools and education systems look the same…

But with a little light….

4444

But with a little light….

…important differences become apparent….

In the dark……all students, schools and education systems look the same…

5555

0

10

20

30

40

50

60

70

80

90

100U

nite

d S

tate

s

Ger

man

y

Cze

ch R

epub

lic

Nor

way

Den

mar

k

Can

ada

Slo

vak

Rep

ublic

Swed

en

Aus

tria

Jap

an

New

Zea

land

Swit

zerl

and

Uni

ted K

ingd

om

Fin

land

Net

her

land

s

Hun

gary

Luxem

bou

rg

Icel

and

Fra

nce

Aus

tral

ia

Bel

gium

Pola

nd

Irel

and

Kor

ea

Gre

ece

Ital

y

Spa

in

Tur

key

Mex

ico

Port

ugal

1960's 1970's 1980's 1990's

Baseline qualificationsA world of change

Approx. by % of persons with upper secondary qualfications in age groups 55-64, 45-55, 45-44 und 25-34 years

24

1

1

9

6666 Overview

11.. The PISA approach Objectives and methods underlying OECD’s

Programme for International Student Assessment (PISA)

2.2. Where we are today - and where we can be

What PISA shows students in different countries can do with what they have learned

3.3. How we can get there Some policy levers that emerge from

international comparisons

7777

The PISA approach

Measuring the quality of learning outcomes

8888

OECD countries participating from PISA 2000

OECD countries participating from PISA from 2003

OECD partner countries participating from PISA 2000



PISA country participationKey features of PISA 2003 Information collected

volume of the tests– 3½ hours of mathematics assessment, less than half in

multiple-choice format– 1 hour for each of reading, science and problem

solving each student

– 2 hours on paper-and-pencil tasks (subset of all questions)

– ½ hour for questionnaire on background, learning environment, engagement and motivation

school principals– questionnaire (school demography, learning

environment quality) Coverage

PISA covers roughly nine tens of the world economy Representative samples of between 3,500 and 50,000

students

9999Deciding whom to assess...

grade-based sample

OR

age-based sample

For PISA, the OECD countries chose the latter, selecting 15-year-olds in school as the population.

10101010Deciding what to assess...

looking back at what students were expected to have learned

…or…

looking ahead to what they can do with what they have learned.

For PISA, the OECD countries chose the latter.

11111111 Three broad categories of key competencies

Using “tools” interactively to engage with the

world

Acting autonomously

Interacting in diverse groups

e.g.

Using language, symbols and texts

Interacting with informationCapitalising on the potential

of technologies

e.g.

Relating well to othersCo-operating, working in

teamsManaging and resolving

conflicts

e.g.

Acting within the bigger picture

Learning strategiesTaking responsibility and understanding rights and

limits

To analyse, compare, contrast, and evaluate

To think imaginatively

To apply knowledge in real-life situations

To communicate thoughts and ideas effectively

PISA concept of literacyAccessing, managing, integrating

and evaluating written information in order to develop ones knowledge and potential,

and to participate in, and contribute to, society

12121212Using “tools”

interactively to engage with the

world

Acting autonomously


e.g.



of technologies

e.g.



conflicts

e.g.


Forming and conducting life plans

Taking responsibility and understanding rights and

limits





Reading literacy

Using, interpreting and reflecting on written material



world

Acting autonomously


e.g.



of technologies

e.g.



conflicts

e.g.




limits





Scientific literacyUsing scientific knowledge, identifying scientific

questions, and drawing evidence-based conclusions to understand and make decisions about the natural

world



world

Acting autonomously


e.g.



of technologies

e.g.



conflicts

e.g.




limits





Mathematical literacyEmphasis is on mathematical knowledge put into functional use in a multitude of different

situations in varied, reflective and insight-based ways

15151515

Where we are - and where we can be

What PISA shows students can doExamples of the best performing countries

16161616 Average performanceof 15-year-olds in mathematics

High mathematics performance

Low mathematics performanceGreece

Russian Federation

Liechtenstein

Korea

Hong Kong- China

Finland

Netherlands

Canada

Macao- China Switzerland

New Zealand

Belgium

J apan

Australia

I celandCzech Republic

SwedenFranceDenmark

I reland GermanyAustria

Slovak Republic

LuxembourgPoland Hungary

Norway

SpainUnited StatesLatvia

PortugalI taly

440

460

480

500

520

540

61626

17171717 Mathematical literacy in PISAThe real world The mathematical World

A real situation

A model of reality A mathematical model

Mathematical results

Real results

Understanding, structuring and simplifying the situation

Making the problem amenable to mathematical

treatment

Interpreting the mathematical results

Using relevant mathematical tools to solve the problemValidating

the results

18181818 Average performanceof 15-year-olds in mathematics

Low average performance

Large socio-economic disparities

High average performance



High social equity


High social equity

Strong socio-economic impact

on student performance

Socially equitable distribution of

learning opportunities



Russian Federation

Liechtenstein

Korea

Hong Kong- China

Finland

Netherlands

Canada

Macao- China Switzerland

New Zealand

Belgium

J apan

Australia


SwedenFranceDenmark

I reland GermanyAustria

Slovak Republic

LuxembourgPoland Hungary

Norway

SpainUnited StatesLatvia

PortugalI taly

440

460

480

500

520

540

61626

19191919 Durchschnittliche Schülerleistungen im Bereich Mathematik






High social equity


High social equity







Russian Federation

Liechtenstein

Korea

Hong Kong- China

Finland

Netherlands

Canada

Switzerland

New Zealand

Belgium

J apan

Australia


SwedenFrance

Denmark

I relandGermanyAustria

Slovak Republic

LuxembourgPolandHungary

Norway

SpainUnited States Latvia

Portugal I taly

440

460

480

500

520

540

61626

20202020

200

500

800

-3 -2 -1 0 1 2 3

Stu

dent

perf

orm

ance

School performance and schools’ socio-economic background -

Germany

AdvantagePISA Index of social backgroundDisadvantage

Figure 4.13

School proportional to size

Student performance and student SES within schools

School performance and school SES

21212121Stu

dent

perf

orm

ance

School performance and schools’ socio-economic background -

Denmark


Figure 4.13

300

500

700

-3 -2 -1 0 1 2 3


Student performance and student SES



OECD

OECD

OECD

22222222

200

500

800

-3 -2 -1 0 1 2 3

Stu

dent

perf

orm

ance

School performance and schools’ socio-economic background - Finland


Figure 4.13

Student performance and student SES




23232323

0

20

40

60

80

100

120

140

Tur

key

Hun

gary

Jap

an

Bel

gium

Ital

y

Ger

man

y

Aus

tria

Net

herl

ands

Cze

ch R

epub

lic

Kor

ea

Slo

vak

Rep

ublic

Gre

ece

Swit

zerl

and

Luxe

mbou

rg

Port

ugal

Mex

ico

Uni

ted

Sta

tes

Aus

tral

ia

New

Zea

land

Spa

in

Can

ada

Irel

and

Den

mar

k

Pola

nd

Swed

en

Nor

way

Fin

land

Icel

and

Is it all innate ability?Variation in student performance

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

24242424

- 80

- 60

- 40

- 20

0

20

40

60

80

100

Tur

key

Hun

gary

Jap

an

Bel

gium

Ital

y

Ger

man

y

Aus

tria

Net

herl

ands

Cze

ch R

epub

lic

Kor

ea

Slo

vak

Rep

ublic

Gre

ece

Swit

zerl

and

Luxe

mbou

rg

Port

ugal

Mex

ico

Uni

ted

Sta

tes

Aus

tral

ia

New

Zea

land

Spa

in

Can

ada

Irel

and

Den

mar

k

Pola

nd

Swed

en

Nor

way

Fin

land

Icel

and

Variation of performance

between schools

Variation of performance within

schools

Is it all innate ability?Variation in student performance in mathematics

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

In some countries, parents can rely on high and consistent standards across schools

In Canada, Denmark, Finland, Iceland and Sweden average student performance is high…

… and largely unrelated to the individual schools in which students are enrolled.

In other countries, large performance differences among schools persist

In Austria, Belgium, Germany, Hungary, Italy, Japan, the Netherlands and Turkey, most of the performance variation among schools lies between schools…

… and in some of these countries, most notably those that are highly stratified, a large part of that variation is explained by socio-economic inequalities in learning opportunities

25252525

How can we get there?

Levers for policy that emerge from international comparisons…

…and what countries have done with the findings

26262626

350

400

450

500

550

600

0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 100,000

Money matters but other things do too

Mexico

Greece

Portugal Italy

Spain

GermanyAustria

Ireland

United States

Norway

Korea

Czech republic

Slovak republicPoland

Hungary

Finland

NetherlandsCanada Switzerland

IcelandDenmark

FranceSweden

BelgiumAustralia

Japan

R2 = 0.28

Cumulative expenditure (US$)

Perf

orm

an

ce in

math

em

ati

cs

Spending per student is positively associated with average student performance…

…but not a guarantee for high outcomes Australia, Belgium, Canada, the Czech Republic,

Finland, Japan, Korea and the Netherlands do well in terms of “value for money”…

…while some of the big spenders perform below-average

27272727High ambitions

and clear standards

Access to best practice and quality

professional development

Sympathy doesn’t raise standards – aspiration does PISA suggests that students and schools

perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations– Among these aspects, students’ perception of

teacher-student relations and classroom disciplinary climate display the strongest relationships

28282828 Challenge and support

Weak support

Strong support

Lowchallenge

Highchallenge

Strong performance

Systemic improvement

Poor performance

Improvements idiosyncratic

Conflict

Demoralisation

Poor performance

Stagnation

29292929 Governance of the school system In many of the best performing countries

School-based decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities

The provision of standards and curricula at national/subnational levels is combined with advanced evaluation and support systems

– That are implemented by professional agencies Process-oriented assessments and/or

centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress

Standard setting and equity-related goals Key objectives:

– Raise educational aspirations, establish transparency over educational objectives, reference framework for teachers

Approaches range from definition of broad educational goals up to formulation of concise performance expectations

Some countries go beyond establishing educational standards as mere yardsticks and use performance benchmarks that students at particular age or grade levels should reach

Instruments– Minimum standards, targets defining excellence,

normative performance benchmarks

Monitoring and equity-related goals Diverging views how evaluation and assessment can

and should be used– Some see them primarily as tools to reveal best practices

and identify shared problems in order to encourage teachers and schools to improve and develop more supportive and productive learning environments

– Others extend their purpose to support contestability of public services or market-mechanisms in the allocation of resources

– e.g. by making comparative results of schools publicly available to facilitate parental choice or by having funds following students

Differences in type of performance benchmarks being used and reported for the various stakeholders involved, including parents, teachers and schools




Accountability and intervention in inverse proportion

to success

Devolved responsibility,

the school as the centre of action







High social equity


High social equity







Russian Federation

Liechtenstein

Korea

Hong Kong- China

Finland

Netherlands

Canada

Switzerland

New Zealand

Belgium

J apan

Australia


SwedenFrance

Denmark


Slovak Republic


Norway


Portugal I taly

440

460

480

500

520

540

61626







Low mathematics performance

School with responsibility for deciding which courses are offered

High degree of autonomy

Low degree of autonomy Greece

Russian Federation

Liechtenstein

Korea

Hong Kong- China

Finland

Netherlands

Canada

Switzerland

New Zealand

Belgium

J apan

Australia


SwedenFrance

Denmark


Slovak Republic


Norway


Portugal I taly

440

460

480

500

520

540

61626







Low mathematics performance

Early selection and institutional differentiation

High degree of stratification

Low degree of stratification Greece

Russian Federation

Liechtenstein

Korea

Hong Kong- China

Finland

Netherlands

Canada

Switzerland

New Zealand

Belgium

J apan

Australia


SwedenFrance

Denmark


Slovak Republic


Norway


Portugal I taly

440

460

480

500

520

540

61626

34343434Strong ambitions



Accountability

Devolvedresponsibility,


Integrated educational opportunities

Individualisedlearning




Accountabilityand intervention in inverse proportion

to success

Individualisedlearning

Devolved responsibility,


Integrated educational opportunities

36363636

37373737Creating a knowledge-rich profession in which schools and

teachers have the authority to act, the necessary knowledge to do so wisely, and access to effective support

systems

The tradition of education systems

has been “knowledge poor”

The future of education systems needs to be

“knowledge rich”

National prescription

Professional judgement

Informed professional judgement, the teacher

as a “knowledge worker”

Informed prescription

Uninformed professional judgement, teachers working in isolation

Uninformed prescription,

teachers implement curricula

38383838

Further informationFurther information

www.pisa.oecd.org– All national and international publications– The complete micro-level database

email: [email protected]

Documents

Quality and equity in educational outcomes Seeing school systems through the prism of PISA Organisation for Economic Cooperation and Development (OECD)