Knowledge Economy Forum Organisation for Economic Cooperation and Development (OECD) Results from PISA 2003 Istanbul March 22, 2005 Dr John Cresswell OECD

11

Knowledge Economy Forum

Organisation for Economic Cooperation and Development (OECD)

Results from PISA 2003Istanbul March 22, 2005

Dr John CresswellOECD/Directorate for Education

22

In the dark all education systems look the same

33

In the light, differences between education systems can be seen

44

In the light, differences between education systems can be seen

55 Origins of PISA OECD work on education statistics and

indicators major development commenced in late 1980s most of it funded by voluntary contributions substantial Member engagement through networks

Network on educational outcomes led by US developed a proposal for measurement of outcomes Education Committee formally initiated activity in

1996– 11 Members initially committed

Council decision in 1997– established decentralised Part II programme (became PISA)– Council required that OECD face no costs or financial risks– virtually all OECD Members signed on by then– Education Ministries have paid all costs and bear all risks

6666

OECD Partner countries

OECD countries

PISA 2000 country participation

7777


OECD countries


8888


OECD countries


99

Making international comparisons of achievement requires decisions

about...

what to assess,

whom to assess.

1010Deciding what to assess...

looking back at what they were expected to have learned

OR

looking ahead to what they can do with what they have learned.

For PISA, the OECD countries chose the latter.

1111PISA assessments

Reading literacy Using, interpreting and reflecting on written material.

Mathematical literacy Recognising problems that can be solved

mathematically, representing them mathematically, solving them.

Scientific literacy Identifying scientific questions, recognising what counts

as scientific evidence, using evidence to draw conclusions about the natural world.

1212

Development of the PISA tests

1313 Development of assessments Frameworks by international experts Assessment materials

submitted by countries developed by research consortium screened for cultural bias

– by countries– by expert, international panel– items with prima facie cultural bias removed at this stage

translated from English & French originals trialled to check items working consistently in all countries

Final tests items shown in trial to be culturally biased removed best items chosen for final tests

– balanced to reflect framework– range of difficulties– range of item types (constructed response, multiple choice)

1414

Measuring mathematical literacy inPISA 2003

1515 Mathematical literacy in PISA The capacity to:

identify, understand and engage in mathematics; make well-founded judgements about the role that

mathematics plays in an individual’s current and future:

– private life– occupational life– social life with peers and relatives– life as a constructive, concerned and reflective citizen.

Seen as depending on… mathematical knowledge and skills, ability to think and work mathematically, ability to apply the knowledge in a wide variety of

contexts.

1616 Measuring mathematical literacy in PISA 2003

Content Space and shape (assessed in PISA 2000) Change and relationships (assessed in PISA 2000) Quantity Uncertainty

Process skills Reproduction: use of practised knowledge, routine procedures… Connections: somewhat familiar but not routine… Reflection: insight, creativity in choosing mathematical

concepts… Context

Personal Educational or occupational Public Scientific

1717 Space & shape item

Answers: Yes, No, Yes, Yes

Process skill: connections

Context: educational quasi-realistic problem typical in maths classes not genuine occupational problem

Form: complex multiple-choice

Source: OECD (2004) Learning for tomorrow’s world: First results from PISA 2003, Figure 2.4a, p.52.

1818 Change and relationships item

Scores: 1 for n = 140x0.8 = 112 but no further work shown 2 for correct steps/min but not m/min; correct m/min but not km/hr; correct method but error of calculation; correct km/hr but not giving m/sec 3 for correct m/min (89.6) and m/hr (5.4), rounding acceptable.

Process skill: score 1=connections score 2=connections score 3=reflectionContext: personal

Form: open-constructed

1919 Quantity item

Form: open constructed response

Answer: Yes, with adequate explanation

Process skill: reflectionContext: public

Form: short constructed response

Answer: 12 600 ZAR (unit not required)

Process skill: reproductionContext: public

Form: short constructed response

Answer: 975 SGD (unit not required) Process skill: reproductionContext: public

2020 Uncertainty item

Scores: 1 for “No, not reasonable” but explanation lacking detail (e.g. focusing on exact increase in number of robberies without comparison with total) 2 for “No, not reasonable” with argument focusing on only small part of graph shown, ratio or percentage increase, or need for trend data.

Process skill: connections

Context: personal

Form: open- constructed

2121Deciding whom to assess...

grade-based sample

OR

age-based sample

For PISA, the OECD countries chose the latter, selecting 15-year-olds in school as the population.

2222 Key features of PISA 2003 assessment Information collected

each student

– 2 hours on paper-and-pencil tasks (subset of all questions)

– ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation

school principals

– questionnaire (school demography, learning environment quality)

Sample 275,000 students 41 participating countries

2323 PISA sampling requirements

Population: all 15-year-olds in school Sample

minimum of 150 schools per country two random samples: schools and replacement

schools if school declines, replacement school is invited stringent requirements set by countries (85% of

selected schools, 80% of selected students within schools)

2424

Results from PISA 2003

2525 PISA provides five key benchmarks for the quality of education systems

1. Overall performance of education systems

2. Equity in the distribution of learning opportunities

Measured by the impact students’ and schools’ socio-economic background has on performance…

… not merely by the distribution of learning outcomes

3. Consistency of performance standards across schools

4. Gender differences

5. Foundations for lifelong learning Learning strategies, motivation and attitudes

2626300 350 400 450 500 550 600

Finland

Korea

Canada

Czech Rep.

Austria

Germany

Slovak Rep.

Poland

Hungary

Latvia

USA

Russian Fed.

I taly

Greece

Serbia

Turkey

Mexico

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5c, p.356.

Mean mathematics scores – selected countries

2727OECD

Level 6

Level 5

Level 4

Level 3

Level 2

Level 1

BelowLevel 1

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5a, p.354.

What students can do in mathematics

15%

21%

22%

18%

10%

4%

11%

2929 What students can do in reading

10%

22%

12%

6%

22%

29%

OECD Average

Level 5

Level 4

Level 3

Level 2

Level 1

Below Level 1

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.

3030

0%

20%

40%

60%

80%

100%

Fin

land

Kor

ea

Can

ada

Pola

nd

Uni

ted S

tate

s

Ger

man

y

Aus

tria

Latv

ia

Cze

ch R

epub

lic

Hun

gary

Ital

y

Gre

ece

Slo

vak

Rep

ublic

Rus

sian

Fed

erat

ion

Tur

key

Mex

ico

Ser

bia

Percentage of students at each of the proficiency levels in reading

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.

3131 Performance in all domains

350

400

450

500

550

600

Hong KongFinland

KoreaNetherlands

LiechtensteinJ apan

CanadaBelgiumMacao

SwitzerlandAustralia

New ZealandCzech Rep.

I celandDenmark

FranceSwedenAustria

GermanyI reland

Slovak Rep.Norway

LuxembourgPoland

HungarySpainLatviaUnited

Russian Fed.Portugal

I talyGreeceSerbiaTurkey

UruguayThailandMexico

I ndonesiaTunisia

Brazil

Mathematics

350

400

450

500

550

600

350

400

450

500

550

600

Reading

350

400

450

500

550

600

Science Problem Solving

32323232

Securing an equitable distribution of learning opportunities

Measured by the impact students’ and schools’ socio-economic background has on performance – not merely by the distribution

of learning outcomes

3333

-3 -1 1 3-3 -2 -1 0 1 2 3

HighStu

dent

perf

orm

ance

Social background and student performance

AdvantagePISA Index of social backgroundDisadvantage

Low

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Figure 4.8, p.176.

34343434

Ensuring consistent performance standards across schools

Between and within-school variation in performance

3535

0

20

40

60

80

100

120

140

Tur

key

Hun

gary

Jap

an

Bel

gium

Ital

y

Ger

man

y

Aus

tria

Net

herl

ands

Cze

ch R

epub

lic

Kor

ea

Slo

vak

Rep

ublic

Gre

ece

Swit

zerl

and

Luxe

mbou

rg

Port

ugal

Mex

ico

Uni

ted

Sta

tes

Aus

tral

ia

New

Zea

land

Spa

in

Can

ada

Irel

and

Den

mar

k

Pola

nd

Swed

en

Nor

way

Fin

land

Icel

and

Is it all innate ability?Variation in student performance


3636

- 80

- 60

- 40

- 20

0

20

40

60

80

100Tur

key

Hun

gary

Jap

an

Bel

gium

Ital

y

Ger

man

y

Aus

tria

Net

her

land

s

Cze

ch R

epub

lic

Kor

ea

Slo

vak

Rep

ublic

Gre

ece

Swit

zerl

and

Luxem

bou

rg

Port

ugal

Mex

ico

Uni

ted S

tate

s

Aus

tral

ia

New

Zea

land

Spa

in

Can

ada

Irel

and

Den

mar

k

Pola

nd

Swed

en

Nor

way

Fin

land

Icel

and

Variation of performance

between schools

Variation of performance within

schools

Is it all innate ability?Variation in student performance in mathematics


3737

- 80

- 60

- 40

- 20

0

20

40

60

80

100

Tur

key

Hun

gary

Jap

an

Bel

gium

Ital

y

Ger

man

y

Aus

tria

Net

her

land

s

Cze

ch R

epub

lic

Kor

ea

Slo

vak

Rep

ublic

Gre

ece

Swit

zerl

and

Luxem

bou

rg

Port

ugal

Mex

ico

Uni

ted S

tate

s

Aus

tral

ia

New

Zea

land

Spa

in

Can

ada

Irel

and

Den

mar

k

Pola

nd

Swed

en

Nor

way

Fin

land

Icel

and

Variation in student performance in mathematics

Variation of performance between

schools

Variation of performance within schools

Variation explained by socio-economic level of students and schools


3838

200

500

800

-3 -2 -1 0 1 2 3

Stu

dent

perf

orm

ance

School performance and schools’ socio-economic background - Japan

AdvantagePISA Index of social backgroundDisadvantage

Figure 4.13

Student performance and student SES

Student performance and student SES within schools

School performance and school SES

School proportional to size

40404040

Bridging the gender gap

Performance, attitudes and motivation

4141Gender differences

In reading, girls are far ahead In all countries, girls significantly outperform boys in reading

In mathematics, boys tend to be somewhat ahead In most countries, boys outperform girls

…but mostly by modest amounts…… within classrooms and schools, the gender gap is often larger

Strong problem-solving performance for girls suggests……that it is not the cognitive processes underlying mathematics

that give boys an advantage……but the context in which mathematics appears in school

Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap

– Girls report much lower interest in mathematics, more negative attitudes and much greater anxiety with mathematics…

…and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects

44444444

Creating strong foundations for lifelong learning

Performance, attitudes and motivation

4545 Interest in and enjoyment of mathematics

0 10 20 30 40 50 60 70

I enjoy reading about mathematics.

I look forward to my mathematics lessons.

I do mathematics because I enjoy it.

I am interested in the things I learn about

mathematics.

OECD average Greece Latvia Slovak RepublicPercentage of students

- 60 - 50 - 40 - 30 - 20 - 10 0 10 20 30 40 50 60

Change in

mathematics per unit

of the index

OECD average Greece Latvia Slovak Republic Score points

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.

4646 Anxiety in mathematics0 20 40 60 80 100

I often worry that it will be difficult for me in

mathematics classes.

I get very tense when I have to do

mathematics homework.

I get very nervous doing mathematics

problems.

I feel helpless when doing a mathematics

problem.

I worry that I will get poor marks in

mathematics.

OECD average AustriaCzech Republic Switzerland

Percentage of students

- 60 - 50 - 40 - 30 - 20 - 10 0 10 20 30 40 50 60

Change in

mathematics per unit

of the index

OECD average Austria Czech Republic Switzerland Score points

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.8, p.374 and Figure 3.8, p.139.

47474747

Some features of successful education systems

Insights from earlier PISA analysis

4848

350

400

450

500

550

600

0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 100,000

Student performance and spending per student

Mexico

Greece

Portugal Italy

Spain

GermanyAustria

Ireland

United States

Norway

Korea

Czech republic

Slovak republicPoland

Hungary

Finland

NetherlandsCanada Switzerland

IcelandDenmark

FranceSweden

BelgiumAustralia

Japan

R2 = 0.28

Cumulative expenditure (US$)

Perf

orm

an

ce in

math

em

ati

cs

5050Governance of the school system

In many of the best performing countries Decentralised decision-making is combined

with devices to ensure a fair distribution of substantive educational opportunities

The provision of standards and curricula at national/subnational levels is combined with advanced evaluation systems

– That are implemented by professional agencies Process-oriented assessments and/or

centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress

5151 Public and private schools

0 20 40 60 80 100

Luxembourg

J apan

I taly

Switzerland

Finland

Denmark

Czech Republic

Sweden

Hungary

Austria

Portugal

United States

Netherlands

Slovak Republic

Korea

I reland

Spain

Canada

Mexico

New Zealand

Germany

OECD

United Kingdom

Government schools

Government dependent private

Government independent private

-15

0

-10

0

-50

0 50 100

Observed perf ormance diff erence

Diff erence af ter accounting f or socio-economic background of students

Diff erence af ter accounting f or socio-economic background of students and schools

Private schools perform better

Public schools perform better

5252Organisation of instruction

In many of the best performing countries Schools and teachers have explicit strategies

and approaches for teaching heterogeneous groups of learners

– A high degree of individualised learning processes– Disparities related to socio-economic factors and

migration are recognised as major challenges Students are offered a variety of extra-

curricular activities Schools offer differentiated support

structures for students– E.g. school psychologists or career counsellors

Institutional differentiation is introduced, if at all, at later stages

– Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes

5353M

athe

ma

tics

perf

orm

ance

Decreasing effect of socioeconomic background

High performanceLow SES effect

Low performanceLow SES effect

Low performanceHigh SES effect

High performanceHigh SES effect

UruguayTurkey

I talyPortugal

LatviaUnited States Spain

NorwayHungary Poland

LuxembourgSlovak Republic

AustriaGermany I reland

DenmarkFrance Sweden

Czech RepublicI celand

Australia

J apanBelgium

New ZealandSwitzerland Macao-China

Canada

Netherlands

FinlandHong Kong-China

Korea

Liechtenstein

Russian Federation

Greece

Serbia

400

425

450

475

500

525

550

5757 Disciplinary climateStudents’ views

0 20 40 60 80 100

Students don't listen to what the teacher says.

There is noise and disorder.

The teacher has to wait a long time for students

to quieten down.

Students cannot work well.

Students don't start working for a long time

after the lesson begins.

OECD averageHungary Russian Federation Serbia Slovak Republic Turkey


OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.3a, p.408 and Figure 5.3, p.217.

5858 Teachers’ morale and commitmentPrincipals’ views

0 20 40 60 80 100

The morale of

teachers in this

school is high.

Teachers work with

enthusiasm.

Teachers take pride

in this school.

Teachers value

academic

achievement.

OECD average Austria

Czech Republic Hungary

Russian Federation Turkey


OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.5a, p.412 and Figure 5.5, p.223.

5959Further information

www.pisa.oecd.org– All national and international publications– The complete micro-level database

email: [email protected]

[email protected]

Documents

Knowledge Economy Forum Organisation for Economic Cooperation and Development (OECD) Results from PISA 2003 Istanbul March 22, 2005 Dr John Cresswell OECD