10 things policymakers should know about learning goals and assessment

OECD EMPLOYER BRANDPlaybook

1

10 things policymakers should know about learning goals and assessment

Andreas SchleicherSt. Petersburg, May 2014

22 Benchmark globally

Lesson 1In a global economy, the benchmark for educational success is no longer solely

improvement by national standards, but the best performing education systems internationally

A world of change – higher education

0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

1995

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)

Tertiary-type A graduation rate (%)

Cost

per

stud

ent

Graduate supply


0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

1995

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)

Tertiary-type A graduation rate (%)

Cost

per

stud

ent

Graduate supply

United States


0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2000

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)

Tertiary-type A graduation rate

United Kingdom


0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2001

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2002

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2003

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2004

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2005

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2006

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2007 Ex

pend

itur

e pe

r st

uden

t at

ter

tiar

y le

vel

(USD

)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2008

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2009 Ex

pend

itur

e pe

r st

uden

t at

ter

tiar

y le

vel

(USD

)



0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2010

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)


IcelandPoland

UK


0 10 20 30 40 50 60 700

5,000

10,000

15,000

20,000

25,000

30,000

2010

Expe

ndit

ure

per

stud

ent

at t

erti

ary

leve

l (U

SD)


US

1717 Don’t sacrifice validity gains for efficiency gains

Lesson 2The kinds of things that are easy to

teach and test are also easy to digitise, automate and outsource

18

1960 1970 1980 1990 2000 2006 200935

40

45

50

55

60

65

70

Routine manualNonroutine manualRoutine cognitiveNonroutine analyticNonroutine interpersonal

Mean task input in percentiles of 1960 task distribution

Changes in the demand for skillsTrends in different tasks in occupations (United States)

Source: Autor, David H. and Brendan M. Price. 2013. "The Changing Task Composition of the US Labor Market: An Update of Autor, Levy, and Murnane (2003)." MIT Mimeograph, June.

1919 Monitor excellence and equity jointly

Lesson 3

The false choice betweenequity and excellence

410

420

430

440

450

460

470

480

490

500

510

520

530

540

550

560

570

580Mean score

High mathematics performance

Low mathematics performance

… Shanghai-China performs above this line (613)

… 12 countries perform below this line

Average performanceof 15-year-olds in

MathematicsFig I.2.13

Socially equitable distribution of learning

opportunities

High mathematics performance

Low mathematics performance

Average performanceof 15-year-olds in

mathematics

Strong socio-economic impact on student

performance

02468101214161820222426

AustraliaAustria

Belgium Canada

Chile

Czech Rep.Denmark

Estonia

Finland

France

Germany

Greece

Hungary

IcelandIreland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US

Singapore

Hong Kong-ChinaChinese Taipei

Macao-ChinaLiechtenstein

Viet Nam

Latvia

Russian Fed.Lithuania

Croatia

SerbiaRomania

Bulgaria United Arab Emirates

KazakhstanThailand

Malaysia

2012Shanghai-China


opportunities


performance

AustraliaAustria

Belgium Canada

Chile

Czech Rep.Denmark

Estonia

Finland

France

Germany

Greece

Hungary

IcelandIreland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US

AustraliaAustriaBelgiumCanadaChileCzech Rep.DenmarkEstoniaFinlandFranceGermanyGreeceHungaryIcelandIrelandIsraelItalyJapanKoreaLuxembourgMexicoNetherlandsNew ZealandNorwayPolandPortugalSlovak Rep.SloveniaSpainSwedenSwitzerlandTurkeyUKUS

2012


opportunities


performance

AustraliaAustria

Belgium Canada

Chile

Czech Rep.Denmark

Estonia

Finland

France

Germany

Greece

Hungary

IcelandIreland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US


2525 Spend wisely to make a difference

Lesson 4

Not more money but better spending choices make a difference

Portu

gal

Spain

Switz

erlan

d

Belg

ium

Kore

a

Luxe

mbo

urg

Germ

any

Gree

ce

Japa

n

Aust

ralia

Unite

d Ki

ngdo

m

New

Zeala

nd

Fran

ce

Neth

erlan

ds

Denm

ark

Italy

Aust

ria

Czec

h Re

publ

ic

Hung

ary

Norw

ay

Icela

nd

Irelan

d

Mexic

o

Finlan

d

Swed

en

Unite

d St

ates

Polan

d

Slov

ak R

epub

lic

-10

-5

0

5

10

15

Salary as % of GDP/capita Instruction time 1/teaching time 1/class size

Contribution of various factors to upper secondary teacher compensation costs, per student as a percentage of GDP per capita (2004)

Percentage points

Difference with OECD average

AustraliaAustria

Belgium Canada

Chile

Czech Rep.Denmark

Estonia

Finland

France

Germany

Greece

Hungary

IcelandIreland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US


Singapore

Shanghai

Singapore

2003 - 2012 Germany, Turkey and Mexico improved both their mathematics performance and equity levels

2828 Keep track of ‘added value’

Lesson 5The country where students go to class matters more than what social class students come from

29School performance and socio-economic background: Brazil

AdvantagePISA Index of socio-economic backgroundDisadvantage

Stu

dent

per

form

ance

Brazil

30

Vietnam

Desempeño del colegio y nivel socioeconómico

VentajaÍndice de PISA de nivel socioeconómicoDesventaja

Des

empe

ño d

e lo

s es

tudi

ante

s

31School performance and socio-economic background: Brazil

AdvantagePISA Index of socio-economic backgroundDisadvantage

Stu

dent

per

form

ance

Brazil

3232Resilience in educationPISA performance by decile of social background

Mex

ico

Gre

ece

Swed

en

Isra

el

Unite

d St

ates

Denm

ark

Aust

ralia

Unite

d Ki

ngdo

m

Cana

da

Aust

ria

Liec

hten

stei

n

Esto

nia

Slov

enia

New

Zea

land

Net

herl

ands

Switz

erla

nd

Belg

ium

Mac

ao-C

hina

Kore

a

Chin

ese

Taip

ei300

325

350

375

400

425

450

475

500

525

550

575

600

625

650

675

Source: PISA 2012

33

33Provide a data-rich school environmentto combat inequities

National and school data

School-based strategies for

analysis

Data for sup-porting learning

3434Le

sson

s fro

m h

igh

perfo

rmer

s

Catching up with the top-performers

Low impact on outcomes

High impact on outcomes

Low feasibility High feasibility

Money pits

Must haves

Low hanging fruits

Quick wins

3535Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins

Commitment to universal achievement

Gateways, instructional systems

Capacity at point of delivery

Incentive structures and accountability

Resources where they yield most

A learning systemCoherence

3636Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 6: Believe that all children can achieve Universal educational standards and personalization as

the approach to heterogeneity in the student body…… as opposed to a belief that students have different

destinations to be met with different expectations, and selection/stratification as the approach to heterogeneity

Clear articulation who is responsible for ensuring student success and to whom

3737Le

sson

s fro

m h

igh

perfo

rmer

s

37

37 High expectations for all students

-0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 0.80 1.00 1.20300

350

400

450

500

550

600

650

481.366786279212

517.501096817955

561.241096454551

391.459888954175

499.749902827587

452.973426858907

409.291567937716

493.934230896316

520.545521676786518.750335282979

394.329333356314

471.131460759248

490.571021411359

481.644744006327489.845098037208

513.525055819928

478.823277433358

505.540743249801

498.95788231768

559.824796201498

494.98467432064426.737491293011

536.406918234208

447.984414978954 478.260635903011

477.044455015488504.150766311124

466.48143014931

518.078519433354

501.497460196644438.738259877415

385.595556395556

422.632355405519

538.134494733918

U.A.E.

514.745238582901522.971758192682

484.319297801971

388.431709907139

375.114451681749

500.026756625414

431.798408505078

368.102547127357

406.999866988793

530.931003950397

409.626613284347

387.824629620249

492.795697239492

501.127422390953

376.4483986347

573.468314296641

487.063181343903

489.373070348755

376.488601072821

420.512967619054

413.281466667708

534.96508297892

553.766659143613

448.859130247604

Russian Fed.

444.554242787643

511.338207501182

485.321181012553

612.675536305453

f(x) = 138.160916953927 x + 477.587612682211R² = 0.368631715648504

PISA mean index of mathematics self-efficacy

Mea

n m

athe

mat

ics

perfo

rman

ce

OEC

D a

vera

ge

Countries where students have stronger be-liefs

in their abilities perform better in mathe-matics

Source: PISA 2012

38 Perceived self-responsibility for failure in mathematics

Percentage of students who reported "agree" or "strongly agree" with the following statements:

I’m not very good at solving mathematics problems

My teacher did not explain the concepts well this week

This week I made bad guesses on the quiz

Sometimes the course material is too hard

The teacher did not get students interested in the material

Sometimes I am just unlucky

0 10 20 30 40 50 60 70 80 90

Russian Federation United States OECD average

%

Fig III.3.6

39

39

39 A continuum of supportBe demanding for every student

Ensure that students feel valued and included and learning is collaborative

4040Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 7: Have clear ambitious goals that are shared across the system and aligned with high stakes gateways and instructional systems

Well established delivery chain through which curricular goals translate into instructional systems, instructional practices and student learning (intended, implemented and achieved)

High level of metacognitive content of instruction …

4141Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 8: Build capacity at the point of delivery Attracting, developing and retaining high quality

teachers and school leaders and a work organisation in which they can use their potential

Instructional leadership and human resource management in schools

Keeping teaching an attractive profession System-wide career development …

4242Le

sson

s fro

m h

igh

perfo

rmer

sLu

xem

bour

g

Thai

land

Sha

ngha

i-Chi

na

Col

ombi

a

Chi

le

Mex

ico

Vie

t Nam

Uru

guay

Kaz

akhs

tan

Bel

gium

Mal

aysi

a

Bra

zil

U.A

.E.

New

Zea

land

Sw

itzer

land

Mac

ao-C

hina

OE

CD

ave

rage

Arg

entin

a

Aus

tria

Irela

nd

Fran

ce

Uni

ted

Kin

gdom

Alb

ania

Can

ada

Latv

ia

Uni

ted

Sta

tes

Cro

atia

Mon

tene

gro

Hun

gary

Slo

veni

a

Ser

bia

Bul

garia

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

Bottom quarter of this index

Mea

n in

dex

Teacher shortage Fig IV.3.5

43

43

43 Prepare for work in disadvantaged schools

Preparation

• Reinforce initial teacher training including curriculum content for disadvantage

• Strengthening diagnostic capacity• Include practical field experience

• Both new and experienced teachers benefit

• Pedagogical and relational strategies

4444Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 9: Align autonomy with accountability Aligned incentive structures

For students How gateways affect the strength, direction, clarity and nature of the incentives

operating on students at each stage of their education Degree to which students have incentives to take tough courses and study hard Opportunity costs for staying in school and performing well

For teachers Make innovations in pedagogy and/or organisation Improve their own performance

and the performance of their colleagues Pursue professional development opportunities

that lead to stronger pedagogical practices A balance between vertical and lateral accountability Effective instruments to manage and share knowledge and spread

innovation – communication within the system and with stakeholders around it

A capable centre with authority and legitimacy to act

4545Le

sson

s fro

m h

igh

perfo

rmer

s

45

45 Align autonomy with accountability

The question is not how many charter schools you have but how you enable every teacher to assume charter-like autonomy

4646Le

sson

s fro

m h

igh

perfo

rmer

s

46

46

-1.5 -1 -0.5 0 0.5 1 1.5300

350

400

450

500

550

600

650

531.551979302783

414.947431329217

430.53288984921

423.795593172672

484.685067484024

507.375949559565

493.913526079401

557.719613495498

454.493852942216459.674291542381

419.468595641077

488.357558008343

404.86657067849406.81928697245

410.692469685374

455.967032005237

396.468122669645

431.953772561969416.098738598916

300.849653448456

527.668467891543

404.539944308878

440.111661967012

474.054187560775

464.989161819408

547.743708881437

626.566663790363

452.789179885987

529.511834268283

497.071637137884

453.49524309675

482.577394045123

532.465311188924

506.274697797594

488.818411796174

402.907104971934

498.55233132561486.358212456265

502.809277446549

485.011835724539

525.143096315803

466.514022482625

460.853234111852

488.150072840935484.3703865799

468.514073102546

499.317279833724

438.810335285436

499.440165643771501.844010272146

478.664970193416480.554307802789

498.658254792673

481.116171960251

503.011259906496490.67709912419

463.432481043829

552.313972933536

478.845972683071R² = 0.133981453407518

Index of school responsibility for curriculum and assessment (index points)

Mat

hem

atic

s pe

rform

ance

(sco

re p

oint

s)Countries that grant schools autonomy over cur-

ricula and assessments tend to perform better in mathe-

matics

Source: PISA 2012

Schools with more autonomy perform better than schools with less autonomy in systems with more collaboration

Less school autonomy

More school autonomy

455

460

465

470

475

480

485

Teachers don't participate in management

Teachers participate in management

Score points

School autonomy for resource allocation x System's level of teachers participating in school managementAcross all participating countries and economies

Fig IV.1.17

Schools with more autonomy perform better than schools with less autonomy in systems with more accountability arrangements



464

466

468

470

472

474

476

478

School data not public

School data public

Score points

School autonomy for curriculum and assessment x system's level of posting achievement data publicly

Fig IV.1.16



455

460

465

470

475

480

485

No standardised math policy

Standardised math policy

Schools with more autonomy perform better than schools with less autonomy in systems with standardised math policies

Score points

School autonomy for curriculum and assessment x system's extent of implementing a standardised math policy (e.g. curriculum and instructional materials)

Fig IV.1.16

50

Written specification of the school's curriculum and educational goals

Written specification of student-performance standards

Systematic recording of data, including teacher and student attendance and graduation rates, test results and professional development of teachers

Internal evaluation/self-evaluation

External evaluation

Written feedback from students (e.g. regarding lessons, teachers or resources)

Teacher mentoring

Regular consultation with one or more experts over a period of at least six months with the aim of improving the school

Implementation of a standardised policy for mathematics

0 10 20 30 40 50 60 70 80 90 100

Percentage of students in schools whose principal reported that their schools have the following for quality assurance and improvement:

Singapore OECD average

%

Quality assurance and school improvement Fig IV.4.14

51

51

51 Square school choice with equity

Financial incentives

for schools

Assistance for disadvantaged

parents

Controlled choice

Financial incentives

Inform par-ents

Foster col-laboration

among teachers and

schools

Use student and school

assessments

5252Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 10: Invest resources where they can make most of a difference

Alignment of resources with key challenges (e.g. attracting the most talented teachers to the most challenging classrooms)

Effective spending choices that prioritise high quality teachers over smaller classes

5353 Align the resources with the challenges

-0.500.511.5300

350

400

450

500

550

600

650

700R² = 0

Equity in resource allocation (index points)

Mat

hem

atic

s pe

rform

ance

(sco

re p

oint

s)

Greater equityLess equity

Adjusted by per capita GDP

Countries with better performance in mathematics tend to allocate educational resources more equitably

Source: PISA 2012

5454 Adequate resources to address disadvantage

Disadvantaged schools reported more teacher shortage

Advantaged schools reported more teacher shortage

Kor

eaEs

toni

aIs

rael

Latv

iaSl

oven

iaIta

lyPo

land

Sing

apor

eA

rgen

tina

Net

herla

nds

Portu

gal

Col

ombi

aFr

ance

Finl

and

Tuni

sia

Mac

ao-C

hina

Spai

nG

reec

eSw

itzer

land

Nor

way

Rus

sian

Fed

.Ja

pan

Aus

tria

Mon

tene

gro

Cro

atia

Can

ada

OEC

D a

vera

geG

erm

any

Den

mar

kH

unga

ryU

nite

d K

ingd

omLu

xem

bour

gH

ong

Kon

g-C

hina

Bel

gium

Icel

and

Viet

Nam

Irela

ndU

nite

d St

ates

Chi

leC

zech

Rep

ublic

Serb

iaTu

rkey

Mex

ico

Indo

nesi

aU

rugu

aySh

angh

ai-C

hina

Slov

ak R

epub

licSw

eden

Bra

zil

New

Zea

land

Aus

tralia

Chi

nese

Tai

pei-0.5

-0.3

-0.1

0.1

0.3

0.5

0.7

0.9

1.1

1.3

1.5Difference between socio-economically disadvantaged and socio-economically advantaged schools

Mea

n in

dex

diffe

renc

e

A shortage of qualified teachers is more of concern in disadvantaged schools

55

55

55 Reduce tracking and grade repetition

Both vertical and horizontal stratification hurt equity

5656Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







A final thought Alignment of policies

across all aspects of the system Coherence of policies

over sustained periods of time Consistency of implementation Fidelity of implementation

(without excessive control)

CAN

5757Le

sson

s fro

m h

igh

perfo

rmer

s

57

57 Thank you

Find out more about our work at www.oecd.org– All publications– The complete micro-level database

Email: [email protected]: SchleicherEDU

and remember:Without data, you are just another person with an opinion

Documents

10 things policymakers should know about learning goals and assessment