11
Potential impact of PISAPotential impact of PISA
Organisation for Economic Cooperation and Organisation for Economic Cooperation and Development (OECD)Development (OECD)
Michael DavidsonMichael DavidsonOECD/OECD/Directorate for EducationDirectorate for Education
22 What is PISA?
A three-yearly assessment that… …examines the performance of 15-year-olds in
key subject areas as well as a wider range of educational outcomes
• Including students attitudes to learning, their beliefs about themselves, and their learning strategies
…
collects contextual data from students schools, parents and systems to identify policy levers
Contextual data explain 71% of the performance variation among schools internationally
Coverage Representative samples of between 3,500 and
50,000 15-year-old students drawn in each country
Most federal countries also draw regional samples
3333C
reati
ng E
ffect
ive T
each
ing a
nd
Learn
ing E
nvir
on
ments
1st
res
ults
fro
m T
ALI
S
Results from PISA 2006Results from PISA 2006
Scientific knowledge and Scientific knowledge and useuse of that of that knowledge in… knowledge in… …… identifying scientific issues,identifying scientific issues,
…… explaining scientific phenomena,explaining scientific phenomena,
…… using scientific evidenceusing scientific evidence
44 PISA provides key benchmarks for PISA provides key benchmarks for the quality of education systemsthe quality of education systems
11.. Overall performance of education Overall performance of education systemssystems
2.2. Equity in the distribution of learning Equity in the distribution of learning opportunitiesopportunities
Measured by the impact students’ and Measured by the impact students’ and schools’ socio-economic background has on schools’ socio-economic background has on performanceperformance……
…… not merely by the distribution of learning not merely by the distribution of learning outcomesoutcomes
3.3. Consistency of performance standards Consistency of performance standards across schoolsacross schools
4.4. Gender differencesGender differences
55300 350 400 450 500 550 600
FinlandCanadaJ apan
New ZealandAustralia
NetherlandsKorea
GermanyUK
Czech RepublicSwitzerland
AustriaBelgiumI reland
HungarySwedenPoland
DenmarkFranceCroatiaI celand
United StatesSlovak Republic
SpainNorway
LuxembourgI taly
PortugalGreeceSerbia
BulgariaTurkey
RomaniaMontenegro
Mexico
Mean science scoresMean science scores
OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Table 2.1c
The centre line is the mean
The boundaries indicate the 95% confidence limitsThere is only a
significant difference if there is no overlap of scores
66
0
20
40
60
80
100
120
140
Ger
man
y
Cze
ch R
epub
lic
Aus
tria
Hun
gary
Net
herl
ands
Bel
gium
Jap
an
Ital
y
Gre
ece
Slo
vak
Rep
ublic
Tur
key
Swit
zerl
and
Kor
ea
Luxe
mbou
rg
Uni
ted
Sta
tes
Port
ugal
Mex
ico
Uni
ted
Kin
gdom
New
Zea
land
Aus
tral
ia
Can
ada
Irel
and
Den
mar
k
Spa
in
Pola
nd
Swed
en
Nor
way
Icel
and
Fin
land
Is it all innate ability?Variation in student performance
OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 4.1a
Performance variationPerformance variation across schoolsacross schools
77
- 80
- 60
- 40
- 20
0
20
40
60
80
100
120Bul
gari
aG
erm
any
Cze
ch R
epub
lic
Aus
tria
Hun
gary
Net
herl
ands
Bel
gium
Jap
an
Ital
yG
reec
e
Slo
vak
Rep
ublic
Tur
key
Sw
itze
rlan
d
Rom
ania
Ser
bia
Cro
atia
Kor
eaLu
xem
bou
rg
Uni
ted
Sta
tes
Port
ugal
Mex
ico
UK
Mon
tene
gro
New
Zea
land
Aus
tral
iaCan
ada
Irel
and
Den
mar
k
Spa
in
Pola
ndS
wed
en
Nor
way
Icel
and
Fin
land
Variation in student Variation in student performanceperformance
PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.
88
- 80
- 60
- 40
- 20
0
20
40
60
80
100
120Bul
gari
aG
erm
any
Cze
ch R
epub
lic
Aus
tria
Hun
gary
Net
herl
ands
Bel
gium
Jap
an
Ital
yG
reec
e
Slo
vak
Rep
ublic
Tur
key
Sw
itze
rlan
d
Rom
ania
Ser
bia
Cro
atia
Kor
eaLu
xem
bou
rg
Uni
ted
Sta
tes
Port
ugal
Mex
ico
UK
Mon
tene
gro
New
Zea
land
Aus
tral
iaCan
ada
Irel
and
Den
mar
k
Spa
in
Pola
ndS
wed
en
Nor
way
Icel
and
Fin
land
Variation of performance between
schools
Variation of performance within
schools
PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.
Variation in student Variation in student performanceperformance
99 Average performanceof 15-year-olds in science – extrapolate and apply
High science performance
Low science performance
I srael
I talyPortugal Greece
Russian Federation
LuxembourgSlovak Republic,Spain,Iceland Latvia
Croatia
Sweden
DenmarkFrancePoland
Hungary
AustriaBelgiumIreland
Czech Republic SwitzerlandMacao- ChinaGermanyUnited Kingdom
Korea
J apanAustralia
Slovenia
NetherlandsLiechtenstein
New ZealandChinese Taipei
Hong Kong- China
Finland
CanadaEstonia
United States LithuaniaNorway
445
465
485
505
525
545
565
616 … 18 countries perform below this line
EquityEquity
1010
1111 Impact of selected student and school factors on school performance
(after accounting for all other factors in the model)
OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a
Schools practising ability grouping for all
subjects
Academically selective schools but no system-
wide effect
School results posted publicly
One additional hour of science learning at
school
One additional hour of self-study or homework
School activities to promote science
learning
Schools with greater autonomy (resources)
Effect after accounting for the socio-economic
background of students, schools and countries
12121212C
reati
ng E
ffect
ive T
each
ing a
nd
Learn
ing E
nvir
on
ments
1st
res
ults
fro
m T
ALI
S
Evaluating the policy impact Evaluating the policy impact of PISAof PISA
1313 External evaluation of the policy impact of PISA
Objective To what degree does PISA impact
policies, meet expectations and priorities of the participants?
Evaluation criteria Relevance Effectiveness Sustainability Unexpected / Unplanned impacts
1414 External evaluation of the policy impact of PISA
Team of three external evaluators David Hopkins, Dianne Pennock, Jo Ritzen
Two-step procedure Questionnaire to stakeholders in the country In-depth evaluation of 5 case studies
Stakeholder groups Policy makers at the national/federal and local
level as well as members of the PISA Governing Board
School principals, representatives of teacher and parent organisations
Academics and researchers Representatives of the business community and
the media Evaluation report
Conclusions and recommendations based on the empirical results obtained
1515 Strategy for an evaluation of the policy impact of PISA
Effectiveness – Specific questions To what extent are the participating countries and other
stakeholders aware of the output results of PISA? To what extent are these output results being used as
input into the policy-making processes of clients and other stakeholders?
How significant is the influence of the PISA output results used in the policy-making processes?
What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data?
To what extent has PISA provided added value to the national assessment strategies?
What impact has PISA had on policy coherence in participating countries?
1616
• Policy makers are considered the most significant stakeholder group in relation to PISA and its results.
• Policy makers are also considered as the major group responsible for implementing policies in light of PISA.
• The various stakeholder groups assume relatively low levels of responsibility for the PISA results in their countries.
Outcomes – Relevance of PISA
1717
• Overall, PISA is seen as a useful tool for identifying and addressing nationally relevant themes and problems
• In many countries PISA is used to monitor and evaluate the quality and equity of the education system
• Impact of PISA varies from country to country: • In countries with unsatisfactory
outcomes often direct policy impact• In high-performing systems: PISA as an
evaluation mechanism
Outcomes – Relevance of PISA
1818Important aspects of PISA
1919
• In all countries, PISA is seen as an important instrument for policy making with a high level of credibility
• Impact of PISA on policy formation increasing from PISA 2000 to PISA 2006
• Influence of PISA greater at the national / federal level than at the local level or in school practices and classroom instruction
• Policy makers, academics and researchers and the media most aware of PISA
Outcomes – Effectiveness and sustainability of PISA
2020Addressing policy needs
2121Contributing to change
2222
2323
• According to respondents, the influence of PISA could be increased by:• a better coordinated and strategic approach
for the dissemination of PISA results;• further support for various stakeholder groups
in interpreting PISA results and in designing strategies for improvement in light of PISA;
• In some countries, clearer linkage of PISA results to national or federal assessment strategies; and,
• greater utilisation of the PISA results by participating countries.
Outcomes – Effectiveness and sustainability of PISA
2424
• Great public interest and debate• PISA as an instrument to justify reform• “Culture of blame”• Increased interest in empirical educational
research
Unexpected / unplanned outcomes
2525Next step
Evaluation criteria
OutputResults
Processes
Impacts (1)
Impacts (2)
Impacts (3)
Inputs
Expected Outcomes (1)
Expected Outcomes (2)
Expected Outcomes (3)
Policy needs, issues
and problems facing
Membercountries
Relevance Effectiveness
Efficiency
Unexpected/Unplanned Impacts & Sustainability
CommitteeOrientation*
Committee functioning
Polic
y im
pacts
in
Me
mb
er c
ountrie
s
*Also includes any objectives defined in committee mandates. Level 3 Expected Outcomes include the OECD’s Strategic Objectives.
Recommended
