Portugal’s Educational Asymmetries Through the Lens of PISAiave.pt/images/FicheirosPDF/Estudos_Internacionais/...Alentejo Central Alentejo Litoral Algarve Alto Alentejo Alto Minho

Portugal’s Educational AsymmetriesThrough the Lens of PISA

João Marôco, Ph. D. (PISA 2015 NPM)[email protected]

16 de maio 2017

What is PISA?

(OECD, 2016)

Programme for International Student Assessment …«(…) assesses the extent to which 15-year-old students, near the end of their compulsory education, have acquired key knowledge and skills that are essential for full participation in modern societies.»

What is PISA?

SCIENCE LITERACY: «the ability to engage with science related issues, and with the ideas of science, as a reflective citizen».

READING LITERACY: «understanding, using, reflecting on and engaging with written texts, in order to achieve one’s goals, knowledge and potential, and to participate in society»

MATHEMATICAL LITERACY:«capacity to formulate, employ and interpret mathematics in a variety of contexts; reasoning and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena.»

COLLABORATIVE PROBLEM SOLVING:

«ability to work with two or more people to solve a problem.»

33%

33% 22%

4%4%

4%

TEST DESIGN:Total testing time for all domains: 13h 30min.

Test Duration: 2 h - Planned Missingness/Multiple Matrix Design( + 30 min. Student Questionnaire)

• Major Domain – SCIENCE – 1 h testing – All Students• Minor Domains – Different Student proportions – 1 h testing• 66 Test versions

STUDENTS, PARENTS and SCHOOL QUESTIONNAIRES

What is PISA?

The PISA test• Multiple choice and open-ended items• Variety of information sources/stimulus (texts, maps, graphics, figures and computer simulations• 3 Classical domains (SCIE, READ, MATH) + 1 New Domain (Collaborative Problem Solving)

Why should we care about PISA?

Why should we care about PISA?

• Data from international standardized assessments can be useful in research on causal /correlational factors within or across education systems (Rey, 2010)

• S. Breakspear (2012):o Policy-makers in most participating countries see PISA as an important indicator of system

performance;o PISA reports impact policy problems and set the agendas for national policy debate; o Policymakers accept PISA as a valid and reliable instrument for internationally benchmarking system

performance and changes over time; o Countries have started policy reforms in response to PISA reports

«Your education today is your economy tomorrow!»Andreas Schleicher, OECD

Breakspear S ‘The Policy Impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance’, OECD Education Working Paper number 71, 2012

72 Countries and Economies17 565 Schools

509 000 Students

143 000 Parents

95 000 TeachersOECD MEMBERS: OECD PARTNERS:Australia, Austria, Belgium, Canada, Chile, Czech Republic,Denmark, Estonia, Finland, France, Germany, Greece,Hungary, Iceland, Ireland, Israel, Italy, Japan, Latvia,Luxembourg, Mexico, Netherlands, New Zealand, Norway,Poland, Portugal, Slovak Republic, Slovenia, Republic ofKorea, Spain, Sweden, Switzerland, Turkey, UnitedKingdom, United States of America.

Albania, Algeria, Argentina, Brazil, Bulgaria,,Beijing-Shangai-Jiangsu-Guandong [B-S-J-G (China)], Hong Kong (China), Macau (China),Colombia, Costa Rica, Croatia, Cyprus, Dominican Republic, Georgia,Indonesia, Jordan, Kazakhstan, Kosovo, Lebanon, Lithuania, FYRMacedonia, Malaysia, Malta, Moldova, Montenegro, Peru, Qatar, Romania,Russian Federation, Singapore, Chinese Taipei, Thailand, Trinidad andTobago, Tunisia, United Arab Emirates, Uruguay, Vietnam.(italics – PBA)

Who Participated in PISA 2015?

PISA 2015 Portugal

Schools/Students selected by a multistage random sampling procedure:• 1st Stage: Stratified (NUTS III and School type) random sample of schools• 2nd Stage: Simple random sample of students [15 yrs 3 mo. and 16 yrs 2 mo. who have completed at least 6 yrs of

formal schooling either academic, vocational or professional].

246 (222 public + 24 private or cooperative)(Sampling rate: 24%)

4228 (M = 46.7 anos; 72% ♀)

7325 (M = 15.8 years; 50% ♂) (Sampling rate: 7.5%)

R. A. Azores: 21% (oversampling)A. M. Lisbon: 18%A. M. Porto: 13%Other NUTS III: 1 5 %

6881 Parents/Legal guardians

Sample

Alto Minho AltoTâmegaTerras de

Trás-os-Montes

Cávado

A. M. Porto

AveTâmega e Sousa

Douro

Beiras e Serrada Estrela

Beira Baixa

Alto Alentejo

Alentejo Central

Baixo Alentejo

Algarve

Altentejo Litoral

A.M. Lisboa

Lezíria do Tejo

Oeste

Médio Tejo

R. LeiriaR. Coimbra

R. AveiroViseu Dão Lafões

R. A. Açores

R. A. Madeira

459

501*

470

498*

454

492ns

2000 2003 2006 2009 2012 2015

Ano

SCIE READ MATH1000

0

OCDE

Portugal

Scor

e on

PIS

A’s

Scal

e

PRT PISA Results

400 450 500 550READ Score

PORTUGAL

PISA 2015 Results by NUTS III

450 475 500 525 550SCIE Score

400 450 500 550MATH Score

Which variables can explain Regional Asymmetries?

Scientific Literacy:Major Domain in PISA 2015Strong Correlations withMath and Reading Literacies

SCI as proxy for PISA literacy

0

200

400

600

800

1000

0 200 400 600 800 1000

SCI S

core

Score (MATH or READ)

rSCI ,MATH = .89***

rSCI, READ = .86***

SCI by NUTS III

450 475 500 525 550SCIE Score Sig.< national mean = national mean Sig. > national mean


Student level Parents level School level


BELONG

DISCLISCI

EPIST

INSTSCIE

JOYSCIE

MOTIVAT

PRESUPP

SCIEEFF

b=-0.02

b=0.11

b=0.22

b=0.07

b=0.01

b=0.10

b=0.18

b=0.12

R2 = 0.31***

BSMJ

b=0.28LEVEL 1 OLSAssuming NO Regional Effects(IDB Analyzer v4.0)


0

100

200

300

400

500

600

700

800

900

1000

0 20 40 60 80 100

SCI

Scor

e

BSMJ Score

y = 2,07x + 379,61R2 = 0,15***

OECD R2 = 0,13

Students’ Expected Occupational Status (BSMJ) The index of the expected occupational status Students’ responses concerning their expected occupation at age 30 and a description of this job. The index is derived from recoding the responses into four-digit International Standard Classification of Occupations (ISCO) codes, which are then mapped to the International Socio-Economic Index of Occupational Status (ISEI) index. Higher scores of BSMJ indicate higher levels of expected occupational status.


450 475 500 525 550SCIE Score

50 55 60 65 70BSMJ (ISEI) Score


BELONG

DISCLISCI

EPIST

INSTSCIE

JOYSCIE

MOTIVAT

PRESUPP

SCIEEFF

b=0.01

b=0.09

b=0.24

b=0.01

b=0.02

b=0.07

b=0.15

b=0.09

BSMJ

b=0.29LEVEL 1 HLM

Clusters = NUTS IIIAv. Cluster Size = 293ICC = 0.04Des. Effect = 12.01(mPlus v7.2)

STDYX Var =0.664*** R2 = 0.34***


0,00 0,25 0,50 0,75 1,00

Alentejo CentralAlentejo Litoral

AlgarveAlto Alentejo

Alto MinhoAlto TâmegaA. M. LisboaA. M. Porto

AveBaixo Alentejo

Beira BaixaBeiras e Serra da…

CávadoDouro

Lezíria do TejoMédio Tejo

OesteR. A. Madeira

R. A. AçoresRegião de Aveiro

Região de CoimbraRegião de LeiriaTâmega e Sousa

Terras de Trás-os-…Viseu Dão Lafões

bSCI.BSJM

p 0,05 p > 0,05

Alentejo Central

Alentejo Litoral

Algarve

Alto Alentejo

Alto Minho

Alto Tâmega

A. M. Lisboa

A. M. Porto

Ave

Baixo Alentejo

Beira Baixa

Beiras e Serra da Estrela

Cávado

Douro

Lezíria do Tejo

Médio TejoOeste

R. A. Madeira

R. A. Açores

Região de Aveiro

Região de CoimbraRegião de Leiria

Tâmega e Sousa Terras de Trás-os-Montes

Viseu Dão Lafões

y = 2,9162x + 322,16R² = 0,1638

440

460

480

500

520

540

30 40 50 60 70 80

SCI

Scor

e

BSMJ Score

OECD PRT


0

100

200

300

400

500

600

700

800

900

1000

-4 -2 0 2

SCI

Scor

e

EPIST Score

y = 33,42x + 494,47R2 = 0,13***

OECD R2 = 0,10

EPIST: Epistemological Beliefs

Students beliefs about the nature of knowledge in science and about the validity of scientific methods of enquiry as a source of knowing. Students whose epistemic beliefs are in agreement with current views about the nature of science can be said to value scientific approaches to enquiry.


450 475 500 525 550SCIE Score

0.1 0.2 0.3 0.4 0.5EPIST Score


Alentejo Central

Alentejo Litoral

Algarve

Alto Alentejo

Alto Minho

Alto Tâmega

A. M. Lisboa

A. M. Porto

Ave

Baixo Alentejo

Beira Baixa


Cávado

Douro

Lezíria do Tejo

Médio TejoOeste

R. A. Madeira

R. A. Açores

Região de Aveiro

Região de CoimbraRegião de Leiria

Tâmega e Sousa Terras de Trás-os-Montes

Viseu Dão Lafões

y = 71,181x + 480,04R² = 0,1768***

440

460

480

500

520

540

-0,5 -0,3 0,0 0,3 0,5 0,8 1,0

SCI

Scor

e

EPIST Score

OECD PRT

0,00 0,25 0,50 0,75 1,00




AveBaixo Alentejo

Beira BaixaBeiras e Serra da Estrela

CávadoDouro


OesteR. A. Madeira



Terras de Trás-os-…Viseu Dão Lafões

bSCI.BSJM

p 0,05 p > 0,05


R2 = 0.16***

ESCS

PQGENSCI

PQSCHOOL

b=0.11

b=-0.03

EMOSUPP

b=0.34

CURSUPP

b=0.08

b=-0.05

LEVEL 1 OLSAssuming NO Regional Effects(IDB Analyzer v4.0)


R2 = 0.19***

ESCS

PQGENSCI

PQSCHOOL

b=0.11

b=-0.03

EMOSUPP

b=0.29

CURSUPP

b=0.01

b=-0.05

LEVEL 1 HLM

Clusters = NUTSIIIAv. Cluster Size = 293ICC = 0.04Des. Effect = 12.01(mPlus v7.2)

STDYX Var =0.811***

0

100

200

300

400

500

600

700

800

900

1000

-4,0 -3,0 -2,0 -1,0 0,0 1,0 2,0 3,0

SCI

Scor

e

ESCS

y = 30,84x + 513,57R2 = 0,15***

OECD R2 = 0,13


ESCS - index of economic, social and cultural status (ESCS) was derived from three variables related to family background: parents’ highest level of education (PARED), parents’ highest occupation status (HISEI), and home possessions (HOMEPOS), including books in the home. HOMEPOS is a proxy measure for family wealth.

450 475 500 525 550SCIE Score

-1.00-0.75-0.50-0.25ESCS Score


0,00 0,25 0,50 0,75 1,00

R. A. AçoresR. A. Madeira

AlgarveAlentejo Central

Alto AlentejoLezíria do TejoBaixo Alentejo

Alentejo LitoralA.M. Lisboa

Beiras Serra EstrelaMédio TejoBeira Baixa

Viseu Dão LafõesR. Leiria

R. CoimbraR. Aveiro

OesteTerras de Trás-os-Montes

DouroTâmega e Sousa

Alto TâmegaA.M. Porto

AveCávado

Alto Minho

bSCI. ESCS

Alentejo Central

Alentejo Litoral

Algarve

Alto Alentejo

Alto Minho

Alto Tâmega

A. M. Lisboa

A. M. Porto

Ave

Baixo Alentejo

Beira Baixa


Cávado

Douro

Lezíria do Tejo

Médio TejoOeste

R. A. Madeira

R. A. Açores

Região de Aveiro

Região de Coimbra

Região de Leiria

Tâmega e Sousa

Terras de Trás-os-Montes

Viseu Dão Lafões

y = 48,706x + 521,89R² = 0,4528

450

460

470

480

490

500

510

520

530

540

550

-1,4 -1,2 -1 -0,8 -0,6 -0,4 -0,2 0 0,2 0,4

Mea

n SC

I Sc

ore

Mean ESCS

Overall PRT R2 = 0,15 OECD R2 = 0,13

p 0,05 p > 0,05


OECDPRT

R2 = 0.301***


SCHTYPE

CLSIZE

STUDBEHAV b=0.22

STAFSHORT

b=-0.10

TEACHBEHAV

RATCMP1

SCIRES

PROSTCE

b=-0.39

b=-0.12

b=0.31

b=-0.17b=0.18

b=0.02

LEVEL 1 OLS Aggregated at Schools’ levelAssuming NO Regional Effects(PV1-10 on SCH Level Variables with SCHWEIGHTS)

EDUSHORTb=-0.17


300

350

400

450

500

550

600

650

700

-4 -2 0 2 4

SCI

Scor

e

STUDBEHA

y = -3,81x + 483,21R2 = 0,01*

PRTOECD

Região de Coimbra R. A. Açores Alentejo Central

STUDBEHAV - Student behaviours hindering learning School principals’ views of how student behaviours affects learning.


3 LEVEL HLMClusters = NUTSIII CNTSCHIDAv. Cluster Size = 33.4ICC = 0.167; Des. Effects = 6.4STDYX Var =0.643***Assuming constant slopes R2 = 0.34***

SCHTYPE

CLSIZE

STUDBEHAV b=0.24

EDUSHORT

b=-0.19

TEACHBEHAV

RATCMP1

SCIRES

PROSTCE

b=-0.31

b=-0.09

b=0.135

b=-0.19b=0.05

b=0.04

STAFSHORT b=-0.05


450 475 500 525 550SCIE Score

-0.5 0.0 0.5STUDBEHA Score


-1,5 -1,0 -0,5 0,0 0,5 1,0 1,5




AveBaixo Alentejo


CávadoDouro


OesteR. A. Madeira



Terras de Trás-os-MontesViseu Dão Lafões

bSCI.STUDBEHA

p .05 p > .05

Alentejo Central

Alentejo Litoral

Algarve

Alto Alentejo

Alto Minho

Alto Tâmega

A. M. LisboaA. M. Porto

Ave

Baixo Alentejo


Cávado

DouroLezíria do Tejo

Médio TejoOeste

R. A. MadeiraR. A. Açores

Região de AveiroRegião de Coimbra

Região de Leiria

Tâmega e Sousa


Viseu Dão Lafões

y = -16.176x + 499.92R² = 0.13

440

460

480

500

520

540

-1,0 -0,8 -0,5 -0,3 0,0 0,3 0,5 0,8 1,0

SCI

Scor

e

STUDBEHA Score

OECD PRT

300

350

400

450

500

550

600

650

700

10 15 20 25 30 35

SCI

Scor

e

CLSIZE

PRT OECDy = 3.67x + 415.1

R2 = 0,15***


Ave R. A. Açores Médio Tejo


450 475 500 525 550SCIE Score

20.0 22.5 25.0 27.5Mean CLSIZE


Several schools/NUTS III with no data OR homogenous average class size per school

!

300

350

400

450

500

550

600

650

700

10 15 20 25 30 35

SCI

Scor

e

CLSIZE

PRT OECDy = 3.67x + 415.1

R2 = 0,15***

Alentejo Central

Alentejo Litoral

Algarve

Alto Alentejo

Alto Minho

Alto Tâmega

A. M. LisboaA. M. Porto

Ave

Baixo Alentejo

Beira Baixa


Cávado

DouroLezíria do Tejo

Médio TejoOeste

R. A. Madeira

R. A. Açores

Região de Aveiro

Região de Coimbra

Região de Leiria

Tâmega e Sousa


Viseu Dão Lafões

y = 0.9077x + 477.19R² = 0.01

440

460

480

500

520

540

15 20 25 30SC

I Sc

ore

CLSIZE

OECDPRT

Ave R. A. Açores Médio Tejo

So...

Which variables can explain the regional differences in the PRT PISA results?

STANDARDIZED MODEL RESULTS

STDYX Standardization

Two-Tailed Rate ofEstimate S.E. Est./S.E. P-Value Missing

Within LevelPVSCIE ON

BSMJ 0.261 0.012 22.687 0.000 0.165EPIST 0.267 0.018 15.119 0.000 0.110ESCS 0.225 0.019 11.838 0.000 0.081

Residual VariancesPVSCIE 0.718 0.014 50.793 0.000 0.095

Between LevelPVSCIE ON

CLSIZE 0.398 0.089 4.472 0.000 0.033STUBEHA -0.311 0.104 -2.995 0.003 0.039

InterceptsPVSCIE 16.967 1.499 11.316 0.000 0.157

Residual VariancesPVSCIE 0.764 0.078 9.746 0.000 0.031

R-SQUARE

Within Level

Observed Two-Tailed Rate ofVariable Estimate S.E. Est./S.E. P-Value Missing

PVSCIE 0.282 0.014 19.902 0.000 0.095

Between Level

Observed Two-Tailed Rate ofVariable Estimate S.E. Est./S.E. P-Value Missing

PVSCIE 0.236 0.078 3.007 0.003 0.031


BSMJ

EPIST

ESCS

CLSIZE

b=0.26

STUDBEH

b=0.27

b=0.23

b=0.39

b=-0.31

ICC

= 0

.17*

**

R2 =

0.2

8***

Level 1

Level 2

Level 3

R2 =

0.2

4***

Portugal’s Educational AsymmetriesThrough the Lens of PISA

Thank you!

[email protected]

Documents

Portugal’s Educational Asymmetries Through the Lens of PISAiave.pt/images/FicheirosPDF/Estudos_Internacionais/...Alentejo Central Alentejo Litoral Algarve Alto Alentejo Alto Minho