Óscar D. Marcenaro-Gutiérrez y Anna Vignoles

A comparisson of teacher and test based assessment for

Spanish primary and secondary education

Óscar D. Marcenaro-Gutiérrez*Anna Vignoles**

*Departamento de Estadística y Econometría (UMA) y Fundación Pública Centro de Estudios Andaluces (FCEA)

** University of Cambridge

XI Conference FESMadrid, 10-12th July 2013

G13

1

Outline:

Motivation

Aim

Data

Methodology

FES Madrid

Conclusions

3

FES Madrid

Motivation:

1. Needless to say: importance of education as an engine of socio-economic change.

4

The profitability of investment in education is higher

that investment in capital goods (Psacharopoulos, 1985)

FES Madrid

Importance of training as an engine of socio- economic change: Budgetary effort.

Motivation:

FES Madrid

Motivation:

Central to Economics of Education: Do we have precise measures of students’ outcomes? To what extent does the nature of the assessment criteria produce social class inequalities in academic outcomes?

FES Madrid

Motivation:

2. Teacher assessment matters, because:- Affects how students perceive their own academic ability (self-esteem, etc.).- It is used to determine whether a child can proceed to the next grade.

1. Virtual lack of datasets allowing the comparison “Teachers’ actual assesment-external test assesment”; novelty of the paper.

7

It is crucial to determine whether such assessment is a good guide to pupils’

actual level of achievement:PISA? Actual exams?....

FES Madrid

Aim:

Are assesment tests (test-based assessment –TestA-) and official marks (teachers-based assessment –TeachA-) telling different things? We explore this empirically.

Do these differences vary systematically with pupil characteristics?

8

gender

ethnicity

socio-economic background

FES Madrid

Aim:

Are assesment tests (test-based assessment –TestA-) and official marks (teachers-based assessment –TeachA-) telling different things? We explore this empirically.

Do these differences vary systematically with pupil characteristics?

Differences in achievement levels of pupils in public (100%) schools and semi-private schools.

Do this potential gap discourage further study, in general or in any particular subject area (such as science, arts, etc.), after completing compulsory education? (i.e. future success)

9

FES Madrid

10

Literature:

Gender or Ethnicity of the teacher might interact with that of the student to produce systematic differences in pupil achievement (Dee, 2005 & 2007): teachers over grade students of the same gender to themselves.Comparing teacher assessments of pupil achievement and test scores: Reeves et al (2001) , Gibbons and Chevalier (2008) –UK-, TestA-TeachA consistent.

Gutierrez and Adserá (2012) –as opposed to Calero and Waisgrais (2008)- public school students higher grades: might be explained by differences in grading practices between public and private school teachers.

FES Madrid

Data:

PISA: 15-years-old students -4th year ESO-(OECD countries).

Dependent variables: - Math Test scores (normalised).- Reading Test scores (normalised).

Independent variables:- Student characteristics- Family background 26 variables- School characteristics

PISA-2009: Spain: 25.887 students within 889 schools.Andalusia: 1700 students within 51 schools

11

Data (1st stage):

FES Madrid

Linked databases (novelty):

- Andalusian Social Survey: Education and Homes in Andalusia (Encuesta social: Educación y Hogares en Andalucía, ESOC10); includes diagnostic assesment tests.

- Administrative marking records for those in the survey sample.

Data (2nd stage):

FES Madrid

ASS10:- Students born in 1994 + students born in

1998.

15 years 11 years

- Pseudo-panel: past and future outcomes.

- Very important: Includes interviews aimed at parents and children.

13

Crossing information

(consistency)

Data (2nd stage):

FES Madrid

ASS10:- ADDITIONAL added value: link ASS10 with

information from an administrative information system.

14

Information entered by the educators from public and semi-

private schools

• Repeaters.• Disabled (or with special educational requirements).• Private schools (only 2.5%) –data non avaialble-.

Data (2nd stage):

Excluding

FES Madrid

Andalusia 2009, huge differences with

the rest of Spain?

16

• Differences between natives and immigrants men more substantial in Andalusia as compared to the rest of Spain

• Public school Vs Private school: same differences as for the rest of Spain.

• Mother educational level: variables even more relevant.

• Once again…. IMPORTANCE OF HUMAN CAPITAL ACCUMULATION OF THE PROGENITOR AS A WAY OF ACADEMIC SUCCESS FOR CHILDREN, PARTICULARLY FOR WOMEN.

A) School failure:

B) Performance determinants:

FES Madrid

17

¿Consistency? (PISA-ASS)0

.1.2

.3.4

Den

sity

-4 -2 0 2 4Normalized scores

PISA reading TestA

ASS10 reading TestA

Reading

0.1

.2.3

.4D

ensi

ty


PISA math TestA

ASS10 math TestA

MathSubstantial overlap

FES Madrid

High grade of consistency between PISA-2009 y ASS10-SEN conditional effects:

18

ReadingTestA

(ASS10)TestA

(PISA09)Women 25,93**

*

6,73***

Inmigrant -36,07**

-34,73**

Semi-private 40,10***

8,40*

Mother ed.: primary -9,53 14,01Mother ed.: lower secondary 1,42 8,73Mother ed.: upper secondary 25,54** 20,88*Mother ed.: Higher ed. 35,77*** 41,83***Constant 504,70*

**

477,37**

Observations 1287 843R2 0,12 0.07

¿Consistency?

FES Madrid

0.1

.2.3

.4D

ensi

ty


Difference in assesments (reading)

0.1

.2.3

.4D

ensi

ty


Difference TestA-TeachA (female students)

Difference TestA-TeachA (male studnets)

Difference in assesments (maths)

women men

Differences TestA-TeachA:

FES Madrid

Multivariate analysis

Multinomial Probit

Probability of: Dropping out or

repeating. Enrol in Vocational

training. Academic track

(Sciences, etc.)

OLS regressions/FE/RE

Effects of variables on the difference TestA-

TeachA(ASS-SEN)

20

FES Madrid

Table 2.a. Discrepancies TestA - TeachA (1994 cohort).

21

Specification VReading Maths

Female -0.03 -0.50***Immigrant 0.10 0.26Semi-private school (=1) 0.49*** 0.20**Education level of father:

Degree or higher -0.06 -0.05Education level of mother:

Degree or higher -0.11 -0.11Constant -0.14** 0.12*Number of observations 1,087 1,051R-squared 0.04 0.06

Differences TestA-TeachA: (conditional analyses)

FES Madrid

Table 2.b. Discrepancies TestA - TeachA (1994 and 1998 cohorts).

22


Specification V

Reading Maths

Cohort 1994 (=1) -0.05 -0.06Female -0.06 -0.34***

Immigrant -0.12 -0.02Semi-private school (=1) 0.37*** 0.22***

Education level of father:

Degree or higher -0.07 -0.05Education level of mother:

Degree or higher -0.01 -0.03

Constant -0.07* 0.09**

Number of observations 2,778 2,725R-squared 0.03 0.04

Teachers over estimating achievement of girls?

(preconceived ideas about ability’?)

Teachers under estimating achievement of students at

Semi-private schools.(Do teachers measure the child’s

achievement relative to other????)RE: Virtually identical results (2-3 students from each school).

FES Madrid

23

¿Difference TestA-TeachA varies

across the ability distribution?


FES Madrid

24

Note: Including controls for immigrant, father education level and mother education level.

Table 4: Controlling for quintile in the previous yearSpecification V

Reading MathsPrior achievement -Reading- (TeachAt-1):

2nd quintile -0.113rd quintile -0.64***4th quintile -0.33***5th quintile (top) -0.74***

Prior achievement -Math- (TeachAt-1):

2nd quintile -0.023rd quintile -0.65***4th quintile -0.49***5th quintile (top) -0.60***

Female 0.08 -0.40***Semi-private school (=1) 0.54*** 0.22***Constant 0.21 0.44***Number of observations 995 1,001Number of centres 447 451


pupils TeachAt-1 narrows gap

FES Madrid

25

Note: Including controls for immigrant, father education level and mother education level.

Table 5: Controlling for TeachAt-1


Specification VReading Maths

Prior achievement -Reading- (TeachAt-1): -0.26***

Prior achievement -Math- (TeachAt-1): -0.20***

Female 0.03 -0.50***Semi-private school (=1) 0.48*** 0.17**Household cultural index level:

Average cultural index -0.01 0.07Higher cultural index 0.15* 0.18*

Constant -0.18*** 0.11Number of observations 1,057 1,030Number of centres 466 460

≈

“teachers’ views about the ability of pupils are

persistent”

FES Madrid

26

Table 6: By school type


≈

Specification VSemi-private

schoolPublic school

Reading Maths

Reading

Maths

Female -0.12 -0.50**

*

-0.00 -0.49***

Constant 0.49*** 0.42***

-0.16** 0.11

Number of observations

265 248 822 803

Number of centres 148 137 334 331

FES Madrid

27

¿ To what extent TeachA-TestA at 15 predict education outcomes at age 16?

Choice at 16:

1) Repeating or dropping out of school at age 16 (ref.).

2) Undertaking vocational study;

3) For those staying on the academic track, enrolled in: Sciences, Arts & Humanities or Social Sciences)

FES Madrid

28

Table 7: Exit to:

VocationalTraining

Arts SciencesSocial

SciencesTestA-TeachA (reading) -0. 49*** -0.33** -0.60*** -0.47***Female 0.05 0.25 0.07 0.41***Immigrant -0.05 0.50 -0.34 -0.27

Semi-private school (=1) 0.41 0.19 0.50*** 0.41**Constant -0.71*** -1.31*** 0.97*** 0.90***Number of observations 1164

Χ2 90.39***

“pupils who are under estimated by teachers in terms of their

achievement are more likely to drop out”

Choice at 16:

FES Madrid

Conclusions

High relevance of parents human capital (particularly mothers); this is deeper in Andalusia, relevant to economic growth and labour market opportunities of women!!! (higher performance and lower failure)

29

PISA09 and ASS10 similar results, TestA consistent.

FES Madrid

Conclusions

Systematic differences between teacher assessments and actual test scores.

Girls’ achievement is over estimated by teacher assessments relative to test scores (preconceived ideas?????).

Teachers under estimating achievement of students at Semi-private schools (measuing relative acievements –peer effect-?????).

Pupils who are under estimated by teachers in terms of their achievement are more likely to drop out→ TestA-TeachA has a longer term impact on pupil outcomes.

30

FES Madrid

Thank you!

31

FES Madrid

A comparisson of teacher and test based assessment for

Spanish primary and secondary education

Óscar D. Marcenaro-Gutiérrez*Anna Vignoles**

*Departamento de Estadística y Econometría (UMA) y Fundación Pública Centro de Estudios Andaluces (FCEA)

** University of Cambridge

XI Conference FESMadrid, 10-12th July 2013

G13

32