How Big was Response Bias in England to PISA 2003?

John Micklewright & Sylke V. Schnepf

July 2008

2

Motivation• PISA – Programme of International Student Assessment

• Reports from OECD for 2003 exclude the UK due to perceived non-response bias in England

• Simon Briscoe, Economics Editor at The Financial Times: the exclusion is among the ‘Top 20’ recent threats to public confidence in official statistics in the UK

Presentation draws on:

• Response Bias in England in PISA 2000 and 2003, DfES Research Report 771

3

PISA response rates (%): England

2000 2003

England OECD average

England OECD average

School response before replacement

59 86 64 90

School response after replacement

82 92 77 95

Student response 81 90 77 90

4

‘Bias’ in what?

• Mean

• Variance

• % beneath a given threshold

5

Five groups of 15 year olds

i. all pupils in England (less permitted exclusions from the

target population)

ii. all pupils in sampled schools (initial, 1st repl., 2nd repl.)

iii.all pupils in responding schools

iv. sampled pupils in responding schools

v. responding pupils

6

What we do

• Show how (i) mean, (ii) variance, and (iii) % <

thresholds of KS3 and KS4 scores change from

group to group

• Estimate biases in these 3 measures of PISA scores in group v, focusing on problem of pupil response

• Look at both 2000 and 2003

7

KS4 v. PISA scores - respondents

R = 0.78

KS4 score

PISA maths score

8

Means and SDs of KS4 scores, 2003

9

% below or above KS4 thresholds, 2003

group % < bottom decile, KS4 score, group (i)

% > top decile of KS4 score,

group (i) i 10.0 10.0 ii 9.7 10.5 iii 9.5 9.3 iv 7.6 9.3 v 4.9 9.9

10

Creating response weights

Model probability of pupil response for sampled pupils in responding schools (group iv)

Prob(response)i = F(domestic test scorei, other characteristicsi)

Predicted probability of response

KS4 score

11

Bias due to pupil response

reading maths science

Mean +6.5 +6.2 +6.9

SD -3.6 -2.9 -3.0

Extent of bias, calculated as number of PISA score points

(PISA international mean = 500, international SD = 100

Compare with the estimated standard errors (SEs) of the mean and SD for England in 2003:

SE (mean) = 2.9 (reading and maths) and 3.0 for science

SE (SD) = 1.6 (reading and maths) and 1.7 for science

12

However…

• Adjusting the mean by the estimated bias of about 6 points moves England one place in the country rankings

• ‘Post-stratification’ weights are used in many surveys – they could be provided with the data for England

• Think of possible biases in countries just above the 80% student response threshold (Australia, Austria, Canada, Ireland, Poland and the USA)

13

Conclusions

1. It is possible to assess reliably both the direction and magnitude of response biases in England in 2003, and hence compare UK scores with those for other countries

2. The extent of the bias in means and SDs of scores is equal to about double the SEs, but the impact on ‘league table’ rankings is small

3. Biases were similar in 2000, when pupil response in England just reached the required threshold