Building Evidence in Education:Workshop for EEF evaluators

2nd June: York6th June: London

www.educationendowmentfoundation.org.uk

The EEF by numbers

83 evaluationsfunded to

date

2,300 schools

participating in projects

34 topics in

the Toolkit

16 independent evaluation

teams

500,000 pupils involved in EEF projects

14 members of EEF team

£210mestimated spend over lifetime of

the EEF

6,000 heads

presented to since launch

10 reports

published

Session 1: Design

Adapting DesignCarole Torgerson (Durham) David Torgerson (York Trials Unit)

Calculating effect sizesAdetayo Kasim (Durham)

Calculating Effect Sizes for

Cluster Randomized Trials

Adetayo Kasim

Main Points

• Over estimation of effect size when CLUSTER

LEVEL ANALYSIS is used.

• Disconnection between hypothesis testing and

effect size from MULTILEVEL MODELS

Calculating Effect Size

 

 

Calculating Effect Size• Cluster level analysis (CLA) - Two stage approach

• Summarise data to cluster level

• Calculate effect size using summarised data

• Multilevel models (MLM)

• Analyse pupils level data, but accounts for intra cluster

correlation

• Calculate effect size using WITHIN cluster variability

Calculating Effect Size

Illustration 1: Simulation study

 

 

 

 

Value Method

Mean

MLM 0.41(0.4) 0.39(0.59) 0.41(0.75) 0.39(0.91) 0.4(1.11) 0.38(1.33)

CLA 0.41(0.4) 0.39(0.59) 0.41(0.75) 0.39(0.91) 0.4(1.11) 0.38(1.33)

SE

MLM 0.43(0.06) 0.57(0.13) 0.73(0.18) 0.89(0.22) 1.07(0.27) 1.28(0.33)

CLA 0.39(0.10) 0.56(0.14) 0.73(0.18) 0.89(0.22) 1.07(0.27) 1.28(0.33)

SD

MLM - W 1.98(0.15) 1.99(0.15) 1.99(0.15) 2.00(0.15) 1.99(0.15) 2.00(0.15)

MLM - T 2.00(0.14) 2.10(0.16) 2.23(0.20) 2.38(0.24) 2.56(0.31) 2.80(0.39)

CLA 0.62(0.15) 0.89(0.23) 1.15(0.29) 1.40(0.35) 1.69(0.43) 2.03(0.52)

Calculating Effect Size

Simulation 1: MEAN, SE and SD

• MEAN(SD) from 10,000 simulated data

Calculating Effect Size

Simulation 2: Calculating effect size assuming within cluster variance

Effect size Method 0 0.1 0.2 0.3 0.4 0.5

0.2

MLM - W 0.20(0.20) 0.20(0.26) 0.21(0.30) 0.20(0.36) 0.2(0.42) 0.19(0.49)

MLM - T 0.20(0.20) 0.18(0.23) 0.17(0.26) 0.15(0.28) 0.13(0.29) 0.11(0.28)

CLA 0.64(0.69) 0.50(0.67) 0.43(0.68) 0.36(0.66) 0.31(0.67) 0.25(0.66)

0.3

MLM - W 0.30(0.20) 0.3(0.25) 0.3(0.30) 0.3(0.36) 0.3(0.42) 0.31(0.49)

MLM - T 0.29(0.20) 0.28(0.23) 0.25(0.26) 0.22(0.28) 0.20(0.29) 0.17(0.29)

CLA 0.95(0.72) 0.76(0.68) 0.63(0.69) 0.52(0.67) 0.46(0.67) 0.39(0.67)

0.4

MLM - W 0.41(0.21) 0.40(0.40) 0.40(0.31) 0.40(0.36) 0.40(0.42) 0.40(0.50)

MLM - T 0.39(0.20) 0.37(0.24) 0.34(0.26) 0.30(0.28) 0.26(0.29) 0.22(0.30)

CLA 1.26(0.74) 1.00(0.72) 0.84(0.70) 0.71(0.69) 0.61(0.69) 0.52(0.67)• MEAN(SD) of Hedges Effect Size from 10,000 simulated data

Calculating Effect Size• Cluster level analysis may overestimates effect size

when between variability is negligible and there is substantial variability within clusters

• Effect sizes based on within cluster variance and total variance from multilevel model are comparable when between cluster variance is negligible

• Using only within cluster variance could result in different conclusions based on effect sizes and hypothesis testing when there is a substantial variability between clusters

Discussion

Total variance from multilevel model

Cluster level analysis

Within cluster Variance from multilevel model

OR

OR

?

References

A. Brand, M.T. Bradley, L.A. Best, G. Stoica (2008) Accuracy of effect size estimates from published psychological research. Perceptual and Motor Skills, 106 (2) (2008), pp. 645–649

Larry V. Hedges (2007) Effect sizes in cluster-randomized designs. Journal of Educational and Behavioural Statistics, 32(4), pp. 341-370

Tymms P., Merrell C. and Henderson B. (1997) The first year at school: a quantitative investigation of the attainment and progress of pupils. Educational research and Evaluation, 3(2), pp. 101 - 118