Effect Size

Robert Coe @ProfCoe

ResearchED 2013, Dulwich College, 7 Sept 2013

2

The case for using effect size

I. What is Effect Size? II. The case for using effect size

(5 reasons)

III. Problems in using effect size (7 problems)

IV. Recommendations (13 recommendations)

V. Further reading

© 2003 Robert Coe, University of Durham

3

IWhat is Effect Size?

Effect Size is the difference between the two groups, relative

to the standard deviation

Effect Size = Mean of experimental group – Mean of control

group Standard deviation

© 2003 Robert Coe, University of Durham

6

IIThe case for using effect size measures

© 2003 Robert Coe, University of Durham

7

1. Effect size allows differences in uncalibrated measures to be interpreted

How would you compare/interpreto 5 marks difference on a home-made testo A gain of 8 points on a standardised testo Improvement of 1 sub-level at KS2o An effect of half a grade at GCSE

Comparing each to the spread of values in the population gives it some meaning

© 2003 Robert Coe, University of Durham

8

2. Effect size allows the accumulation of knowledge

Meta-analysis can combine results from different studies:o What is the average effect?o What factors mediate the effect?o Which interventions have the biggest effects?

Small studies are worth doing because they can be added to the pool

Impact vs cost

Cost per pupil

Eff

ect

Siz

e (

mon

ths

gain

)

£00

8

£1000

Meta-cognitive

Peer tutoringEarly Years1-1

tuitionHomework (Secondar

y)

Mentoring

Summer schools After

schoolAspiration

sPerformanc

e pay

Teaching assistant

s

Smaller classes

Ability grouping

Promising May be

worth it

Notworth

it

Feedback

Phonics

Homework (Primary)

CollaborativeSmall

gp tuition

Parental involveme

ntIndividualised learning

ICT

Behaviour

Social

www.educationendowmentfoundation.org.uk/toolkit

© 2003 Robert Coe, University of Durham

10

3. Effect size emphasises amounts, not just statistical significance

The dichotomous “significant/not” decision is almost never appropriate

The size of a difference is almost always important

“Significance” has many meanings, but is inevitably related to the size of the difference

© 2003 Robert Coe, University of Durham

11

Don’t ignore amounts

-1

-0.5

0

0.5

1

1.5

2

2.5

3

(a)"not significant"

(b)"significant"

(c)"significant"

© 2003 Robert Coe, University of Durham

12

4. Effect size draws attention to the margin of error

Statistical power is important, but often overlooked

Much apparent disagreement is actually just sampling error

© 2003 Robert Coe, University of Durham

13

5. Effect size may help reduce reporting bias

The “file-drawer” problem is alive and well

Within-study reporting bias can also be a problem

© 2003 Robert Coe, University of Durham

14

IIIProblems in using effect size measures

© 2013 Robert Coe, University of Durham

15

Problems with effect size

1. Alternative effect size measures2. Which standard deviation?3. Measurement reliability4. Non-normal distributions5. Interpreting effects: small,

medium, large6. Incommensurability: outcomes,

treatments, populations7. Is it really an ‘effect’?

© 2003 Robert Coe, University of Durham

16

VFurther reading …

© 2013 Robert Coe, University of Durham

17

Coe, R. (2002) It's the effect size, stupid: what effect size is and why it is important. Paper presented at the Annual Conference of the British Educational Research Association, University of Exeter, England, 12-14 September 2002.www.cem.org/attachments/ebe/ESguide.pdf Coe, R.J. (2012) ‘Effect

Size’ in J. Arthur, M. Waring, R. Coe, and L.V. Hedges (Ed.s) (2012) Research Methods and Methodologies in Education. London: Sage.

An introduction, guide and tool for calculating effect sizes:www.cem.org/evidence-based-education/effect-size-resources

1

2

3

© 2013 Robert Coe, University of Durham

18

http://www.nfer.ac.uk/nfer/publications/SEF01/SEF01.pdf

Coe, R. (2004) ‘Issues arising from the use of effect sizes in analysing and reporting research’ in I. Schagen and K. Elliot (Eds) But what does it mean? The use of effect sizes in educational research. Slough, UK: National Foundation for Educational Research.

4