9.09.0A taste of the ImportanceA taste of the Importance

of Effect Sizeof Effect Size

The Basics of Effect Size The Basics of Effect Size Extraction and Statistical Extraction and Statistical

Applications for Meta-Applications for Meta-AnalysisAnalysis

Robert M. BernardRobert M. Bernard

Philip C. AbramiPhilip C. Abrami

Concordia UniversityConcordia University

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

April 12, 2005 3

What is an Effect size?What is an Effect size?

• A descriptive metric that characterizes the standardized difference (in SD units) between the mean of a control group and the mean of a treatment group (educational intervention)

• Can also be calculated from correlational data derived from pre-experimental designs or from repeated measures designs

April 12, 2005 4

Characteristics of Characteristics of Effect SizesEffect Sizes

• Can be positive or negative • Interpreted as a z-score, in SD units, although individual effect sizes are not part of a z-score distribution

• Can be aggregated with other effect sizes and subjected to other statistical procedures such as ANOVA and multiple regression

• Magnitude interpretation: ≤ 0.20 is a small effect size, 0.50 is a moderate effect size and ≥ 0.80 is a large effect size (Cohen, 1992)

April 12, 2005 5

Zero Effect SizeZero Effect SizeES = 0.00

Control Group

Intervention Group

Overlapping Distributions

April 12, 2005 6

Moderate Effect SizeModerate Effect Size

Control Group

Treatment Group

ES = 0.40

April 12, 2005 7

Control Condition

Intervention Condition

ES = 0.85

April 12, 2005 8

Large Effect SizeLarge Effect Size

Control Group

Intervention Condition

ES = 0.85

April 12, 2005 9

Percentage Interpretation Percentage Interpretation of Effect Sizesof Effect Sizes

• ES = 0.00 means that the average treatment participant outperformed 0% of the control participants

• ES = 0.40 means that the average treatment participant outperformed 65% of the control participants (from the Unit Normal Distribution)

• ES = 0.85 means that the average treatment participant outperformed 80% of the control participants

April 12, 2005 10

Independence of Independence of Effect SizesEffect Sizes

• Ideally, multiple effect sizes extracted from the same study should be independent from one another

• This means that the same participants should not appear in more than one effect size

• In studies with one control condition and multiple treatments, the treatments can be averaged, or one may be selected at random

• Using effect sizes derived from different measures on the same participants is legitimate

April 12, 2005 11

Independence: Treatments Independence: Treatments & Measures& Measures

R O1

R X1 O1

R X2 O1

R X3 O1

R O1

R Xpooled O1

R O1O2

R X1 O1O2

One outcome

Two outcomes, one for O1 and one for O2

April 12, 2005 12

Effect Size ExtractionEffect Size Extraction• Effect size extraction is the process of identifying relevant statistical data in a study and calculating an effect size based on those data

• All effect sizes should be extracted by two coders, working independently

• Coders’ results should be compared and a measure of inter-coder agreement calculated and recorded

• In cases of disagreement, coders should resolve the discrepancy in collaboration

April 12, 2005 13

gHedges =YExperimental −YControl

((NE −1)⋅SD2E + (NC −1)SD2

C )) / (NTot −2)⋅ 1−

34(NE + NC )−9

⎛

⎝⎜⎞

⎠⎟

ΔGlass =YExperimental − YControl

SDControl

ES Calculation: ES Calculation: Descriptive StatisticsDescriptive Statistics

dCohen =YExperimental −YControl

(SD2E + SD2

C ) / 2

April 12, 2005 14

Examples from Three StudiesExamples from Three StudiesStudy

nE nC ME MC SDE SDC SDP ΔG dC gH

Study 1: Equal ns and roughly equal standard deviations

S-1 41 4162.5

59.3

7.0 5.6 6.30.57

0.51

0.50

Study 3: Roughly equal ns and different standard deviations

S-3 19 22 62.5

48.6

14.1

5.6 12.2

2.48

1.14

1.11

Study 2: Different ns and roughly equal standard deviations

S-2 38 14 70.4

80.5

10.8

10.1

10.5

–1.00

–0.96

–0.95

April 12, 2005 15

Extracting Effect Sizes in the Extracting Effect Sizes in the Absence of Descriptive StatisticsAbsence of Descriptive Statistics• Inferential Statistics (t-test, ANOVA, ANCOVA, etc.) when the exact statistics are provided

• Levels of significance, such as p < .05, when the exact statistics are not given (t can be set at the conservative t = 1.96 (Glass, McGaw & Smith, 1981; Hedges, Shymansky & Woodworth, 1989)

• Studies not reporting sample sizes for control and experimental groups should be considered for exclusion

April 12, 2005 16

Other Codable Data Other Codable Data Regarding Effect sizeRegarding Effect size

• Type of statistical data used to extract effect size (e.g., descriptives, t-value)

• Type of effect size, such as posttest only, adjusted in ANCOVA, etc.

• Direction of the statistical test • Reliability of dependent measure• In pretest/posttest design, the correlation between pretest and posttest

April 12, 2005 17

Examples from CT Examples from CT Meta-AnalysisMeta-Analysis

• Study 1: pretest/posttest, one-group design, all descriptives present

• Study 2: posttest only, two-group design, all descriptives present

• Study 3: pretest/posttest, two-group design, all descriptives present

• Coding Sheet for 3 studies

April 12, 2005 18

Mean and VariabilityMean and Variability

Variability

ES+

Note: Results from Bernard, Abrami, Lou, et al. (2004) RER

April 12, 2005 19

Variability of Effect Variability of Effect SizeSize

• The standard error of each effect size is estimated using the following equation:

σ̂ 2 (d) =nE + nC

nEnc

+d2

2(nE + nC )

The average effect size (d+) is tested using the following equation: with N – 2 degrees of freedom (Hedges & Olkin, 1985).

t =d+ σ̂ 2 (d)

April 12, 2005 20

Testing Homogeneity of Testing Homogeneity of Effect SizeEffect Size

Q =(di −d+)2

σ̂ 2 (di )i=1

k

∑Note the similarity to a t-ratio.

Q is tested using the sampling distribution of 2 with k – 1 degrees of freedom where k is the number of effect sizes (Hedges & Olkin, 1985).

April 12, 2005 21

Homogeneity vs. Homogeneity vs. Heterogeneity of Effect Heterogeneity of Effect

SizeSize• If homogeneity of effect size is established, then the studies in the meta-analysis can be thought of as sharing the same effect size (i.e., the mean)

• If homogeneity of effect size is violated (heterogeneity of effect size), then no single effect size is representative of the collection of studies (i.e., the “true” average effect size remains unknown)

April 12, 2005 22

Example with Fictitious DataExample with Fictitious Data

Study nE nC SDP d Q

Study 1

19 22 62.5 48.6 13.912.2

1.140.11

7.85

Study 2

12 15 18.7 16.9 1.8 4.3 0.420.15

0.33

Study 3

32 22 79.6 82.2 –2.618.9

–0.14

0.08

1.45

Study 4

41 41 62.5 59.3 3.2 6.3 0.510.05

1.98

Study 5

38 24 70.4 80.5–

10.110.5

–0.96

0.08

17.66

Totals

142 124d+ =

0.135*

∑Q =

29.28**

YE YCΔ σ̂ 2 (d)

*d+ is not significant, p > .05; **2 is significant, p < .05

April 12, 2005 23

Graphing the Graphing the Distribution of Effect Distribution of Effect

SizesSizesForest Plot

–1.5 –1.0 –0.5 0.0 0.5 1.0 1.5

Study 1

Study 2

Study 3

Study 4

Study 5

Mean

Favors Control

Favors Treatment

Units of SD

April 12, 2005 24

Statistics in Statistics in Comprehensive Comprehensive

Meta-Analysis™ Meta-Analysis™

Comprehensive Meta-Analysis 1.0 is a trademark of BioStat®

Note: Results from Bernard, Abrami, Lou, et al. (2004) RER

April 12, 2005 25

Examining Study FeaturesExamining Study Features

• PurposePurpose: to attempt to explain : to attempt to explain variability in effect sizevariability in effect size

• Any nominally coded study Any nominally coded study feature can be investigatedfeature can be investigated

• In addition to mean effect In addition to mean effect size, variability should be size, variability should be investigatedinvestigated

• Study features with small Study features with small ks ks may be unstable may be unstable

April 12, 2005 26

Examining the Study Examining the Study Feature GenderFeature Gender

d+ = +0.14

k = 60

Overall

Effect

Males

d+ = –0.14

k = 18

Females

d+ = +0.24

k = 32

April 12, 2005 27

ANOVA on Levels of Study Features

Note: Results from Bernard, Abrami, Lou, et al. (2004) RER

April 12, 2005 28

Sensitivity AnalysisSensitivity Analysis

• Tests the robustness of the findings• Asks the question: Will these results stand up when potentially distorting or deceptive elements, such as outliers, are removed?

• Particularly important to examine the robustness of the effect sizes of study features, as these are usually based on smaller numbers of outcomes

April 12, 2005 29

Meta-RegressionMeta-Regression

• An adaptation of multiple linear regression

• Effect sizes weighted by in regression

• Used to model study features and blocks of study features with the intention of explaining variation in effect size

• Standard errors , test statistics (z) and confidence intervals for individual predictors must be adjusted (Hedges & Olkin, 1984)

[σ̂ 2 (d)]

1 σ̂ 2 (d)

April 12, 2005 30

Selected ReferencesSelected ReferencesBernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade,

A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How Does Distance Education Compare to Classroom Instruction? A Meta-Analysis of the Empirical Literature. Review of Educational Research, 74(3), 379-439.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

Hedges, L. V. & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. [ERIC Document Reproduction Service No. ED 309 952].