Moving Beyond the Average Treatment Effect: A Look at Power …cyfs.unl.edu/.../151204-Spybrook.pdf · 2015. 12. 21. · Moving Beyond the Average Treatment Effect: A Look at Power

Moving Beyond the Average

Treatment Effect: A Look at Power Analyses for Moderator Effects in Cluster

Randomized Trials Jessaca Spybrook Western Michigan University December 3, 2015 (Joint work with Ben Kelcey, University of Cincinnati, Nianbo Dong, University of Missouri) *This work was funded by a grant (DGE-1437692) from the National Science Foundation.

Overview

n Background n  Purpose n  2-level CRT n  3-level CRT n  Implications

Slide 2

Background

n What works?

Slide 3

Background

n What works?

n Under what conditions?

Slide 4

Background

n What works?


n For whom?

Slide 5

Purpose

n What works?


n For whom?

Slide 6

Purpose

n Designs ¨ 2-level cluster randomized trial

(2-level CRT)

¨ 3-level cluster randomized trial (3-level CRT)

Slide 7

2-level CRT

n  Suppose a team of researchers are planning an impact study to determine the effectiveness of a new 5th grade science curriculum. They are planning a 2-level CRT with students nested within schools and schools are randomly assigned to the new curriculum or the current curriculum.

Slide 8

2-level CRT – what works?

n What works?

What is the effect of the new science curriculum relative to the current curriculum on science achievement for fifth graders?

Slide 9


n Model

Slide 10

( )( )τσγγ

,0~

,0~

0

200100

NrNeerTy

j

ijijjjij +++=

Main effect of Treatment


n Model

Set

Slide 11

( )( )τσγγ

,0~

,0~

0

200100

NrNeerTy

j

ijijjjij +++=


2σττ

ρ+

=2

01

στ

γδ

+=


n Minimum detectable effect size (MDES) (Bloom, 1995)

MJ-2 is multiplier that approaches 2.8 as J increases, two-tail test with 0.05 level of significance J is total number of sites n is number of individuals per site is the intraclass correlation

Slide 12

( )( )J

nMMDES J/14

2ρρ −+

= −

ρ


n To increase precision ¨ Common practice to include a cluster-level

covariate ¨ School-level pretest

n Model:

Slide 13

ijjjjij erWTy ++++= 0020100 γγγ


2-level CRT

Question Effect of Interest

MDES

What works? Main effect of

treatment

Main effect of treatment w/covariate

Slide 14

( )( )J

nM J/14

2ρρ −+

−

( )( )J

nRM W

J

/1)1(4 2|

3

ρρ −+−−

2-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

J=30 n=10 0.39 n=50 0.29 n=100 0.27 J=60 n=10 0.27 n=50 0.20 n=100 0.19

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80,

Slide 15 .75.0,23.0 2

| == WRρ

2-level CRT – under what conditions?


Is there a difference in the mean effects by school type (urban vs. suburban)?

Slide 16


n Model:

Slide 17

ijjjjjjjij erWSTSTy ++++++= 00403020100 γγγγγ

Cluster level moderator effect

2-level CRT


MDES MDESD


treatment


Under what conditions?

Cluster-level moderator

Slide 18

( )( )J

nM J/14

2ρρ −+

−

( )( )J

nRM W

J

/1)1(4 2|

3

ρρ −+−−

( )( )J

nRM SW

J

/1)1(16 2|

5

ρρ −+−−


What Works?

Main Effect


CL Mod

J=30 n=10 0.39 0.75

n=50 0.29 0.53

n=100 0.27 0.49

J=60 n=10 0.27 0.52

n=50 0.20 0.37

n=100 0.19 0.34


Slide 19 .80.0,75.0,23.0 2

|2| === SWW RRρ

2-level CRT – for whom?

n For whom?

Is there a difference in the mean effect for boys and girls?

Slide 20


n Model:

Slide 21

ijjijjijjij erXTXTy +++++= 011100100 γγγγ

Individual level moderator effect

2-level CRT


MDES MDESD


treatment




For whom? Individual-level moderator

Slide 22

( )( )J

nM J/14

2ρρ −+

−

( )( )J

nRM W

J

/1)1(4 2|

3

ρρ −+−−

( )( )J

nRM SW

J

/1)1(16 2|

5

ρρ −+−−

( )( )J

nRM X

JJn

/1)1(16 2|

)2(*

ρ−−−−


What Works?

Main Effect


CL Mod

For whom?

Ind Mod

J=30 n=10 0.39 0.75 0.51

n=50 0.29 0.53 0.23

n=100 0.27 0.49 0.16

J=60 n=10 0.27 0.52 0.36

n=50 0.20 0.37 0.16

n=100 0.19 0.34 0.11


Slide 23 .20.0,80.0,75.0,23.0 2

|2|

2| ==== XSWW RandRRρ

3-level CRT

n  Suppose a team of researchers are planning an impact study to determine the effectiveness of a school-wide science curriculum. They are planning a 3-level CRT with students nested within teachers nested within schools and schools are randomly assigned to the new curriculum or the current curriculum.

Slide 24


n What works?

What is the effect of the new science curriculum relative to the current curriculum on science achievement for all students?

Slide 25



Is there a difference in the mean effects by school type (urban vs. suburban)? Is there a difference in the mean effects by teacher experience level (0-3 years vs. more than 3 years)?

Slide 26


n For whom?

Is there a difference in the mean effect for boys and girls?

Slide 27

3-level CRT


MDES

What works?

Main effect of treatment

Slide 28

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

3-level CRT


MDES MDESD

What works?




Slide 29

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLLSW

K

//1116 23232|

5

ρρρρ −−++−−

3-level CRT


MDES MDESD

What works?





Teacher-level moderator

Note: u=J*K-(K-2) Slide 30

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLLSW

K

//1116 23232|

5

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLM

u

//1116 2322| ρρρ −−+−

3-level CRT


MDES MDESD

What works?





Teacher-level moderator

For whom? Individual-level moderator

Note: u=J*K-(K-2), v=n*J*K-(J*K)-(K-2) Slide 31

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLM

u

//1116 2322| ρρρ −−+−

( )( )[ ]{ }K

JnRM LLX

v

//1116 232| ρρ −−−

( ) ( )[ ]{ }K

JnRM LLLLSW

K

//1116 23232|

5

ρρρρ −−++−−


What Works?

Main Effect

J=5 n=10 0.24 n=30 0.22 J=30 n=10 0.19 n=30 0.18

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,

Slide 32 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|

2|23 ====== XPSWLL RandRRRρρ


What Works?

Main Effect


CL Mod

J=5 n=10 0.24 0.45

n=30 0.22 0.41

J=30 n=10 0.19 0.34

n=30 0.18 0.33


Slide 33 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|



What Works?

Main Effect


CL Mod


TL Mod

J=5 n=10 0.24 0.45 0.29

n=30 0.22 0.41 0.23

J=30 n=10 0.19 0.34 0.12

n=30 0.18 0.33 0.09


Slide 34 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|



What Works?

Main Effect


CL Mod


TL Mod

For whom?

Ind Mod

J=5 n=10 0.24 0.45 0.29 0.18

n=30 0.22 0.41 0.23 0.11

J=30 n=10 0.19 0.34 0.12 0.08

n=30 0.18 0.33 0.09 0.04


Slide 35 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|


Implications

n Under what conditions? ¨ Cluster level moderator

n  Challenging given current study sizes n  Effectiveness studies rather than efficacy studies n  If priority, need to consider in design stage

60 60 60

30 S 30 U

15C 15C 15 T 15 T

30 T 30 C

40S 20U

20C 10C 10T 20T

30 T 30 C

40S 20U

30C 0C 20T 10T

30 T 30 C

Implications

n Under what conditions? ¨ Teacher level moderator

n  Number of teachers per school matters n  If priority, need to consider in design phase

Slide 37

Implications

n For whom? ¨ Individual level moderator

n  In many cases, reasonable given current study sizes n  If priority, need to consider in design phase

Slide 38

Implications

n Extensions ¨ Multisite studies ¨ R code available – [email protected] ¨ Some parts implemented in PowerUP! -

http://web.missouri.edu/~dongn/ ¨ Add to Optimal Design Plus

Slide 39

Thank You!

Questions? Please email me: [email protected]

Slide 40

Documents

Moving Beyond the Average Treatment Effect: A Look at Power …cyfs.unl.edu/.../151204-Spybrook.pdf · 2015. 12. 21. · Moving Beyond the Average Treatment Effect: A Look at Power