Testing Differences Among Several Sample Means Multiple t Tests vs. Analysis of Variance

Testing Differences Among Several Sample Means

Multiple t Tests vs. Analysis of Variance

Several Sample Means

• What might we do if we had more than two samples?

X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3


• Specifically how many t Tests could you do?

X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3



X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3



X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3



X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3

Multiple t Tests and Family-wise Error Rate

• If you do all possible pair-wise comparisons (C), what happens to the overall probability of making a Type I error?


• If you do all possible pair-wise comparisons (C), what happens to the overall probability of making a Type I error?

€

α(C)Family-wise Error Rate =


• How could we prevent the family-wise error rate from exceeding .05 ?


• How could we prevent the family-wise error rate from exceeding .05 ?

• Set the alpha-level for each pair-wise t test to be a fraction of .05; specifically:

€

α pair =α family

C

Multiple t Tests and Familywise Error Rate

• This isn’t usually done in practice because only a few different alpha-levels appear in the t tables

Multiple t Tests and Familywise Error Rate

• This isn’t usually done in practice because only a few different alpha-levels appear in the t tables

• More importantly, consider that C increases dramatically as more samples are added– for 4 samples: C = 6– for 5 samples: C = 10– for 6 samples: C = 15

• Which leads to a precipitous drop in power

Analysis of Variance

• What is needed is a technique that controls family-wise error rate while looking for one or more differences between several sample means


• What is needed is a technique that controls family-wise error rate while looking for one or more differences between several sample means

• That technique is a one-way Analysis of Variance (ANOVA)

Analysis of Variance• Here are three samples, each are

measurements under different treatment conditions:

X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3

Each sample has a mean and variance and the 3 means are a sampling distribution of means


• What would the null hypothesis be?

• What would the alternative hypothesis be?

X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3

Analysis of Variance• What would the null hypothesis be?

– All three samples are taken from the same population so:

€

μ1 = μ2 = μ3 = μ

Analysis of Variance• What would the null hypothesis be?

– All three samples are taken from the same population so:

• What would the alternative hypothesis be?– At least one of the samples is from a different population and hence has a different mean

€

μ1 = μ2 = μ3 = μ

Analysis of Variance• We can estimate the variance of the “null hypothesis”

population by averaging the j variance estimates


• This is called the “Mean Square Error” or “Mean Square Within”

€

MSerror =ˆ σ 1

2 + ˆ σ 22 + ˆ σ 3

2

3€

MSerror =

ˆ σ j2

j=1

k

∑

kin our example:


• MSerror is an estimate of the population variance

€

MSerror =ˆ σ 1

2 + ˆ σ 22 + ˆ σ 3

2

3

Analysis of Variance• MSerror is an estimate of the population variance

• What’s another way we could estimate the population variance (hint: assume the null hypothesis is true)?

€

MSerror =ˆ σ 1

2 + ˆ σ 22 + ˆ σ 3

2

3

Analysis of Variance• Each sample has a mean and variance and the

3 means are a sampling distribution of means

X11

X12

.

.

.X1n

X21

X22

.

.

.X2n

Sample 1 Sample 2

€

X 1

€

X 2means:

X31

X32

.

.

.X3n

Sample 3

€

X 3

Analysis of Variance• Recall that we estimated the variance of a sampling distribution of

means (since we only had one sample) using the equation:

€

ˆ σ X 2 =

ˆ σ 2

n

Analysis of Variance• Now we’ve got more than one sample! So we can turn this

equation around and make an estimate of the population variance called the “Mean Square Effect” or “Mean Square Between”:

€

MSeffect = ˆ σ 2 = n ˆ σ X 2 = n

(X j − X overall )2

j=1

k

∑

k −1€

ˆ σ X 2 =

ˆ σ 2

n


• We now have two different estimates of the population variance: MSerror and MSeffect

• Why might these two estimates disagree?


• MSerror is based on deviation scores within each sample but…



• MSeffect is based on deviations between samples




• MSeffect would overestimate the population variance when…




• MSeffect would overestimate the population variance when…there is some effect of the treatment pushing the means of the different samples apart


• We compare MSeffect against MSerror by constructing a statistic called F


• We compare MSeffect against MSerror by constructing a statistic called F

• If the hull hypothesis:

is true then we would expect:

except for random sampling variation €

μ1 = μ2 = μ3 = μ

€

X 1 = X 2 = X 3 = μ


• F is the ratio of MSeffect to MSerror

€

Fk−1,k(n−1) =MSeffect

MSerror


• F is the ratio of MSeffect to MSerror

• If the null hypothesis is true then F should equal 1.0

€

Fk−1,k(n−1) =MSeffect

MSerror


• Of course there is a sampling distribution of F - if you repeated your experiment many times you would get a distribution of Fs


• Of course there is a sampling distribution of F - if you repeated your experiment many times you would get a distribution of Fs

• The shape of that distribution depends on two different degrees of freedom:– MSeffect has k-1 degrees of freedom

– MSerror has k(n-1) degrees of freedom


• We can look up a critical F from an F table for any given number of degrees of freedom



• If the F statistic we’ve obtained in our experiment exceeds Fcrit then we know that fewer than 5% of such experiments would be likely to obtain this F statistic if the null hypothesis was true



• If the F statistic we’ve obtained in our experiment exceeds Fcrit then we know that fewer than 5% of such experiments would be likely to obtain this F statistic if the null hypothesis was true

• So we can reject the null and conclude that at least one pair of means is different

Documents

Testing Differences Among Several Sample Means Multiple t Tests vs. Analysis of Variance