1

Multiple ComparisonProcedures

Once we reject H0: ==...c in favor of H1: NOT all ’s are equal, we don’t yet know the way in which they’re not all equal, but simply that they’re not all the same. If there are 4 columns, are all 4 ’s different? Are 3 the same and one different? If so, which one? etc.

2

These “more detailed” inquiries into the process are called MULTIPLE COMPARISON PROCEDURES.

Errors (Type I):We set up “” as the significance level for a hypothesis test. Suppose we test 3 independent hypotheses, each at = .05; each test has type I error (rej H0 when it’s true) of .05. However, P(at least one type I error in the 3 tests) = 1-P( accept all ) = 1 - (.95)3 .14 3, given true

3

In other words, Probability is .14 that at least one type one error is made. For 5 tests, prob = .23.Question - Should we choose = .05, and suffer (for 5 tests) a .23 OVERALL Error rate (or “a” or experimentwise)?

OR

Should we choose/control the overall error rate, “a”, to be .05, and find the individual test by 1 - (1-)5 = .05, (which gives us = .011)?

4

The formula 1 - (1-)5 = .05

would be valid only if the tests are independent; often they’re not.

[ e.g., 1=22= 3, 1= 3

IF accepted & rejected, isn’t it more likely that rejected? ]

1 2

21

3

3

5

When the tests are not independent, it’s usually very difficult to arrive at

the correct for an individual test so that a specified value results for the

overall error rate.

6

Categories of multiple comparison tests

- “Planned”/ “a priori” comparisons (stated in advance, usually a linear combination of the column means equal to zero.)

- “Pairwise” comparisons (every column mean compared with each other column mean)

- “Post hoc”/ “a posteriori” comparisons (decided after a look at the data - which comparisons “look interesting”)

7

(Pairwise comparisons are traditionally considered as “post hoc” and not “a priori”, if one needs to categorize all comparisons into one of the two groups)

There are many multiple comparison procedures. We’ll cover only a few.

Method 1: Do a series of pairwise t-tests, each with specified value (for individual test).

This is called “Fisher’s LEAST SIGNIFICANT DIFFERENCE” (LSD).

8

Example: Broker StudyExample: Broker Study

A financial firm would like to determine if brokers they use to execute trades differ with respect to their ability to provide a stock purchase for the firm at a low buying price per share. To measure cost, an index, Y, is used.

Y=1000(A-P)/AwhereP=per share price paid for the stock;A=average of high price and low price per share, for the day.

“The higher Y is the better the trade is.”

9

}1

1235-112

5 6

27

1713117

17 12

381743

7 5

524131418141917

R=6

CoL: broker

421101512206

14

Five brokers were in the study and six trades were randomly assigned to each broker.

10

= .05, FTV = 2.76

(reject equal column MEANS)

Source SSQ df MSQ FCol

Error

640.8

530

4

25

160.2

21.2

7.56

“MSW”

11

0

For any comparison of 2 columns,

/2/2

CL Cu

Yi -Yj

AR: 0+ t/2 x MSW x 1 + 1

ninj

25 df(ni = nj = 6, here)

SQ Root of Pooled Variance, “s2”, perhaps, in earlier class in basic statistics

:p

12

In our example, with=.05

0 2.060 (21.2 x 1 + 1 )0 5.48

6 6

This value, 5.48 is called the Least Significant Difference (LSD).

When same number of data points, R, in each column, LSD = t/2 x 2xMSW

.R

13

Col: 3 1 2 4 5 5 6 12 14 17

Now, rank order and compare:

Underline Diagram

14

Step 1: identify difference > 5.48, and mark accordingly:

5 6 12 14 173 1 2 4 5

2: compare the pair of means within each subset:Comparison difference vs. LSD

3 vs. 12 vs. 42 vs. 54 vs. 5

**

*

<<<<

* Contiguous; no need to detail

5

15

Conclusion : 3, 1 2, 4, 5

Can get “inconsistency”: Suppose col 5 were 18:

3 1 2 4 5 5 6 12 14 18

Now: Comparison |difference| vs. LSD3 vs. 12 vs. 42 vs. 54 vs. 5

* *

*

<<><

Conclusion : 3, 1 2 4 5 ???

6

16

• Broker 1 and 3 are not significantly different but they are significantly different to the other 3 brokers.

Conclusion : 3, 1 2 4 5

• Broker 2 and 4 are not significantly different, and broker 4 and 5 are not significantly different, but broker 2 is different to (smaller than) broker 5 significantly.

17

MULTIPLE COMPARISON TESTING

AFS BROKER STUDYBROKER ----> 1 2 3 4 5TRADE 1 12 7 8 21 24 2 3 17 1 10 13 3 5 13 7 15 14 4 -1 11 4 12 18 5 12 7 3 20 14 6 5 17 7 6 19

COLUMN MEAN 6 12 5 14 17

ANOVA TABLE

SOURCE SSQ DF MS Fcalc

BROKER 640.8 4 160.2 7.56

ERROR 530 25 21.2

18

Using SPSS Variable Score By Variable Broker

Analysis of Variance

Sum of Mean F F Source D.F. Squares Squares Ratio Prob.

Between Groups 4 640.8000 160.2000 7.5566 .0004Within Groups 25 530.0000 21.2000Total 29 1170.8000

19

Fisher’s LSD USING SPSS 5.0 - MAC Variable Score By Variable Broker

Multiple Range Tests: LSD test with significance level .05

The difference between two means is significant ifMEAN(J)-MEAN(I) >= 3.2558 * RANGE * SQRT(1/ N(I) + 1/ N(J))

with the following value(s) for RANGE: 2.91

(*) Indicates significant differences which are shown in the lower triangle

G G G G G r r r r r p p p p p

3 1 2 4 5 Mean Broker

5.0000 Grp 3 6.0000 Grp 1 12.0000 Grp 2 * * 14.0000 Grp 4 * * 17.0000 Grp 5 * *

Subset 1Group Grp 3 Grp 1Mean 5.0000 6.0000- - - - - - - - - - - - - - - - -Subset 2Group Grp 2 Grp 4 Grp 5Mean 12.0000 14.0000 17.0000- - - - - - - - - - - - - - - - - - - - - - - -

20

USING WINDOWS 8.0

1=column of interest, 2=compared column, 3=difference, 4=std. error, 5=p-value

6, 7 = 95 confidence limits

(1) (2) (3) (4) (5) (6) (7)LSD 1 2 -6.00* 2.658 .033 -11.47 -.53

3 1.00 2.658 .710 -4.47 6.474 -8.00* 2.658 .006 -13.47 -2.535 -11.00* 2.658 .000 -16.47 -5.53

2 1 6.00* 2.658 .033 .53 11.473 7.00* 2.658 .014 1.53 12.474 -2.00 2.658 .459 -7.47 3.475 -5.00 2.658 .072 -10.47 .47

3 1 -1.00 2.658 .710 -6.47 4.472 -7.00* 2.658 .014 -12.47 -1.534 -9.00* 2.658 .002 -14.47 -3.535 -12.00* 2.658 .000 -17.47 -6.53

4 1 8.00* 2.658 .006 2.53 13.472 2.00 2.658 .459 -3.47 7.473 9.00* 2.658 .002 3.53 14.475 -3.00 2.658 .270 -8.47 2.47

5 1 11.00* 2.658 .000 5.53 16.472 5.00 2.658 .072 -.47 10.473 12.00* 2.658 .000 6.53 17.474 3.00 2.658 .270 -2.47 8.47

21

Fisher's pairwise comparisons (Minitab)

Family error rate = 0.268

Individual error rate = 0.0500

Critical value = 2.060 t_(/2)

Intervals for (column level mean) - (row level mean)

1 2 3 4

2 -11.476

-0.524

3 -4.476 1.524

6.476 12.476

4 -13.476 -7.476 -14.476

-2.524 3.476 -3.524

5 -16.476 -10.476 -17.476 -8.476

-5.524 0.476 -6.524 2.476

Minitab: Stat>>ANOVA>>one way anova then click “comparisons”.

22

In the previous procedure, each individual comparison has error rate =.05. The overall error rate is, were

the comparisons independent, 1- (.95)10= .401.

However, they’re not independent.

Method 2: A procedure which takes this into account and pre-sets the overall error rate is “TUKEY’S HONESTLY SIGNIFICANT DIFFERENCE TEST ”.

23

Tukey’s method works in a similar way to Fisher’s LSD, except that the “LSD” counterpart (“HSD”) is not

t/2 x MSW x 1 + 1ni nj

t/2 x 2xMSWR

=or, for equal number of data points/col( ) ,

but tuk X 2xMSW ,R

where tuk has been computed to take into account all the inter-dependencies of the different comparisons.

/2

24

HSD = tuk/2x2MSW R

________________________________________

A more general approach is to write

HSD = q/2xMSW R

where q/2 = tuk/2 x2

--- q = (Ylargest - Ysmallest) / MSW R

---- probability distribution of q is called the

“Studentized Range Distribution”.

--- q = q(c, df), where c =number of columns,

and df = df of MSW

25

q table

26

With c = 5 and df = 25,from table:

q = 4.16 (between 4.10 and 4.17)tuk = 4.16/1.414 = 2.94

Then,

HSD = 4.16

alsox

27

In our earlier example:

Rank order:

3 1 2 4 5

5 6 12 14 17

(No differences [contiguous] > 7.82)

28

Comparison |difference| >or< 7.823 vs. 13 vs. 23 vs. 43 vs. 51 vs. 21 vs. 41 vs. 52 vs. 42 vs. 54 vs. 5

* <<>><>><<<

912*8

11*5*

(contiguous)

7

3, 1, 2 4, 52 is “same as 1 and 3, but also same as 4 and 5.”

29

Tukey’s HSD (“LSD”) Mac

Variable Score By Variable Broker

Multiple Range Tests: Tukey-HSD test with significance level .05

The difference between two means is significant ifMEAN(J)-MEAN(I) >= 3.2558 * RANGE * SQRT(1/ N(I) + 1/ N(J))with the following value(s) for RANGE: 4.15

(*) Indicates significant differences which are shown in the lower triangle G G G G G r r r r r p p p p p

3 1 2 4 5 Mean Broker 5.0000 Grp 3 6.0000 Grp 1 12.0000 Grp 2 14.0000 Grp 4 * * 17.0000 Grp 5 * *

Subset 1Group Grp 3 Grp 1 Grp 2

Mean 5.0000 6.0000 12.0000- - - - - - - - - - - - - - - - - - - - - - - -Subset 2Group Grp 2 Grp 4 Grp 5

Mean 12.0000 14.0000 17.0000

30

Windows 8.0 format, with the same column meanings:

(1) (2) (3) (4) (5) (6) (7)Tukey HSD 1 2 -6.00 2.658 .192 -13.81 1.81

3 1.00 2.658 .995 -6.81 8.814 -8.00* 2.658 .043 -15.81 -.195 -11.00* 2.658 .003 -18.81 -3.19

2 1 6.00 2.658 .192 -1.81 13.813 7.00 2.658 .094 -.81 14.814 -2.00 2.658 .942 -9.81 5.815 -5.00 2.658 .353 -12.81 2.81

3 1 -1.00 2.658 .995 -8.81 6.812 -7.00 2.658 .094 -14.81 .814 -9.00* 2.658 .018 -16.81 -1.195 -12.00* 2.658 .001 -19.81 -4.19

4 1 8.00* 2.658 .043 .19 15.812 2.00 2.658 .942 -5.81 9.813 9.00* 2.658 .018 1.19 16.815 -3.00 2.658 .790 -10.81 4.81

5 1 11.00* 2.658 .003 3.19 18.812 5.00 2.658 .353 -2.81 12.813 12.00* 2.658 .001 4.19 19.814 3.00 2.658 .790 -4.81 10.81

For Tukey’s HSD, the Windows SPSS output also provides another format, called

“homogeneous Subsets” (it doesn’t provide it for Fisher’s LSD):

Broker N Subset 1 Subset 2 Subset 3Tukey HSD 3 6 5.00

1 6 6.002 6 12.00 12.004 6 14.005 6 17.00Sig. .094 .353

31

Tukey's pairwise comparisons (Minitab)

Family error rate = 0.0500

Individual error rate = 0.00706

Critical value = 4.15 q_(1-/2)

Intervals for (column level mean) - (row level mean)

1 2 3 4 2 -13.801 1.801 3 -6.801 -0.801 8.801 14.801 4 -15.801 -9.801 -16.801 -0.199 5.801 -1.199 5 -18.801 -12.801 -19.801 -10.801 -3.199 2.801 -4.199 4.801

32

Exercise: Drug StudyExercise: Drug StudyA drug company are developing two new drug formulations for treating flu, denoted as drug A and drug B. Two groups of 10 volunteers were taken drug A and drug B, respectively, and after three days, their responses (Y) were recorded. A placebo group was added to check the effectiveness of drugs. The larger the Y value is, the more effective the drug is. Here is the data: (MSE=1)

Drug A Drug B Placebo

Index i 1 2 3

Column mean -5.3 -6.1 -12.3

Sample size 10 10 10

33

LSD = t97.5%;27 df 2/10 = 2.052 2/10 = 0.9177

HSD = q97.5%;27 df 1/10 = 3.51 1/10 = 1.110

Placebo Drug B Drug A

Placebo Drug B Drug A

34

Method 3: Dunnett’s testDesigned specifically for (and incorporating the interdependenciesof) comparing several “treatments” to a “control.”

Example: 1 2 3 4 5

6 12 5 14 17

Col

} R=6CONTROL

Analog of LSD (=t1-/2 x 2 MSW )R = Dut1-/2 x 2 MSW

R

35

D tablep. 107

36

Dut1-/2 x 2 MSW/R

= 2.61 (2(21.2) )= 6.94

- Cols 4 and 5 differ from the control [ 1 ].- Cols 2 and 3 are not significantly differentfrom control.

6

In our example: 1 2 3 4 5 6 12 5 14 17

CONTROL

Comparison |difference| >or< 6.941 vs. 21 vs. 31 vs. 41 vs. 5

618

11

<< > >

37

DUNNETTDependent Variable: SCOREDunnett t (2-sided)

MeanDifference

(I-J)

Std.Error

Sig. 95% ConfidenceInterval

(I)BROKER

(J)BROKER

LowerBound

UpperBound

2 1 6.00 2.658 .103 -.93 12.933 1 -1.00 2.658 .987 -7.93 5.934 1 8.00* 2.658 .020 1.07 14.935 1 11.00* 2.658 .001 4.07 17.93

* The mean difference is significant at the .05 level.

38

Dunnett's comparisons with a control (Minitab)

Family error rate = 0.0500Individual error rate = 0.0152

Critical value = 2.61 Dut_1-/2

Control = level (1) of broker

Intervals for treatment mean minus control mean

Level Lower Center Upper --+---------+---------+---------+-----2 -0.930 6.000 12.930 (---------*--------) 3 -7.930 -1.000 5.930 (---------*--------) 4 1.070 8.000 14.930 (--------*---------) 5 4.070 11.000 17.930 (---------*---------) --+---------+---------+---------+----- -7.0 0.0 7.0 14.0

39

This procedure provides a subset of treatments that cannot distinguished from the best. The probability of that the “best” treatment is included in this subset is controlled at 1-.

Method 4: MCB Procedure (Compare to the best)

*Assume that larger is better.

40

STEP 1: Calculate the following for all index i

)11

(),1li

ilnn

MSEvcDM

)(max and .. jiji yy

where l (not i) is the group of which mean reaches )(max .jij y

41

STEP 2: Conduct tests

The treatment i is included in the best subset if

.)](max[ .. iljijii MyyD

42

Drug A Drug B Placebo

Index i 1 2 3

Column mean -5.3 -6.1 -12.3

-6.1 -5.3 -5.3.max jij y

Di 0.8 -0.8 -7

894.0)10

1

10

1(12

)10

1

10

1()27,2(%5

MSEDMil

What drugs are in the best subset?

(Given MSE = 1.)

43

Identify the subset of the best brokers

Hsu's MCB (Multiple Comparisons with the Best)

Family error rate = 0.0500

Critical value = 2.27

Intervals for level mean minus largest of other level means

Level Lower Center Upper ---+---------+---------+---------+----1 -17.046 -11.000 0.000 (------*-------------) 2 -11.046 -5.000 1.046 (-------*------) 3 -18.046 -12.000 0.000 (-------*--------------) 4 -9.046 -3.000 3.046 (------*-------) 5 -3.046 3.000 9.046 (-------*------) ---+---------+---------+---------+---- -16.0 -8.0 0.0 8.0

Brokers 2, 4, 5

44

----Post Hoc comparisons*F test for contrast (in “Orthogonality”)

*Scheffe test (p.108; skipped)

To test all linear combinations at once. Very conservative; not to be used for pairwise

comparisons.

----A Priori comparisons* covered later in chapter on

“Orthogonality”