MANOVA: Part 2 Proﬁle Analysis and 2-way MANOVA · MANOVA: Part 2 Proﬁle Analysis and 2-way MANOVA Edps/Soc 584, Psych 594 Carolyn J. Anderson Department of Educational Psychology

MANOVA: Part 2

Profile Analysis and 2-way MANOVAEdps/Soc 584, Psych 594

Carolyn J. Anderson

Department of Educational Psychology

I L L I N O I Suniversity of illinois at urbana-champaign

c© Board of Trustees, University of Illinois

Spring 2017

Profile Analysis Two or More Groups MANOVA Example: Practice Validity of Assumptions Unbalanced Designs Conclusions

Outline

Extensions of 1–Way MANOVA:

◮ Profile Analysis◮ Two groups◮ Two or more groups

◮ 2–way MANOVA◮ Review 2-way ANOVA◮ 2-way MANOVA◮ Example

◮ Validity of Assumptions

◮ Unbalanced Designs

Reading: Johnson & Wichern pages 312-328.

C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 2.1/ 82


Profile Analysis

Profile analysis is an extension of 1–way MANOVA involving p

response variables administered to g groups of individuals or cases.

e.g., A battery of personality tests (sub-scales) where the p testscores are measured in the same units (or at least similar orcommensurate units).

Profile Analysis gives you more specific hypotheses to test than thestandard

Ho : µ1 = µ2 = · · · = µg .

Consider p × 1 mean vector for each group

µ′l = (µl1, µl2, . . . , µlp)

and plot the means as “profiles” for each group.



Example of Profiles for Two Groups

Variable

One Two Three . . .p

MeanValue

5

10

15

q

q

q

q Group 1q

q

q

q

Group 2



Profile Analysis: 2 GroupsThe usual null hypothesis in a 1-way MANOVA is

Ho : µ1 = µ2

In profile analysis, we break this down into three sequential tests.

◮ Question 1: Are the profiles parallel?Are successive (adjacent) differences between means equal?If the answer is ”no”, then stop.If the answer is ”yes”, then go onto the next question/step.

◮ Question 2: Are the profiles the same?Are the population means equal for the two groups?If the answer is ”no”, then stop.If the answer is ”yes”, then go onto the next question/step.

◮ Question 3: Are the profiles flat?Are the population means equal over the variables?



Question 1: Parallel Profiles

Variables

1 2 3 4 5

MeanValue

Group 1

r

r

r

r

r

µ11

µ12

Group 2

r

r

r

r

r

µ21

µ22

Ho1 : µ1i − µ1,i−1 = µ2i − µ2,i−1

for all i = 2, . . . , p



Question 2: Are Profiles Coincident?

Assuming that profiles are parallel, are they coincident?Are the population means equal for the two groups?

1 2 3 4 5

MeanValue

Group 1

r

r

r

r

r

µ11

µ12

Group 2

r

r

r

r

r

µ21 =

µ22 =

Ho2 :µ1i = µ2i for all i = 1, . . . p

or equivalently Ho2 :µ1 = µ2



Question 3

Assuming that the profiles are coincident, are they Level (flat)?Are all the means equal? (over groups and variables).

1 2 3 4 5

MeanValue

Group 1r r r r rµ1i

Group 2r r r r

rµ2i =

Ho3: µ11 = µ12 = · · · = µ1p

= µ21 = µ22 = · · · = µ2p



Testing Question 1: Parallel

Assuming X1j ∼ N (µ1,Σ) and X2j ∼ N (µ2,Σ)for j = 1, . . . , nl , and independent.

Question 1: Are the profiles parallel?

We can write the hypothesis are Ho1 : Cµ1 = Cµ2 where C(p−1)×p

contrast matrix. e.g.,

C(p−1)×p =

−1 1 0 0 · · · 0 00 −1 1 0 · · · 0 0...

......

.... . . · · · · · ·

0 0 0 0 · · · −1 1

What we’re doing is linearly transforming our original p variablesinto (p − 1) new variables.



Question 1 continuedWhen we take

CX =

−X1 + X2

−X2 + X3...

−Xp−1 + Xp

,

which are measured on both groups (populations, etc).Random variable CX1j ∼ Np−1(Cµ1,CΣC

′),& Random variable CX2j ∼ Np−1(Cµ2,CΣC

′).To estimate the covariance matrix Σ, use Spool ,

Spool(n1 − 1)S1 + (n2 − 1)S2

n1 + n2 − 2

and the estimate of CΣC′ equals CSpoolC

′.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 10.1/ 82


Finishing Question 1How to test Ho1 : Cµ1 − Cµ2 = C(µ1 − µ2) = 0?Hotelling’s T 2 for Two independent samples.

Reject Ho1 if

T 2 = (X̄1 − X̄2)′C′[(

1

n1+

1

n2

)

CSpoolC′]−1

C(X̄1 − X̄2) > c2

where

c2 =(n1 + n2 − 2)(p − 1)

n1 + n2 − pF(p−1),(n1+n2−p)(α)

◮ If you reject Ho1 −→ STOP.. . . You can do any follow-up to examine differences.

◮ If you retain Ho2 −→ Conclude profiles are parallel andPROCEED to next question.



Testing Question 2: Coincident profilesAssuming that the the profiles are parallel, are the profilescoincident?

If profiles are parallel, then one will be “above” the other for alli = 1, . . . , p; that is,

µij > µij for all i = 1, . . . , p

or

µij < µij for all i = 1, . . . , p

So, profiles will be coincident only if the total “heights” are equal

(µ11 + µ12 + · · ·+ µ1p) = (µ21 + µ22 + · · ·+ µ2p)

1′pµ1 = 1

′pµ2

where 1p is a (p × 1) vector of ones.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 12.1/ 82


Question 2 continuedThe null hypothesis for question 2 is

Ho2 : 1′µ1 = 1

′µ2

We are forming a new variable, 1′X, which is a simple sum and testwhether this variable is equal for the two groups.

1′X1j ∼ N1(1

′µ1,1′Σ1) j = 1, . . . , n1

1′X2j ∼ N1(1

′µ2,1′Σ1) j = 1, . . . , n2

We estimate

1′µ1 by 1

′x̄1

1′µ2 by 1

′x̄2

and

1′Σ1 by 1

′Spool1



Finishing Up Question 2To test

Ho2: 1′µ1 = 1

′µ2,

we can do a simple univariate 2 independent sample t-test.

We will reject Ho2 at the α-level if

t =1′(x̄1 − x̄2)

√(

1n1

+ 1n2

)

1′Spool1

> tn1+n2−2(α)

◮ If Reject Ho2 −→ STOP and conclude that the profiles areparallel but not coincident.

◮ If Retain Ho2 −→ PROCEED and test whether profiles areflat.



Question 3: FlatIf the profiles are coincident, do they all have the same mean? i.e.,

Ho3 : µ11 = µ12 = · · · = µ1p = µ21 = µ22 = · · · = µ2p

We can test this using a contrast matrix C, such as the one weused for Question 1 (testing parallel profiles)

Ho3 : C(µ1+µ2) = 0

Note that we are adding rather than subtracting as we did in Ho1.This tests

Cµ1 + Cµ2 = 0

(µ12 − µ11) + (µ22 − µ21)(µ13 − µ12) + (µ23 − µ22)

...(µ1p − µ1,p−1) + (µ2p − µ2,p−1)

= 0



Question 3 continuedTesting

Cµ1 + Cµ2 = 0

(µ12 − µ11) + (µ22 − µ21)(µ13 − µ12) + (µ23 − µ22)

...(µ1p − µ1,p−1) + (µ2p − µ2,p−1)

= 0

Is the same as this, which is what we want to test,

(µ12 + µ22) = (µ11 + µ21)

(µ13 + µ23) = (µ12 + µ22)

......

(µ1p + µ2p) = (µ1,p−1 + µ2,p−1)

Because If the profiles are coincident, then µ1i = µ2i for alli = 1, . . . , p. So if Ho3 is true, then C(µ1 + µ2) = 0.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 16.1/ 82


Question 3 continuedFor Ho3 : C(µ1 + µ2) = 0, flat profiles, we are taking linearcombinations in two ways simultaneously

◮ Over variables via the use of C.

◮ Over groups via an additive function.

The mean C(µ1 + µ2) is estimated by taking the grand mean

X̄ =

∑n1j=1 X1j +

∑n2j=1X2j

n1 + n2=

n1X̄1 + n2X̄2

n1 + n2

and the distribution is of CX̄ is

CX̄ ∼ Np−1

(

C(µ1 + µ2),CΣC′(

1

n1 + n2

))

How do we get this?C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 17.1/ 82


Finishing up Question 3

Reject Ho3 : C(µ1 + µ2) = 0 (profiles are flat) at α-level if

(n1+n2)X̄′C′(CSC)−1

CX̄ >(n1 + n2 − 1)(p − 1)

n1 + n2 − p + 1F(p−1),(n1+n2−p+1(α)

where S is the total sample covariance matrix

S =1

n1 + n2 − 1

n1∑

j=1

(X1j − X̄)(X1j − X̄)′ +n2∑

j=1

(X2j − X̄)(X2j − X̄)′

Note that we use the total sample mean in computing S.



Example: WAIS dataThis example is from Morrison (2005): 49 elderly men in a studyof human aging were classified into the diagnostic categories“senile factor present” and “no senile factor” on the basis of anintensive psychiatric examination. The Wechsler Adult IntelligenceScale (WAIS) was administered to all subjects by an independentinvestigator.

Mean scores by group on some of the WAIS subtests.Not Senile (n = 37) Senile (n = 12)

Sub-Test x̄ std dev x̄ std dev

Information 12.566 3.387 8.750 3.251Similarities 9.486 3.380 5.333 4.271Arithmetic 11.514 3.363 8.500 3.631Picture 7.973 1.922 4.750 3.571

Note: My results differ slightly from Morrison’s. There is either a typo in the text or in the data set. (no way to

find out which).C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 19.1/ 82


WAIS Profiles

WAIS Sub-Tests

Information

Similarities

Arithmetic

Picture completion

Sub-testGroupMean

5

10

15

q

q

q

q No Senile Factorq

q

q

q

Senile Factor



WAIS: Are Profiles Parallel?The within group (pooled) sample covariance matrix:

Spool =1

n1 + n2 − 2((n1 − 1)S1 + (n2 − 1)S2)

=

11.262 8.995 7.164 3.3798.995 13.019 7.037 2.3087.164 7.037 11.750 2.6393.379 2.308 2.639 5.813

Test for parallel profiles: The hypothesis isHo : µ1i − µ1,i−1 = µ2,i − µ2,i−1 for i = 2, 3, 4; that is, thefollowing three qualities simultaneously hold in the population,

µ12 − µ11 = µ22 − µ21

µ13 − µ12 = µ23 − µ22

µ14 − µ13 = µ24 − µ23C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 21.1/ 82


WAIS: Are Profiles Parallel?To test this hypothesis, we need a (3× 4) contrast matrix C

C =

−1 1 0 00 −1 1 00 0 −1 1

Our hypothesis can now be,expressed as Ho : Cµ1 = Cµ2, orHo : C(µ1 − µ2) = 0. This hypothesis is tested by Hotelling’s T 2,and we need

CSpoolC′ =

6.291 −4.150 −0.944−4.151 10.694 −4.382−0.944 −4.382 12.286

,

(x̄1 − x̄2)′ = (3.818, 4.153, 3.014, 3.223),

and(C(x̄1 − x̄2))

′ = (0.336,−1.140, 0.209)C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 22.1/ 82


WAIS: Are Profiles Parallel?Putting all of this into our equation for T 2 gives us

T 2 =n1n2

n1 + n2(x̄1 − x̄2)

′C′(CSpoolC

′)−1C(x̄1 − x̄2)

= 1.224

and compare the following statistic to the F distribution,

F =n1 + n2 − p

(n1 + n2 − 2)(p − 1)T 2 =

45

(47)(3)1.224 = 0.391

Since F3,45(α = .05) = 2.8115 is greater than our observedstatistic (or the p-value of F equals .76), we do not reject thehypothesis that the profiles are parallel.

Since we Retained the null, we will PROCEED to test whether theprofiles are coincident.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 23.1/ 82


WAIS: Are Profiles Coincident?Test for Equal (coincident) Profiles.

We now test Ho : µ1i = µ2i for i = 1, 2, 3, 4 variables. Since weconcluded that the profiles are parallel, then we can test thishypothesis by testing whether

(µ11 + µ12 + µ13 + µ14) = (µ21 + µ22 + µ23 + µ24)

1′µ1 = 1

′µ2

where 1 is a (4× 1) vector of ones. This is just a univariate, 2independent sample t.To test Ho : 1′(µ1 − µ2) = 0, we need for each group the sums ofthe means over all variables, which are

1′x̄1 = 41.541 and 1

′x̄2 = 27.333

Estimate of the variance of the difference = 1′Spool1 = 104.891.



WAIS: Are Profiles Coincident?

Putting all of these statistics together for our test statistics gives us

t =41.541 − 27.333

√

104.890(

112 + 1

37

) = 4.176

Since t47(α = .025) = 2.012 (or the p-value of t47 = 4.15 is< .001), we reject the null hypothesis. The profiles are notcoincident.

At this point we STOP and should not go on to test whetherprofiles are level (flat).



Profiles Analysis for g ≥ 2 GroupsTo deal with two or more groups, we follow the same logic:

1. Using assumptions (including results from earlier tests),determine what the hypothesis implies for populationparameters. This has implications for data.

2. Compute a statistic that reflects the implications for data.3. Find a transformation of the statistic with a known sampling

distribution.

It is easier to do profile analysis using the GLM framework. We willuse

X = AB + ǫ,

where

◮ X is (n × p) data matrix.◮ A is the (n × g) design matrix where g = number groups.◮ B is the (g × p) parameter matrix.◮ ǫ is the (n × p) residual matrix.

This is a multivariate regression model.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 26.1/ 82


Two or More Groups

Our multivariate regression model is

X = AB + ǫ,

Test hypotheses of the form

Ho : LBM = 0

where

◮ L is (g× number of tests between groups).

◮ M is (p× number of tests over variables).

Appropriate definitions of L and M will lead to tests forparallelism, coincidence, and level means.



Two or More: ParallelParallel profiles imply

Ho1 : (µ1i − µ1,i−1) = (µ2i − µ2,i−1) = · · · = (µgi − µg ,i−1)

for all i = 2, . . . , p. This will be a MANOVA on differences.This hypothesis takes linear combinations of the means withingroups and then compare the groups. For the linear transformationsof the regression parameters (i.e., the τ or µ’s), define

Mp×(p−1) =

−1 0 · · · 01 −1 · · · 00 1 · · · 0...

.... . .

...0 0 · · · −10 0 · · · 1

For the comparisons of groups on these “new” variables, defineL = C(g−1)×g contrast matrix.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 28.1/ 82


GLM: Parallel and WAISFor the WAIS example, Ho1 is

LBM = (0, 1,−1)

β01 β02 β03 β04β11 β12 β13 β14β21 β22 β23 β24

−1 0 01 −1 00 1 −10 0 1

= 0

= (0, 1,−1)

(β02 − β01) (β03 − β02) (β04 − β03)(β12 − β11) (β13 − β12) (β14 − β13)(β22 − β21) (β23 − β22) (β24 − β23)

= 0

= [(β12 − β11)− (β22 − β21), (β13 − β12)− (β23 − β22),

(β14 − β13)− (β24 − β23)] = (0, 0, 0)

So we have

(β12 − β11)− (β22 − β21)(β13 − β12)− (β23 − β22)(β14 − β13)− (β24 − β23)

=⇒

(µ12 − µ11)− (µ22 − µ21)(µ13 − µ12)− (µ23 − µ22)(µ14 − µ13)− (µ24 − µ23)

=

000



Two or More: Coincident?Assuming parallel, then coincidence implies

Ho2 : 1′µ1 = 1′µ2 = · · · = 1

′µg

So we’re testing equivalence of a single variable over g groups:1-way ANOVA.The linear transformation of the variables is given by

Mp×1 =

11...1

For the groups,L = I

See SAS for WAIS exampleC.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 30.1/ 82


Two or More: Level ProfilesIf profiles are coincidence, then flat or level profiles means

Ho3 : µ11 = µ12 = · · · = µ1p = µ21 = · · · = µgp

That is, sums of two variables are the same for pairs of groups.The required linear transformation,

M(p×(p−1) =

1 0 0 · · · 01 1 0 · · · 00 1 1 · · · 0...

......

. . ....

0 0 0 · · · 1

The required contrasts of the groups,

L = C(g−1)×g

which is the same one used for testing parallel profiles.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 31.1/ 82


2-way MANOVA

Mini Outline:

1. Review 2–way ANOVA

2. 2–way MANOVA for balanced data

3. Example: Distributed versus Massed Practice/instruction

4. Unbalanced designs.

5. Multivariate GLM and further extensions (MANCOVA,longitudinal)



Two-Way ANOVA“2–way” ANOVA −→ 2 Factors (qualitative variables). Notation:

Factor B1 · · · k · · · b

1 X11r · · · X1kr · · · X1br ←− n observations in...

......

... each of the gb

Factor A l Xl1r · · · Xlkr · · · Xlbr combinations of levels...

......

... of the factors.g Xgkr · · · Xgkr · · · Xgbr

◮ Observation: Xlkr = the r th observation at level l of Factor A andlevel k of Factor B.

◮ Factor A: l = 1, . . . , g

◮ Factor B: k = 1, . . . , b

◮ Replications: r = 1, . . . , n For now, a balanced design.



ANOVA Model for Two FactorsXlkr = µ

︸︷︷︸

overalllevel

+ τl︸︷︷︸

fixed effectfactor Aat level l

+ βk︸︷︷︸

fixed effectfactor Bat level k

+ γlk︸︷︷︸

interactionbetween

factors A &Bat levelsl & , k

+ ǫlkr︸︷︷︸

residual

where

g∑

l=1

τg =b∑

k=1

βk +

g∑

l=1

γlk =b∑

k=1

γlk = 0

and ǫlkr ∼ N (0, σ2) and all independent.Model for the mean:

E (Xlkr ) = µlk = µ+ τl + βk + γlk

TThe effects are not additive; there is an interaction.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 34.1/ 82


Examples: Interaction & No Interaction

Factor A

a1 a2 a3 a4

CellMean

5

10

15Factor B

p

p

p

p b1p

p

p

p b2

p

p

p

p b3

Factor A

a1 a2 a3 a4

5

10

15

Factor B

p

p

p

p b1p

p

p

p b2

p

p

p

p b3



Observation as Sum of Means

xlkr = x̄︸︷︷︸

µ̂

+(x̄l �� − x̄)︸︷︷︸

τ̂l

+(x̄�k� − x̄)︸︷︷︸

β̂k

+(x̄lk� − x̄l �� − x̄�k� + x̄)︸︷︷︸

γ̂lk

+(xlkr − x̄lk�)︸︷︷︸

ǫ̂lkr

where

◮ x̄ = overall sample mean.

◮ x̄l �� = mean for l th level of Factor A (or “row” mean).

◮ x̄�k� = mean for k th level of Factor B (or “column” mean).

◮ x̄lk� = mean for the l th level of Factor A and k th level ofFactor B (or the “cell” mean).

◮ The decomposition of the observations above shows theestimates of the population effects.



Decomposition of Sums of Squares

xlkr = x̄︸︷︷︸

µ̂

+(x̄l �� − x̄)︸︷︷︸

τ̂l

+(x̄�k� − x̄)︸︷︷︸

β̂k

+(x̄lk� − x̄l �� − x̄�k� + x̄)︸︷︷︸

γ̂lk

+(xlkr − x̄lk�)︸︷︷︸

ǫ̂lkr

To get the sums of squares:

1. Subtract x̄ from both sides.

2. Square both sides.

3. Sum over observations within cells, levels of Factor A, andlevels of Factor B.

After a little algebra,



Decomposition of Sums of Squares

The sum of squares decomposition isg∑

l=1

b∑

k=1

n∑

r=1

(xlkr − x̄)2 =

g∑

l=1

bn(x̄l �� − x̄)2 +

b∑

k=1

gn(x̄�k� − x̄)2

+

g∑

l=1

b∑

k=1

n(x̄lk� − x̄l �� − x̄�k� + x̄)2

+

g∑

l=1

b∑

k=1

n∑

r=1

(xlkr − x̄lk�)2



Sums of Squares Decomposition

We get an orthogonal decomposition of the sums of squares:

SStotal = SSA + SSB + SSAB + SSresidual

Note: SStotal is corrected for the grand mean.

Also, we get decomposition of degrees of freedom,

(gbn − 1)︸︷︷︸

total

= (g − 1)︸︷︷︸

Factor A

+ (b − 1)︸︷︷︸

Factor B

+ (g − 1)(b − 1)︸︷︷︸

Interaction

+ gb(n − 1)︸︷︷︸

Residual



ANOVA Summary Table

Source of Degrees SSvariation of freedom Sums of SquaredFactor A (g − 1) SSA =

∑gl=1 n(x̄l�� − x̄)2

Factor B (b − 1) SSB =∑b

k=1 gn(x̄�k� − x̄)2

Interaction (g − 1)(b − 1) SSAB =∑g

l=1

∑b

k=1 n(x̄lk� − x̄l�� − x̄�k� + x̄)2

Residual gb(n− 1) SSres =∑g

l=1

∑bk=1

∑nr=1(xlkr − x̄lk�)

2

(error)

Total gbn− 1 SStotal =∑g

l=1

∑bk=1

∑nr=1(xlkr − x̄)2



Hypothesis Tests◮ Interaction: Ho : γ11 = γ12 = · · · = γgb = 0. If the null is true

and all assumptions valid,

F =SSAB/((g − 1)(b − 1))

SSres/(gb(n − 1))∼ F(g−1)(b−1),gb(n−1)

◮ Factor A: Ho : τ1 = τ2 = . . . = τg . If the null is true and allassumptions valid,

F =SSA/(g − 1)

SSres/(gb(n − 1))∼ F(g−1),gb(n−1)

◮ Factor B: Ho : β1 = β2 = · · · = βb. If the null is true and allassumptions valid,

=SSB/(b − 1)

SSres/(gb(n − 1))∼ F(b−1),gb(n−1)

Start with the Interaction before the main effects.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 41.1/ 82


Two-Way MANOVAFor p variables, Xlkr is a p × 1 vector of measurements on p

variables.

The modelXlkr = µ+ τ l + βk + γ lk + ǫlkr

and ǫlkr ∼ Np(0,Σ) and independent.

◮ l = 1, . . . , g

◮ k = 1, . . . , b

◮ r = 1, . . . , nlk◮ And for identification

g∑

l=1

τ l =

b∑

k=1

βk =

g∑

l=1

γ lk =

b∑

k=1

γ lk = 0



Decomposition of ObservationWe decompose the observation vector (xlkr − x̄) into sums ofvarious vectors,

(xlkr − x̄) = (x̄l �� − x̄)︸︷︷︸

τ̂ l

+(x̄�k� − x̄)︸︷︷︸

ˆβk

+(x̄lk� − x̄l �� − x̄�k� + x̄)︸︷︷︸

γ̂ lk

+(xlkr − x̄lk�)︸︷︷︸

ǫ̂lkr

If we take the sums of squares and cross-products of this, weobtain the SSCP decomposition,

∑

l

∑

k

∑

r

(x̄l �� − x̄)(x̄l �� − x̄)′

︸︷︷︸

SSCPtotal

= . . .

which will take more space than is left on this slide.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 43.1/ 82


Decomposition of SSCP∑

l

∑

k

∑

r

(x̄l �� − x̄)(x̄l �� − x̄)′

︸︷︷︸

SSCPtotal

=∑

l

bn(x̄l �� − x̄)(x̄l �� − x̄)′

︸︷︷︸

SSCPFactorA

+∑

k

gn(x̄�k� − x̄)(x̄�k� − x̄)′

︸︷︷︸

SSCPFactorB

+∑

l

∑

k

n(x̄lk� − x̄l �� − x̄�k� + x̄)(x̄lk� − x̄l �� − x̄�k� + x̄)′

︸︷︷︸

SSCPinteraction

+∑

l

∑

k

∑

r

(xlkr − x̄lk�)(xlkr − x̄lk�)′

︸︷︷︸

SSCPresidual



Decomposition of SSCP

So,

SSCPtotal = SSCPFactorA+SSCPFactorB+SSCPInteraction+SSCPresidual

which is orthogonal so long as the design is balanced (i.e., nlk = n)or proportional. . . what to do with unbalanced is discussed a bitlater.

And

dftotal = dfFactorA + dfFactorB + dfInteraction + dfresidual

(gbn − 1) = (g − 1) + (b − 1) + (g − 1)(b − 1) + gb(n − 1)



Hypothesis Testing: Interaction FirstHo : γ11 = γ12 = · · · = γgp

◮ Test Statistic is Wilk’s Lambda

Λ∗ =det(SSCPresidual )

det(SSCPresidual + SSCPinteraction)=

det(E)

det(E+H)

◮ νh = νinteraction = (g − 1)(n − 1)◮ νe = νresidual + gb(n − 1)◮ Distribution of Λ∗ are given on the following slide for various

cases.◮ If you don’t have p ≤ 2 or νh ≤ 2, but you have large n then

the approximate sampling distribution of

−(

νe −p + 1− νh

2

)

ln(Λ∗) is χ2pνh



Distribution of Wilk’s Lambda Λ∗

Wilk’s Λ∗ =|SSCPe |

|SSCPe + SSCPh|Number df forvariables Hypothesis Sampling distribution for multivariate data

p = 1 νh ≥ 1(νeνh

) (1−Λ∗

Λ∗

)∼ Fνh,νe

p = 2 νh ≥ 1(νe−1νh

)(1−

√Λ∗√

Λ∗

)

∼ F2νh,2(νe−1)

p ≥ 1 νh = 1(νe+νh−p

p

) (1−Λ∗

Λ∗

)∼ Fp,(νe+νh−p)

p ≥ 2 νh = 2(νe+νh−p−1

p

)(1−

√Λ∗√

Λ∗

)

∼ F2p,2(νe+νh−p−1)

where νh = degrees of freedom for hypothesis, andνe = degrees of freedom for error (residual).



Hypothesis Testing: Interaction FirstYou should test the interaction first.

◮ If Ho for the interaction is rejected, then you don’t so tests formain effects because the Factor main effects do not have aclear interpretation.

◮ J&W recommend doing p-univariate 2-way ANOVAs to findout “where” the interactions exist. (i.e., are there interactionsfor just some of the p variables and not others?).

◮ For variables with no interaction, you can interpret maineffects.

◮ Note: In some cases you can go on to test main effects eventhough the interaction is significant (we’ll do this in ourexample).

◮ In general, if you retain the null hypothesis for interaction, wecontinue and test main effects. . .



Hypothesis Testing: Main Effects

Factor A: Ho : τ 1 = τ 2 = · · · = τ g = 0 versus Ha : at least one τ l 6= 0

Λ∗A =

det(SSCPresidual )

det(SSCPresidual + SSCPFactorA)=

|E ||E + HA|

and νh = g − 1. For p ≤ 2 or νA ≤ 2 (i.e., g ≤ 3), use the exactdistribution given in the table.

Factor B: Ho : β1 = β2 = · · · = βb = 0 versus Ha : at least one βk 6= 0

Λ∗B =

det(SSCPresidual )

det(SSCPresidual + SSCPFactorB)=

|E ||E + HB |

and νh = b − 1. For p ≤ 2 or νB ≤ 2 (i.e., b ≤ 3), use the exactdistribution given in the table.

Large n: −(

νe −p + 1− νh

2

)

ln(Λ∗h) ≈ χ2

pνh



Following RejectionIf a null hypothesis is rejected, you should perform additionalanalyses to figure out where the effects are. These could include

◮ Multivariate T 2; that is, generalized squared distances

D2(X)lk|l ′k′ = (X̄lk − X̄l ′k′)′S−1pool (X̄lk − X̄l ′k′)

◮ Simultaneous confidence statements/intervals.◮ When interactions are not significant, concentrate on contrasts

of levels of Factor A and of Factor B (if they’re significant).These are the same as those described in 1-way MANOVAnotes.

For example,(τli − τmi ) or (µli − µmi )

Use Bonferroni if planned.◮ 2-way ANOVAs with multiple comparisons.◮ Discriminant Analysis.



Example: Distributed vs Massed Practice1–Way MANOVA: Data from Tatsuoka (1988), Multivariate

Analysis: Techniques for Educational and Psychological Research,pp 273–279.

An experiment was conducted for comparing 2 methods (A & B) ofteaching word processing to 60 female seniors in high school. Alsoof interest were the effects of distributed versus massed practiceC1: 2 hours of instruction/day for 6 weeksC2: 3 hours of instruction/day for 4 weeksC3: 4 hours of instruction/day for 3 weeks

So each subject received a total of 12 hours of instruction. Note:nl = 10 per cell of the design

Two variables (dependent measures):X1 = speed, and X2 = accuracy



The Design and Data

Various Means:C1 C2 C3

X1 X2 X1 X2 X1 X2 X1 X2

A 34.30 21.80 32.50 17.90 29.60 14.10 32.13 17.93B 42.80 25.60 35.50 18.59 27.00 11.60 35.10 18.57

38.55 23.79 34.00 18.20 28.30 12.85 33.62 18.25Notes:

◮ Cell means (black) based on nlk = 10.

◮ Means for method (green) are based on nl � = 30

◮ Means for condition (blue) are based on n�k = 20.

◮ Total sample means (red) are based on n�� = 60.



Cell Means & 95% Confidence Ellipses

5 10 15 20 25 30 35 40

X2(A

ccuracy)

5

10

15

20

25

30

35

40

r

x̄AC1

r

x̄AC2

r

x̄AC3

rx̄BC1

r

x̄BC2

r

x̄BC3



Factor Means & 95% Confidence Ellipses

5 10 15 20 25 30 35 40

X2(A

ccuracy)

5

10

15

20

25

30

35

40

b

x̄A b x̄B

b x̄C1

b

x̄C2b

x̄C3



Test for Interaction

Ho : γAC1= γAC2

= γAC3= γBC1

= γBC2= γBC3

= 0

Ha : not all γ lk = 0

W = E =

(1333.10 211.20211.20 184.70

)

HAB =

(308.04 174.82174.82 99.23

)

Λ∗ =det(E)

det(E+HAB)=

201.618

316.957= .6361

and

F =

(53

2

)(

1−√.6361√

.6361

)

= 6.73

Comparing F = 6.73 to the F4,106 distribution, F has p < .001.



Test for Method Main Effect. . . For purposes of illustration. . .

Ho : τA = τB = 0 versus Ha : not all γ l = 0

W = E =

(1333.10 211.20211.20 184.70

)

HMethod =

(132.02 28.1828.18 6.02

)

Λ∗ =det(E)

det(E+HMethod )=

201.618

222.13= .9077

and

F =

(53

1

)(

1−√.9077√

.9077

)

= 2.63

Comparing F = 2.63 to the F2,106 distribution, F has p = .08.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 56.1/ 82


Test for Condition Main Effect

Ho : βC1= βC2

= βC3= 0 versus Ha : not all βk = 0

W = E =

(1333.10 211.20211.20 184.70

)

HCondition =

(1055.03 1111.551111.55 1177.30

)

Λ∗ =det(E)

det(E+HCondition)=

201.618

1502.966= .1341

and

F =

(53

2

)(

1−√.1341√

.1341

)

= 45.85

Comparing F = 45.85 to the F4,106 distribution, F has p < .001.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 57.1/ 82


Interpretation & Conclusion

◮ The effectiveness of the teaching methods (A and B) dependson which of the three conditions of distributed practice that astudent used.

(i.e., There is a significant interaction).

◮ The method by condition cell means indicate that method Bis more effective in terms of both speed and accuracy underconditions C1 and C2, but A is more effective in terms ofspeed and accuracy under condition C3.

◮ Statistical tests of these conclusions should be performed infollow up analyses. Potentially useful supplement analysesinclude ANOVAs for each of the dependent variables,simultaneous confidence intervals for various treatmenteffects, and discriminant analysis.



Validity of Assumptions

The assumptions are

◮ Multivariate normality.

◮ Equality of Σ’s.

◮ Independence between and within groups.

Checks on multivariate normality:

◮ Scatter plots of residuals.

◮ QQ plots of residuals.

◮ Scatter & QQ plots of principal components.

◮ Tests of univariate normality for each residuals.

◮ Tests of multivariate normality (see text).



Shorthand Example: Scatter Plot



Shorthand Example: QQ Plot 1



Shorthand Example: QQ Plot 2



Equality of Covariance Matrices◮ When there are equal nlk ’s (balanced), violating equality of

Σlk ’s probably doesn’t hurt too much.◮ When there are unequal nlk ’s and |Σlk |’s differ substantially,

then will tend to make more errors.

Where you make more Type I or Type II errors depends on howdifferent the nlk ’s and |Σlk |’s are from each other.

◮ Test of Equality of Σ’s — this gets into ideas of tests andinference for covariance matrices, which leads into topics oncanonical correlation and discriminate analysis.

Assume:

Xlj ∼ Np(µl ,Σl ) for l = 1, . . . , g and j = 1, . . . , nl

and independent over the g groups and with groups.The null hypothesis:

Ho : Σ1 = Σ2 = · · · = Σg ≡ Σ



Testing Equality of Covariance MatricesIf Ho : Σ1 = Σ2 = · · · = Σg ≡ Σ is true, then

Spool = S =

(1

∑gl=1 νl

) g∑

l=1

νl ln(|Si |)

where νl = (ni − 1) = df (Si ). When Ho is true, then Spool is anunbiased estimate of Σ.

Box’s test (1949ish):

M =

(g∑

l=1

νl

)

ln(| S︸︷︷︸

Spool

|)−g∑

l=1

νl ln(|Sl |)

Test statistic is asymptotically

[

1− 2p2 + 3p − 1

6(p + 1)(g − 1)

(g∑

l=1

(1

νl

)

− 1∑g

l=1 νl

)]

︸︷︷︸

Correction factor

M ≈ χ2(g−1)p(p+1)/2



Example: Box’s Test of Equality of Σlk

Ho : ΣA,C1= ΣA,C2

= ΣA,C3= ΣB,C1

= ΣB,C2= ΣB,C3

versusHa : not all Σlk are equal.The method by condition sample covariance matrices:

SA,C1=

(34.68 8.408.40 5.07

)

SB,C1=

(29.73 1.471.47 2.93

)

SA,C2=

(38.50 4.174.17 7.43

)

SB,C2=

(14.50 3.723.72 1.83

)

SA,C3=

(19.16 3.783.78 1.66

)

SB,C3=

(11.56 2.332.33 1.60

)

Pooled Covariance Matrix:

Spool =

(25.69 3.913.91 3.42

)



Example of Box’s Test continued

M = (∑

l ,k

(nlk − 1)) ln |Spool | −∑

l ,k

(nlk − 1)|Slk |

= 6(10 − 1) ln(69.14) − (10 − 1)(ln(105.14 + ln(268.82) + ln(20.30)

+ ln(85.07) + ln(12.73) + ln(13.04))

= 228.75 − 205.34

= 23.41

Since nlk = 10 for all K = (2)(3) = 6 combinations of method andcondition (i.e., equal cell sizes), the correction factor simplifies to

1− (2p2 + 3p − 1)(g + 1)

6(p + 1)gn= 1− (2(2)2 + 3(2) − 1)(6 + 1)

6(2 + 1)10(6)= .9157

Test statistic = .9158(23.41) = 21.44 withdf = (K − 1)p(p + 1)/2 = (6− 1)2(2 + 1)/2 = 15, which using theχ2 distribution has p-value = .123. Retain null hypothesis; that is,the equal covariance matrix assumption is reasonable.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 66.1/ 82


Box’s Test— the easier method

data shorthand;

input speed accuracy method $ conditin $;if conditin=’C1’ and method=’A’ then group=1;

else if conditin=’C1’ and method=’B’ then group=2;

else if conditin=’C2’ and method=’A’ then group=3;


else if conditin=’C3’ and method=’A’ then group=5;


proc discrim simple pool=test Wcov Pcov list;

class group;

var speed accuracy;

. . . and the output. . .



PROC DISCRIM outputNotation:

K = Number of Groups

P = Number of Variables

N = Total Number of Observations - Number of Groups

N(i) = Number of Observations in the i’th Group - 1

N(i)/2

Π |Within SS Matrix(i)|

V = -----------------------------------

N/2

|Pooled SS Matrix|



PROC DISCRIM output2

| 1 1 | 2P + 3P - 1

RHO = 1.0 - | SUM ----- - --- | -------------

| N(i) N | 6(P+1)(K-1)

DF = .5(K-1)P(P+1)

| PN/2 |

| N V |

Under the null hypothesis: -2 RHO ln | ---------------- |

| PN(i)/2 |

| Π N(i) |

is distributed approximately as Chi-Square(DF).

Chi-Square DF Pr > ChiSq

21.218963 15 0.1300



PROC DISCRIM output

Since the Chi-Square value is not significant at the 0.1

level, a pooled covariance matrix will be used in the

discriminant function. Reference: Morrison, D.F. (1976)

Multivariate Statistical Methods p252

There is a more recent version of Morrison.



Checking Independence Assumption

◮ MANOVA is generally relatively robust to violations ofmultivariate normality and equal covariance matrices;however, it is not robust to violations of

◮ Independence of observations within groups.◮ Independence of observations between groups.

◮ Violation of independence generally increases the Type I errorrate (i.e., it’s higher than what you think it is, and possiblymuch larger).

◮ You can check for possible violation by computing theintra-class correlation for each of the variables using

rintra =MSbetween −MSerror

MSbetween + (n − 1)MSerror



Checking Independence Assumption (continued)

◮ You can also consider the data collection procedure/method,because this is where dependence can slip in.

◮ It’s the independence within that is most likely to be violated.In some cases this can be dealt with.



Unbalanced DesignsSimilar to 1-way ANOVA.

◮ With balanced designs (or proportional)

SSCPtotal = SSCPA + SSCPB + SSCPAB + SSCPresidual

orSSCPmodel = SSCPA + SSCPB + SSCPAB

and interpretation is straight forward.◮ With unbalanced designs, the partitioning of the is not longer

unique. It depends on◮ The model.◮ Various sub-models of model as specified by the order in which

various SSCP are extracted.. . .

These notes are based on Khattree & Naik.C.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 73.1/ 82


Order in which SSCP are extractedSuppose

Xlkr = µ+ τ l + βk + γ lk + ǫlkr

where l = 1, . . . , g , k = 1, . . . , b, and r = 1, . . . , nlk .

We could partition total SSCP as

SSCPtotal corrected = SSCPA|µ + SSCPB|µ,A + SSCPAB|µ,A,B

or

SSCPtotal corrected = SSCPB|µ + SSCPA|µ,B + SSCPAB|µ,A,B

When the design is unbalanced, SSCPA|µ 6= SSCPA|µ,BThere are 4 different ways of computing SSCP’s, which can lead todifferent results and different interpretation.



Type I SSCP: “Sequential SSCP”The partitioning of the model SSCP into component SSCP due toeach variable or effect (including interactions) as it’s addedsequentially to the model as given in the model statement in procglm in SAS.e.g.,

model x1 x2 = A B C A ∗ B A ∗ C B ∗ C A ∗ B ∗ C

That is all main effects, 2–way interactions and a 3-way interaction.

The SSCP for A ∗ C = SSCP(A ∗ C |µ,A,B ,C ,A ∗ B); that is,SSCP is adjusted for all previously entered terms,

model x1 x2 = A B C A ∗ B︸︷︷︸

already entered

A ∗ C B ∗ C A ∗ B ∗ C



Example Type I SSCP: “Sequential SSCP”To demonstrate this, I deleted some observations from theshorthand data set to make it a bit unbalanced. The sample sizesare now

Practice ConditionMethod C1 C2 C3 Total

A 5 10 8 23B 10 7 6 23

Total 15 17 14 46

Below are the hypothesis SSCP matrices for two different orders:

Order: M C C ∗M C M C ∗M

Hm =

(164.54 83.2283.22 42.09

)

Hm =

(43.09 0.720.72 0.01

)



Type I: “Sequential SSCP” continued

◮ The sum of Hm +Hc will be equivalent regardless of whethermethod order condition was entered first.

◮ With Type I SSCP, the sum of SSCP matrices of all the effectsin the model will sum to the total for the model, regardless ofthe order in which they are entered into the model.

◮ The total SSCP is completely partitioned.



Type II or “Partial SSCP”

With Type II, an SSCP matrix equals the increase in model SSCPfor a particular variable (effect). e.g.,

model x1 x2 = A B C A ∗ B A ∗ C B ∗ C A ∗ B ∗ C

That is all main effects, 2–way interactions and a 3-way interaction.

Consider the SSCP for A ∗ C . The SSCP for A ∗ C is the increasein model SSCP by adding A ∗ C to the model that has A, B , A ∗ Cand B ∗ C in it.

The three-way interaction A ∗ B ∗ C∗ contains A ∗ C in it, so theSSCP for A ∗ C is “in the model”. The effects A ∗ B ∗ C and A ∗ Care correlated.

With Type II, the sum of the SSCP’s do not equal the total modelSSCP.



Example: Type II or “Partial SSCP”

Below are the SSCP matrices for Type I and Type II and the totals:

Type I: Sequential Type II: PartialM C C ∗M C M C ∗M

Hm

(164.54 83.2283.22 42.09

) (43.09 0.720.72 0.01

) (43.09 0.720.72 0.01

)

Hc

(886.58 925.59925.59 966.37

) (1008.02 1008.081008.08 1008.45

) (886.58 925.59925.59 966.37

Hcm

(280.31 137.0637.06 67.64

) (280.31 137.0637.06 67.64

) (280.31 137.0637.06 67.64

Sum

(1331.43 1145.871145.87 1076.10

) (1209.98 1063.381063.38 1034.03



Type III & Type IV SSCP◮ These are similar to Type II in terms of the idea; that is, they

are a kind of partial SSCP, but what you condition on isdifferent.

◮ Type III SSCP give the increase in model SSCP due to addinga particular variable or interaction (effect) to a model thatcontains all other variables and effects listed in the model.

◮ For example, if

model x1 x2 = A B C A ∗ B A ∗ C B ∗ C A ∗ B ∗ CThat is all main effects, 2–way interactions and a 3-wayinteraction.The Type III SSCP for A ∗ C is the increase in model SSCPthat is obtained by adding A ∗ C to a model that has A, B , C ,A ∗ B , B ∗ C and A ∗ B ∗ C .In general these will tend to be smaller than Types I and IIC.J. Anderson (Illinois) MANOVA: Part 2 Spring 2017 80.1/ 82


Type IV and When to Use WhichType IV are designed for situations where there are empty cells.

◮ For balanced designs, the 4 ways to compute SSCP yield the sameresults.

◮ For unbalanced designs, the 4 ways can lead to different conclusions.

◮ Type III is appropriate when interest is in comparing the effects ofexperimental variables.

◮ For model building purposes (finding a good model for prediction),Types I and/or II might be preferable.

◮ If there are design parameters, make sure that tests do not involvethese.

◮ The choice should be made on a case-by-case basis with thoughtgiven to what it is you want to test.



Conclusions

◮ Any design that you can form for univariate case can beformed for the multivariate situation.

◮ This can be done by creating an appropriate design matrix inthe general linear model context.

◮ Models can include fixed (as we have done), but also randomeffects (as in the univariate case).

◮ It is possible to reformulated MANOVA as a linear mixedmodel, and hence provides a natural way to handle missingobservations (be sure there are systematic reasons for missingvalues). See Snijders & Bosker. . . . see Goldstein.


Documents

MANOVA: Part 2 Proﬁle Analysis and 2-way MANOVA · MANOVA: Part 2 Proﬁle Analysis and 2-way MANOVA Edps/Soc 584, Psych 594 Carolyn J. Anderson Department of Educational Psychology