Factorial Designs & Managing Violated Statistical Assumptions
Dr James Betts
Developing Study Skills and Research Methods (HL20107)
Lecture Outline:
•Factorial Research Designs Revisited
•Mixed Model 2-way ANOVA
•Fully independent/repeated measures 2-way ANOVA
•Statistical Assumptions of ANOVA.
Last Week Recap• In last week’s lecture we saw two worked
examples of 1-way analyses of variance
• However, many experimental designs have more than one independent variable (i.e. factorial design)
Factorial Designs: Technical Terms• Factor
• Levels
• Main Effect
• Interaction Effect
Factorial Designs: Multiple IV’s• Hypothesis:
– The HR response to exercise is mediated by gender
• We now have three questions to answer:
1)
2)
3)
Post Run 1 Pre Run 2 Post Run 2
Mu
scle
Gly
cogen
(mm
ol
glu
cosy
l u
nit
s. k
g d
m-1)
0
50
100
150
200
250
300
350
CHO CHO-PRO
Factorial Designs: Interpretation210
180
150
120
90
60
30
0
Hea
rt R
ate
(bea
tsm
in-1)
Resting Exercise
Main Effect of Exercise
Not significant
Main Effect of Gender
Not significant
Exercise*Gender Interaction
Not significant
Post Run 1 Pre Run 2 Post Run 2
Mu
scle
Gly
cogen
(mm
ol
glu
cosy
l u
nit
s. k
g d
m-1)
0
50
100
150
200
250
300
350
CHO CHO-PRO
Factorial Designs: Interpretation210
180
150
120
90
60
30
0
Hea
rt R
ate
(bea
tsm
in-1)
Resting Exercise
Main Effect of Exercise
Significant
Main Effect of Gender
Not significant
Exercise*Gender Interaction
Not significant
Post Run 1 Pre Run 2 Post Run 2
Mu
scle
Gly
cogen
(mm
ol
glu
cosy
l u
nit
s. k
g d
m-1)
0
50
100
150
200
250
300
350
CHO CHO-PRO
Factorial Designs: Interpretation210
180
150
120
90
60
30
0
Hea
rt R
ate
(bea
tsm
in-1)
Resting Exercise
Main Effect of Exercise
Not significant
Main Effect of Gender
Significant
Exercise*Gender Interaction
Not significant
Post Run 1 Pre Run 2 Post Run 2
Mu
scle
Gly
cogen
(mm
ol
glu
cosy
l u
nit
s. k
g d
m-1)
0
50
100
150
200
250
300
350
CHO CHO-PRO
Factorial Designs: Interpretation210
180
150
120
90
60
30
0
Hea
rt R
ate
(bea
tsm
in-1)
Resting Exercise
Main Effect of Exercise
Not significant
Main Effect of Gender
Not Significant
Exercise*Gender Interaction
Significant
Post Run 1 Pre Run 2 Post Run 2
Mu
scle
Gly
cogen
(mm
ol
glu
cosy
l u
nit
s. k
g d
m-1)
0
50
100
150
200
250
300
350
CHO CHO-PRO
Factorial Designs: Interpretation210
180
150
120
90
60
30
0
Hea
rt R
ate
(bea
tsm
in-1)
Resting Exercise
?
Post Run 1 Pre Run 2 Post Run 2
Mu
scle
Gly
cogen
(mm
ol
glu
cosy
l u
nit
s. k
g d
m-1)
0
50
100
150
200
250
300
350
CHO CHO-PRO
Factorial Designs: Interpretation210
180
150
120
90
60
30
0
Hea
rt R
ate
(bea
tsm
in-1)
Resting Exercise
?
SystematicVariance
(resting vs exercise)
ErrorVariance
(between subjects)
Systematic Variance
(male vs female)
Systematic Variance
(Interaction)
2-way mixed model ANOVA: Partitioning
= variance between means due to exercise
= variance between means due to gender
= variance between means due IV interaction
= uncontrolled factors and within group differences for males vs females.
ErrorVariance
(within subjects)= uncontrolled factors plus random changes within individuals for rest vs exercise
Procedure for computing 2-way mixed model ANOVA
• Step 1: Complete the table
i.e.-square each raw score
-total the raw scores for each subject (e.g. XT)
-square the total score for each subject (e.g. (XT)2)
-Total both columns for each group
-Total all raw scores and squared scores (e.g. X & X2).
• Step 2: Calculate the Grand Total correction factor
GT =
(X)2
N
Procedure for computing 2-way mixed model ANOVA
• Step 3: Compute total Sum of Squares
SStotal= X2 - GT
Procedure for computing 2-way mixed model ANOVA
• Step 4: Compute Exercise Effect Sum of Squares
SSex= - GT
= + - GT
(Xex)2
nex
(XRmale+XRfemale)2
nRmale+fem
Procedure for computing 2-way mixed model ANOVA
(XEmale+XEfemale)2
nEmale+fem
• Step 5: Compute Gender Effect Sum of Squares
SSgen= - GT
= + - GT
(Xgen)2
ngen
(XRmale+XEmale)2
nmaleR+E
Procedure for computing 2-way mixed model ANOVA
(XRfem+XEfem)2
nfemR+E
• Step 6: Compute Interaction Effect Sum of Squares
SSint= - GT
= + + + - (SSex+SSgen) - GT
(Xex+gen)2
nex+gen
nRmale
Procedure for computing 2-way mixed model ANOVA
nRfem nEmale nEfem
(XRmale)2 (XRfem)2 (XEmale)2 (XEfem)2
• Step 7: Compute between subjects Sum of Squares
SSbet= -SSgen- GT
= -SSgen- GT
(XS)2
nk
Procedure for computing 2-way mixed model ANOVA
(XT)2+(XD)2+(XH)2+(XJ)2+(XK)2+(XA)2+(XS)2+(XL)2
nk
• Step 8: Compute within subjects Sum of Squares
SSwit= SStotal - (SSex+SSgen+SSint+SSbet)
Procedure for computing 2-way mixed model ANOVA
• Step 9: Determine the d.f. for each sum of squares
dftotal = (N - 1)
dfex = (k - 1)
dfgen = (r - 1)
dfint = (k - 1)(r - 1)
dfbet = r(n - 1)
dfwit = r(n - 1)(k - 1)
Procedure for computing 2-way mixed model ANOVA
• Step 10: Estimate the Variances Procedure for computing 2-way mixed model ANOVA
SystematicVariance
(resting vs exercise)
ErrorVariance
(between subjects)
Systematic Variance
(male vs female)Systematic
Variance(Interaction)
ErrorVariance
(within subjects)
SSex
dfex
SSgen
dfgen
SSint
dfint
SSwit
dfwit
SSbet
dfbet
=
=
=
=
=
• Step 11: Compute F values Procedure for computing 2-way mixed model ANOVA
SystematicVariance
(resting vs exercise)
ErrorVariance
(between subjects)
Systematic Variance
(male vs female)Systematic
Variance(Interaction)
ErrorVariance
(within subjects)
SSex
dfex
SSgen
dfgen
SSint
dfint
SSwit
dfwit
SSbet
dfbet
=
=
=
=
=
• Step 12: Consult F distribution table as before Exercise
Gender
Procedure for computing 2-way mixed model ANOVA
Interaction
Tests of Within-Subjects Effects
Measure: MEASURE_1
61504.000 1 61504.000 2396.260 .000
61504.000 1.000 61504.000 2396.260 .000
61504.000 1.000 61504.000 2396.260 .000
61504.000 1.000 61504.000 2396.260 .000
121.000 1 121.000 4.714 .073
121.000 1.000 121.000 4.714 .073
121.000 1.000 121.000 4.714 .073
121.000 1.000 121.000 4.714 .073
154.000 6 25.667
154.000 6.000 25.667
154.000 6.000 25.667
154.000 6.000 25.667
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sourceexercise
exercise * Gender
Error(exercise)
Type III Sumof Squares df Mean Square F Sig.
2-way mixed model ANOVA: SPSS OutputCalculated FexSSex dfex
Systematic Varianceex
SSwit dfwitError
Variancewit
Tests of Between-Subjects Effects
Measure: MEASURE_1
Transformed Variable: Average
227529.000 1 227529.000 17064.675 .000
100.000 1 100.000 7.500 .034
80.000 6 13.333
SourceIntercept
Gender
Error
Type III Sumof Squares df Mean Square F Sig.
2-way mixed model ANOVA: SPSS Output
Calculated FgenSSgen
SSbet
dfgen
dfbet
Error Variancebet
GT
Systematic Variancegen
SystematicVariance
(resting vs exercise)
ErrorVariance
(between subjects)
Systematic Variance
(male vs female)Systematic
Variance(Interaction)
ErrorVariance
(within subjects)
2-way mixed model ANOVA
The previous calculation and associated partitioning is an example of a 2-way mixed model ANOVA– i.e. exercise = repeated measures
gender = independent measures
SystematicVariance
(resting vs exercise)
Systematic Variance
(male vs female)Systematic
Variance(Interaction)
ErrorVariance
(within subjects)
2-way Independent Measures ANOVA
So for a fully unpaired design
– e.g. males vs females
&
rest group vs exercise group
ErrorVariance
(between subjects)
Tests of Between-Subjects Effects
Dependent Variable: HeartRate
58386.000a 3 19462.000 2994.154 .000
234256.000 1 234256.000 36039.385 .000
16.000 1 16.000 2.462 .143
58081.000 1 58081.000 8935.538 .000
289.000 1 289.000 44.462 .000
78.000 12 6.500
292720.000 16
58464.000 15
SourceCorrected Model
Intercept
Gender
CoachCyclist
Gender * CoachCyclist
Error
Total
Corrected Total
Type III Sumof Squares df Mean Square F Sig.
R Squared = .999 (Adjusted R Squared = .998)a.
SystematicVariance
(resting vs exercise)
Systematic Variance
(male vs female)Systematic
Variance(Interaction)
ErrorVariance
(within subjects)
2-way Independent Measures ANOVA
ErrorVariance
(between subjects)
SystematicVariance
(resting vs exercise)
Systematic Variance
(am vs pm)Systematic
Variance(Interaction)
Error Variance
(within subjectsexercise)
2-way Repeated Measures ANOVA
…but for a fully paired design
– e.g. morning vs evening
&
rest vs exercise
Error Variance
(within subjectstime)Error Variance
(within subjectsinteract)
Tests of Within-Subjects Effects
Measure: MEASURE_1
.000 1 .000 .000 1.000
.000 1.000 .000 .000 1.000
.000 1.000 .000 .000 1.000
.000 1.000 .000 .000 1.000
1.000 3 .333
1.000 3.000 .333
1.000 3.000 .333
1.000 3.000 .333
67081.000 1 67081.000 10062.150 .000
67081.000 1.000 67081.000 10062.150 .000
67081.000 1.000 67081.000 10062.150 .000
67081.000 1.000 67081.000 10062.150 .000
20.000 3 6.667
20.000 3.000 6.667
20.000 3.000 6.667
20.000 3.000 6.667
.000 1 .000 .000 1.000
.000 1.000 .000 .000 1.000
.000 1.000 .000 .000 1.000
.000 1.000 .000 .000 1.000
3.000 3 1.000
3.000 3.000 1.000
3.000 3.000 1.000
3.000 3.000 1.000
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
SourceTimeofDay
Error(TimeofDay)
Exercise
Error(Exercise)
TimeofDay * Exercise
Error(TimeofDay*Exercise)
Type III Sumof Squares df Mean Square F Sig.
SystematicVariance
(resting vs exercise)
Systematic Variance
(am vs pm)Systematic
Variance(Interaction)
Error Variance
(within subjectsexercise)
2-way Repeated Measures ANOVA
Error Variance
(within subjectstime)Error Variance
(within subjectsinteract)
Summary: 2-way ANOVA• 2-way (factorial) ANOVA may be appropriate
whenever there are multiple IV’s to compare
• We have worked through a mixed model but you should familiarise yourself with paired/unpaired procedures
• You should also ensure you are aware what these effects actually look like graphically.
Statistical Assumptions• As with other parametric tests, ANOVA is
associated with a number of statistical assumptions
• When these assumptions are violated we often find that an inferential test performs poorly
• We therefore need to determine not only whether an assumption has been violated but also whether that violation is sufficient to produce statistical errors.
Energy Intake (calories per day)
1500 2500 3500 4500 5500
Nu
mb
er o
f P
eo
ple
0
20
40
60
80
100
120
140
160
16 17 18 19 20
Sustained Isometric Torque (seconds)
e.g. ND assumption from last year
Independent Samples Test
7.842 .012 -2.333 18 .031 -1.69600 .72710 -3.22358 -.16842
-2.333 15.447 .034 -1.69600 .72710 -3.24188 -.15012
Equal variancesassumed
Equal variancesnot assumed
SwimTime50mF Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Tests of Normality
.161 10 .200* .968 10 .872
.350 10 .001 .747 10 .003
GroupControl
Visualisation
SwimTime50mStatistic df Sig. Statistic df Sig.
Kolmogorov-Smirnova
Shapiro-Wilk
This is a lower bound of the true significance.*.
Lilliefors Significance Correctiona.
e.g. ND assumption from last year
Assumptions of ANOVA• N acquired through random sampling • Data must be of at least the interval LOM (continuous)
• Independence of observations
• Homogeneity of variance
• All data is normally distributed
“ANOVA is generally robust to violations of the normality assumption, in that even when the data
are non-normal, the actual Type I error rate is usually close to the nominal (i.e., desired) value.”
Maxwell & Delaney (1990) Designing Experiments & Analyzing Data: A Model Comparison Perspective, p. 109
“If the data analysis produces a statistically significant finding when no test of sphericity is
conducted…you should disregard the inferential claims made by the researcher.”
Huck & Cormier (1996) Reading Statistics & Research, p. 432
Group BGroup C
Group ASupplement 1Supplement 2
Placebo
Plac. Supp. 1 Supp. 2 Plac.-Supp. 1 Supp. 1-Supp. 2 Plac.-Supp. 2
Tom 2.4 3.0 3.3 -0.6 -0.3 -0.9
Dick 2.2 2.5 2.4 -0.3 0.1 -0.2
Harry 1.8 1.9 2.2 -0.1 -0.3 -0.4
James 1.6 1.1 1.2 0.5 -0.1 0.4
Mean 2.0 2.1 2.3 -0.1 -0.2 -0.3
SD2 0.1 0.7 0.7 0.2 0.04 0.3
Mauchly's Test of Sphericityb
Measure: MEASURE_1
.310 2.343 2 .310 .592 .752 .500Within Subjects EffectTrial
Mauchly's WApprox.
Chi-Square df Sig.Greenhouse-Geisser Huynh-Feldt Lower-bound
Epsilona
Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables isproportional to an identity matrix.
May be used to adjust the degrees of freedom for the averaged tests of significance. Corrected tests are displayed inthe Tests of Within-Subjects Effects table.
a.
Design: Intercept Within Subjects Design: Trial
b.
• Many texts recommend ‘Mauchley’s Test of Sphericity’– A χ2 test which, if significant, indicates a violation to sphericity
• However, this is not advisable on four counts:1.)
2.)
3.)
4.)
• How should we analyse aspherical data?
• Option 1
• Option 2
• Option 3
Managing Violations to Sphericity
Paired 1-way MANOVA: SPSS Output
Multivariate Testsb
.455 .834a 2.000 2.000 .545
.545 .834a 2.000 2.000 .545
.834 .834a 2.000 2.000 .545
.834 .834a 2.000 2.000 .545
Pillai's Trace
Wilks' Lambda
Hotelling's Trace
Roy's Largest Root
EffectTrial
Value F Hypothesis df Error df Sig.
Exact statistica.
Design: Intercept Within Subjects Design: Trial
b.
Mauchly's Test of Sphericityb
Measure: MEASURE_1
.310 2.343 2 .310 .592 .752 .500Within Subjects EffectTrial
Mauchly's WApprox.
Chi-Square df Sig.Greenhouse-Geisser Huynh-Feldt Lower-bound
Epsilona
Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables isproportional to an identity matrix.
May be used to adjust the degrees of freedom for the averaged tests of significance. Corrected tests are displayed inthe Tests of Within-Subjects Effects table.
a.
Design: Intercept Within Subjects Design: Trial
b.
Paired 1-way ANOVA: SPSS Output
Tests of Within-Subjects Effects
Measure: MEASURE_1
.152 2 .076 .840 .477
.152 1.183 .128 .840 .439
.152 1.505 .101 .840 .457
.152 1.000 .152 .840 .427
.542 6 .090
.542 3.550 .153
.542 4.514 .120
.542 3.000 .181
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
Sphericity Assumed
Greenhouse-Geisser
Huynh-Feldt
Lower-bound
SourceTrial
Error(Trial)
Type III Sumof Squares df Mean Square F Sig.
Summary ANOVA and Sphericity• ANOVA is a generally robust inferential test
• Unpaired data are susceptible to heterogeneity of variance only if group sizes are unequal
• Paired data are susceptible to asphericity only if multiple comparisons are made
• Suggested solutions for the latter include either MANOVA or epsilon corrected df depending on sample size relative to number of levels.