Discrimination Among Groups - UMass Amherst · 5 Tests of Among-Group Differences Permutational Multivariate Analysis of Variance (NP-MANOVA) PNonparametric procedure for testing

1

Discrimination Among Groups

Id Species Canopy Snag CanopyCover Density Height

1 A 80 1.2 352 A 75 0.5 323 A 72 2.8 28. . . . .31 B 35 3.3 1532 B 75 4.1 2560 B 15 5.0 3. . . . .61 C 5 2.1 562 C 8 3.4 290 C 25 0.6 15

2

P Are groups significantly different? (How validare the groups?)< Multivariate Analysis of Variance [(NP)MANOVA]< Multi-Response Permutation Procedures [MRPP]< Analysis of Group Similarities [ANOSIM]< Mantel’s Test [MANTEL]

P How do groups differ? (Which variables bestdistinguish among the groups?)< Discriminant Analysis [DA]< Classification and Regression Trees [CART]< Logistic Regression [LR]< Indicator Species Analysis [ISA]

Discrimination Among Groups

3

1 AMRO 15.31 31.42 64.28 20.71 47.14 0.00 0.28 0.14 . . . 1.45 2 BHGR 5.76 24.77 73.18 22.95 61.59 0.00 0.00 1.09 . . . 1.28 3 BRCR 4.78 64.13 30.85 12.03 63.60 0.44 0.44 2.08 . . . 1.18 4 CBCH 3.08 58.52 39.69 15.47 62.19 0.31 0.28 1.52 . . . 1.21 5 DEJU 13.90 60.78 36.50 13.81 62.89 0.23 0.31 1.23 . . . 1.23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19 WIWR 8.05 41.09 55.00 18.62 53.77 0.09 0.18 0.81 . . . 1.36

2-group bird guilds example

Tests of Among-Group Differences

4

P Two-group clusteringsolution resulting fromagglomerativehierarchical clustering(hclust) using Euclideandistance on standardizedvariables and Ward’slinkage method.



5

Tests of Among-Group DifferencesPermutational Multivariate Analysis of Variance

(NP-MANOVA)P Nonparametric procedure

for testing the hypothesis ofno difference between twoor more groups of entitiesbased on the analysis andpartitioning sums of squaredistances (Anderson 2001).

6


P Calculate distance matrix (anydistance metric can be used)

P Calculate average distanceamong all entities (SST)

P Calculate average distanceamong entities withingroups (SSW)

P Calculate average distanceamong groups (SSA = SST -SSW)

P Calculate F-ratio

N = total number of items a = number of groupsn = number of items per groupdij = distance between ith and jth entityεij = 1 if in same group; 0 otherwise

Permutational (nonparametric) MANOVA

7

Tests of Among-Group DifferencesPermutational (nonparametric) MANOVA

8


P Determine the probability of anF-ratio this large or largerthrough Monte carlopermutations.

< Permutations involverandomly assigning sampleobservations to groups (withinstrata if nested design).

< The significance test is simplythe fraction of permuted F-ratios that are greater than theobserved F-ratio. F-ratio

ObservedF-ratio

p

C nNi

i

d i

delta C di i

i

g

1

9


P Conclude that the differences between the twobird species clusters is statistically significant andthat 43% of the “variance” is accounted for bygroup differences


10

Tests of Among-Group DifferencesMulti-Response Permutation Procedures (MRPP)

P Nonparametric procedure for testing the hypothesisof no difference between two or more groups ofentities based on permutation test of among- andwithin-group dissimilarities (Mielke 1984, 1991).

P Calculate distance matrix (Euclideandistance generally recommended,although proportional city-blockmeasures often used with communitydata).

P Calculate average distance in each group=

P Calculate delta (the weighted meanwithin-group distance) for g groups.

ni = number of items in group iN = total number of items

*other options for calculating Ci exit

Group AMRO BHGR RUHU SOSP STJA SWTH WEFL WIWA WIWR BRCR CBCH DEJU EVGR GCKI GRJA HAFL HAWO HEWA VATH1 AMRO 01 BHGR 6.83 01 RUHU 3.77 7.46 01 SOSP 5.60 8.97 3.41 01 STJA 4.88 7.26 3.14 4.56 01 SWTH 3.11 4.70 3.34 5.57 3.54 01 WEFL 3.61 6.57 3.27 4.85 2.58 2.51 01 WIWA 5.13 6.68 3.49 5.42 3.46 3.45 3.44 01 WIWR 3.71 6.78 2.40 4.25 2.02 2.49 1.50 2.56 02 BRCR 9.15 9.29 7.44 7.65 5.98 7.52 6.66 5.18 6.18 02 CBCH 7.04 7.68 5.14 5.66 3.84 5.28 4.50 3.14 3.85 2.70 02 DEJU 6.98 8.20 4.87 5.35 4.11 5.61 4.94 3.49 4.10 3.11 1.82 02 EVGR 10.73 10.57 8.53 8.57 7.70 9.14 8.66 6.43 7.95 4.19 4.82 5.24 02 GCKI 9.62 9.32 7.58 7.51 6.37 7.92 7.17 5.74 6.68 1.97 3.13 3.40 4.08 02 GRJA 10.22 9.71 9.68 9.60 8.50 8.93 8.25 7.85 8.23 5.30 5.88 6.20 8.06 5.53 02 HAFL 11.21 10.18 9.15 9.32 8.07 9.44 9.00 6.97 8.48 3.98 5.10 5.46 3.98 2.96 7.29 02 HAWO 7.49 7.69 5.59 5.83 4.54 5.70 5.38 4.49 4.67 4.19 3.29 3.49 6.30 4.30 7.36 6.02 02 HEWA 12.42 11.75 10.53 10.09 9.70 11.17 10.70 9.24 10.22 6.41 7.47 7.32 6.86 5.17 8.25 4.56 8.04 02 VATH 7.08 8.41 5.83 5.93 4.19 5.97 4.63 5.27 4.68 4.78 4.28 4.51 7.12 5.10 7.44 7.07 4.39 7.89 0

d i 4 34.

d 2 5 24 .

c1 36 81 444 / .

c2 45 81 556 / .

4 84.

11

Note: only with-group dissimilarities are used.



12

P Determine the probability of adelta this small or smallerthrough Monte carlopermutations.

< Permutations involverandomly assigning sampleobservations to groups.

< The significance test is simplythe fraction of permuteddeltas that are less than theobserved delta, with a smallsample correction.

Delta

Observeddelta

p


Aobserved

ected 1 1

exp

13

P Conclude that two clusters(bird species with similarniches) differ significantly interms of the measuredhabitat variables.



14

P Determine the effect sizeindependent of sample size(chance-corrected within-group agreement, A).

P The statistic A is given as a descriptor of within-grouphomogeneity compared to the random expectation

A = 1 when all items are identical within groupsA = 0 when within-group heterogeneity equals expectation

by chanceA < .1 common in ecologyA > .3 is fairly high in ecology (but see simulation results later)

Note: statistical significance (small p-value) may result even whenthe “effect size” (A) is small, if the sample size is large.


15

P Conclude that the differences between the twobird species clusters is statistically significant andthat the difference is moderately large andtherefore probably ecologically significant as well.



16

n=10, p=1, e=1 n=10, p=1, e=4

n=20, p=1, e=1

n=10, p=3, e=1

Simulation Study


17

e=1Sample sizee=0


18

p=10Sample sizep=1


19

p=100Sample sizep=1


20

p=10, e=10Sample sizep=1, e=1


21

p=10Effect sizep=1


22

n=20Effect sizen=100


23

e=10Noise Ratioe=1


24

n=20Noise Ration=100


25



26

Column standardized Raw (same range)Data Standardization(sample size)


27

Column standardized Raw (same range)Data Standardization(effect size)


28

Column standardized Raw (same range)Data Standardization(noise ratio)


Rr r

MA W

2

29

P Test statistic is unreliable for n<20, and for larger samplesizes (e.g., n=40) if the noise ratio is greater than ~10:1.

P Significance (p-value) of test statistic insensitive to degree ofnoise (nondiscriminating variables); i.e., a single effectivediscriminating variable will produce a significant test resultfor n>20, as long as noise ratio <~10:1.

P Effect size (A) sensitive to both magnitude of effect on at leastone discriminating variable and the noise ratio (i.e., ratio ofnondiscriminating variables to true discriminators); noiseratio relatively more important.

P Data standardization (col z-score) has moderate effect oneffect size (A).

Simulation Results Summary


30

Analysis of Group Similarities (ANOSIM)

P Nonparametric procedure for testing the hypothesis ofno difference between two or more groups of entitiesbased on permutation test of among- and within-groupsimilarities (Clark 1993).

P Calculate dissimilarity matrix.

P Calculate rank dissimilarities (smallestdissimilarity is given a rank of 1).

P Calculate mean among- and within-group rank dissimilarities.

P Calculate test statistic R (an index ofrelative within-group dissimilarity).

M = N(N-1)/2= number of

sample pairs


Rr r

MA W

2

Group AMRO BHGR RUHU SOSP STJA SWTH WEFL WIWA WIWR BRCR CBCH DEJU EVGR GCKI GRJA HAFL HAWO HEWA VATH1 AMRO1 BHGR 1021 RUHU 31 1171 SOSP 79 145 211 STJA 60 111 16 511 SWTH 13 55 19 77 281 WEFL 29 97 17 58 9 71 WIWA 64 99 27 74 24 23 221 WIWR 30 101 5 42 4 6 1 82 BRCR 148 151 116 123 89 121 98 67 912 CBCH 106 124 65 81 32 70 47 15 33 102 DEJU 105 134 59 72 38 80 61 25 37 12 22 EVGR 167 165 141 142 126 147 143 96 130 41 57 682 GCKI 156 152 122 120 94 129 110 83 100 3 14 20 362 GRJA 162 159 157 155 140 144 137 127 135 71 86 92 132 762 HAFL 169 161 149 153 133 154 146 104 139 34 62 75 35 11 1122 HAWO 119 125 78 85 49 82 73 46 53 40 18 26 93 44 114 902 HEWA 171 170 164 160 158 168 166 150 163 95 118 113 103 66 136 50 1312 VATH 108 138 84 87 39 88 52 69 54 56 43 48 109 63 115 107 45 128

rw 57 68.ra 11150.

R 0 629.

31

R = 1 when all pairs of samples withingroups are more similar than toany pair of samples fromdifferent groups.

R = 0 expected value under the nullmodel that among-and within-group dissimilarities are the sameon average.

R < 0 numerically possible butecologically unlikely.

M = N(N-1)/2= number of

sample pairs

R is interpreted like a correlation coefficient and isa measure of ‘effect size’, like A in MRPP:

Analysis of Group Similarities (ANOSIM)Tests of Among-Group Differences

32

Note: both within- and among-group dissimilarities are used.

2-group bird guilds exampleAnalysis of Group Similarities (ANOSIM)Tests of Among-Group Differences

33

R Permutation Distribution

ObservedR

p

-1 # R # 1

P Determine the probability of an Rthis large or larger throughMonte carlo permutations.

< Permutations involverandomly assigning sampleobservations to groups.

< The significance test is simplythe fraction of permuted R’sthat are greater than theobserved R.


34

P Conclude that two clustersdiffer significantly in terms of themeasured habitat variables.

2-group bird guilds exampleAnalysis of Group Similarities (ANOSIM)Tests of Among-Group Differences

35

n=10, p=1, e=1 n=10, p=1, e=4

n=20, p=1, e=1

n=10, p=3, e=1

Simulation Study


36

e=1Sample sizee=0


37

e=2Sample sizee=1


38

e=10Sample sizee=1


39

e=10Sample sizee=1


40

p=10Sample sizep=1


41

p=100Sample sizep=1


42

p=10, e=10Sample sizep=1, e=1


43

p=10Effect sizep=1


44

n=20Effect sizen=100


45

e=10Noise Ratioe=1


46



47



48

Column standardized Raw (same range)


Data Standardization(sample size)

49



Data Standardization(effect size)

50

Data Standardization(noise ratio)



51

P Test statistic is unreliable for n<20, and for larger samplesizes (e.g., n=40) if the noise ratio is greater than ~10:1.

P Significance (p-value) of test statistic insensitive to degree ofnoise (nondiscriminating variables); i.e., a single effectivediscriminating variable will produce a significant test resultfor n>20, as long as noise ratio <~10:1.

P Effect size (R) sensitive to both magnitude of effect on at leastone discriminating variable and the noise ratio (i.e., ratio ofnondiscriminating variables to true discriminators); noiseratio relatively more important.

P Data standardization (col z-score) has moderate effect oneffect size (R).

Simulation Results Summary

Tests of Among-Group DifferencesAnalysis of Group Similarities (ANOSIM)

52

Tests of Among-Group DifferencesVariations on a Good Theme

P Blocked MRPP (MRBP) – For randomizedblocked designs, paired-sample data, andsimple repeated measures (haven’t found Rfunction; but BLOSSOM software can do it).

P Qb method – for partitioning variance in thedistance matrix into sums of squares formultiple factors, including interactions (limitedto euclidean distance metric).

P NPMANOVA – for non-euclidean distancemeasures in multifactor designs, includingnested and factorial designs. [adonis(vegan)]

53


P MRPP and related methods (ANOSIM,MRBP, Qb, NPMANOVA) test fordifferences among groups in either location(differences in mean) or spread (differencesin within-group distance). That is, they mayfind a significant difference between groupssimply because one of the groups hasgreater dissimilarities (dispersion) among itssampling units.

P Differences in spread (‘heterogeneity ofdispersions’) can be either an asset or anassumption that must be met.

Considerations with MRPP & Related Methods

54

Simulated Data Sets

Tests of Among-Group DifferencesConsiderations with MRPP & ANOSIM

Equal location(e)Equal spread(d)

n=100, p=1, e=0, d=1

Unequal location(e)Equal spread(d)

n=100, p=1, e=1, d=1

55

Simulated Data Sets


Equal location(e)Unequal spread(d)

n=100, p=1, e=0, d=2

Unequal location(e)Unequal spread(d)

n=100, p=1, e=1, d=2

56

Simulated Data Sets


n=100, p=1, e=0, d=4n=100, p=1, e=0, d=2



57

Simulated Data Sets


n=100, p=1, e=1, d=1 n=100, p=1, e=10, d=1



58

MRPP ANOSIM


Real data set example

z x yi

n

ij ijj i

n

1

1

1

59

Tests of Among-Group DifferencesAssumptions of MRPP & Related Methods

P Independent samples -- Usual problemsassociated with pseudoreplication,subsampling and repeated measures applyhere.

P Chosen distance measure adequately representsthe variation of interest in the data.

P The relative weighting of variables has beencontrolled prior to calculating distance, suchthat the weighting of variables isappropriate for the ecological question athand.

60

YEcologicaldistance

XContrast matrix for

groups

P Mantel statistic tests for differences between twodistance matrices (e.g., between ecological andgeographic distances between points), but can alsobe used to test for differences among groups.

1 = Differentgroups

0 = Samegroups

P Calculate dissimilaritymatrices.

P Calculate test statistic Z.

10 10 1 01 0 1 00 0 1 0 1Hadamard Product

Mantel’s Test (MANTEL)Tests of Among-Group Differences

r

x x

s

y y

s

n

i j

xj

i j

yi

_ _

1

61

Z Permutation Distribution

ObservedZ

p

P Determine the probability of a Zthis large or larger throughMonte carlo permutations.

< Permutations involverandomly shuffling one of thetwo distance matrices.

< The significance test is simplythe fraction of permuted Z’sthat are greater than theobserved Z.

Note, if ecological distances arerank transformed, this test is thesame as ANOSIM and similarto rank-transformed MRPP.


62

P Determine the strength of therelationship between the twomatrices by computing theircorrelation, r (standardizedMantel statistic).

r > 0 indicates that the average within-group distancebetween samples is less than the average overalldistance between samples; the larger the r, themore tightly clustered the groups are.

r < 0 numerically possible, but highly unlikely inecology.


63

P Conclude that two clustersdiffer significantly in terms of themeasured habitat variables.

2-group bird guilds exampleMantel’s Test (MANTEL)


64

Variations on a Really Good Theme

P Because Mantel’s test is merely a correlationbetween distance matrices and the distance matricescan be variously defined, the test can assume avariety of forms as special cases.

P These are, in fact, variants of the same case but areinterpreted somewhat differently. There are at leastsix variants.

XDissimilarity

matrix

YDissimilarity

matrix

ZDissimilarity

matrix


65

P Case 1. Simple Mantel’s Test on Geographic Distance.

GeographicDistance

SpeciesDissimilarity

If the dependent distance matrix is species similarity andthe predictor matrix is geographic distance (“spatialdissimilarity”), the research question is “Are samplesthat are close together also compositionally similar?”This is equivalent to testing for overall autocorrelationin the dependent matrix (i.e., averaged over alldistances).


66

P Case 1. Simple Mantel’s Test on Geographic Distance.

Xmatrix = euclideandistance betweensamples) [24 sites, 2variables]

Ymatrix = Masspine barrens mothabundances[24 x 10 species]


67

P Case 2. Simple Mantel’s Test on a Predictor Matrix.

EnvironmentalDissimilarity

SpeciesDissimilarity

If the dependent matrix is again species similarity and thepredictor matrix is a dissimilarity matrix based on a set ofenvironmental variables, then the simple test is forcorrelation between the two matrices. Such correlation wouldindicate that locations that are similar environmentally tendto be similar compositionally. This, of course, is one of thefundamental questions in ecology.


68

P Case 2. Simple Mantel’s Test on a Predictor Matrix.

Xmatrix = Oregon riparian environment(geomorphology, landscapecomposition and configuration)[164 sites, 48 variables]

Ymatrix = Oregon riparian birdsabundances [164 sites, 49species]


69

P Case 3. Simple Mantel’s Test between an Observed Matrix andOne Posed by a Model.

As a formal hypothesis test, Mantel’s test can be used tocompare an observed dissimilarity matrix to one posed by aconceptual or numerical model. Here, the model is providedas a user-provided matrix of similarities or distances, andthe test is to summarize the strength of thecorrespondence between the two matrices. The modeldistance matrix might be provided as a simple binary matrixof 0’s and 1’s, or a matrix derived from a more complicatedmodel.

ModelDissimilarity

ObservedDissimilarity


70

P Case 3. Simple Mantel’s Test between an Observed Matrix andOne Posed by a Model.

Xmatrix = Hammond’s flycatcherpresence/absence in OregonCoast Range[96 sites, 1 indicator variable]

Ymatrix = Oregon habitat variables[96 sites, 20 habitat variables]


71

P Case 4. The Mantel Correlogram. A special case of case 1 (above) is to partition or subset theanalysis into a series of discrete distance class. That is, a firstdistance matrix is evaluated for all pairs of points within thefirst distance class; then a second matrix is scored for all pairsof points within the second distance interval, and so on. Theresult of this analysis is a Mantel’s correlogram, completelyanalogous to an autocorrelation function but performed on a(possibly multivariate) distance matrix.

GeographicDistance Class

ObservedDissimilarity


72

P Case 4. The Mantel Correlogram.

XEcologicaldistance

YGeographic

distance Mantel correlogram

10 10 1 01 0 1 00 0 1 0 1

02 02 0 20 0 0 20 2 0 0 0

Etc.

rm2

rm1


Species Geomorphology Vegetation

Geo-Spec Veg-Spec

.266 (p<.001) .326 (p<.001)

Spec-Geo|Veg Veg-Geo

.211 (p<.001) .219 (p=.003)

Spec-Veg|Geo Geo-Veg|Spec

.285 (p<.001) .146 (p=.019)

-

-

-Species

Geomorphology

Vegetation

73

P Case 5. Partial Mantel’s Test on Three Distance Matrices.

The idealized Mantel’s test is a partial regression on threedistance matrices. Here, the research question is, “Howmuch of the variability in the dependent matrix isexplained by the independent matrix after removing theeffects of a third constraining matrix?” The analysis in thiscase is partial regression, and both partial correlation (orregression) coefficients are of interest: rYX|Z andrYZ|X.

XDissimilarity

matrix

YDissimilarity

matrix

ZDissimilarity

matrix=


74

P Case 5. Partial Mantel’s Test on Three Distance Matrices.

Species

Geomorphology

Vegetation

Path Diagram (Oregon streamside bird communities)

Simple Mantel r’s

Partial Mantel r’s


75

P Case 5. Partial Mantel’s Test on Multiple Distance Matrices.

SpeciesSpace

Plot

Path Diagram (Mass. pine barrens moth community)

Patch

LandscapeSpace

.134 .014

.119


76

P Case 6. Partial Mantel’s on Multiple Predictor Variables.

Often, knowing that the environment has somerelationship with the dependent variable of interest is notsufficiently satisfying: we wish to know which variables areactually related to the dependent variable. The logicalextension of Mantel’s test is multiple regression, in whichthe predictor variables are entered into the analysis asindividual distance matrices. As a partial regressiontechnique, Mantel’s test provides not only an overall testfor the relationships among distance matrices, but also teststhe contribution of each predictor variable for its purepartial effect on the dependent variable.


Documents

Discrimination Among Groups - UMass Amherst · 5 Tests of Among-Group Differences Permutational Multivariate Analysis of Variance (NP-MANOVA) PNonparametric procedure for testing