Embed Size (px)
Citation preview
Objectives
Statistical argument Safe designs A whizz through some stats Time for questions
2
3
Statistical Argument
Inference is an argument form Prediction is essential– Alternative hypothesis– “X causes Y”
No prediction – measuring noise
4
Gold standard argument
1. Collect data2. Data variation could be chance
(null)3. Predict the variations
(alternative)4. Statistics give probabilities5. Unlikely predictions “prove” your
case
5
Implications
Must have an alt (testable) hyp No multiple testing No post hoc analysis Need multiple experiments
6
Silver standard argument
1. Collect data2. Data variations could be chance
(null)3. Are there “real” patterns in the
data?4. Use statistics to suggest
(unlikely) patterns5. Follow up findings with gold
standard work
7
Fishing: This is bad science
1. Collect lots of data– DVs and IVs
2. Data variations could be chance3. Test until a significant result
appears4. Report the tests that were
significant5. Claim the result is important
Statistical pit…
… is bottomless! Safe designs– One (or two) IV– Two (or three) conditions– One primary DV
Other stuff is not severely tested
8
Choosing a test
What’s the data type? Do you know the distribution? Within or between What are you looking for?
9
Seeing location
Boxplots Median, IQR, “Range” Outliers
10
11
Distributions
Theoretical stance Must have this! Not inferred from samples
12
13
Parametric tests
Normal distribution Two parameters Null = one underlying normal
distribution Differences in location (mean)
t-test: null vs alternate
14
t-test
Two samples Two means Are means showing natural
variation? Compare difference to natural
variation
15
set AB
Effect size
How interesting is the difference?– 2s difference in timings – Significance is not same as
importance Cohen’s d
16
sd AB
ANOVA
Parametric Multiple groups Why not do pairwise comparison? Get an F value Follow up tests
17
ANOVA++
Multiple IV– So more F values!
Within and between Effect size, η2
– Amount of variance predicted by IV
18
Non-parametric tests
Unknown underlying distribution Heterogeneity of variance Non-interval data Usually test location Effect size is tricky!
19
Basic tests
Mann-Whitney Wilcoxon Kruskal-Wallis Friedman No accepted two-way tests
20
Choosing a test
For your fantasy abstract, what test would you choose? Why?
Would you change your design?
21
Questions
Specific problems Specific tests Other tests?
22
Useful Reading
Cairns, Cox, Research Methods for HCI: chaps 6
Rowntree, Statistics Without Tears Howell, Fundamental Statistics for
the Behavioural Sciences, 6th edn. Abelson, Statistics as Principled
Argument Silver, The Signal and the Noise
23
Multivariate
Multiple DV Multivariate normal distribution– Normal no matter how you slice
MANOVA Null = one underlying (mv)
normal distribution
24
25
Issues
Sample size Assumptions Interpretation Communication
26
Monte Carlo
Process but not distribution Generate a really large sample Compare to your sample Still theoretically driven!
27
Example
Event = 4 heads in a row from a set of 20 flips of a coin
You have sample of 30 sets 18 events How likely?– Get flipping!
28
