Quantitative Methods for Researchers Paul Cairns paul.cairns@york.ac.uk

Quantitative Methods for Researchers

Paul Cairnspaul.cairns@york.ac.uk

Objectives

Statistical argument Safe designs A whizz through some stats Time for questions

2

3

Statistical Argument

Inference is an argument form Prediction is essential– Alternative hypothesis– “X causes Y”

No prediction – measuring noise

4

Gold standard argument

1. Collect data2. Data variation could be chance

(null)3. Predict the variations

(alternative)4. Statistics give probabilities5. Unlikely predictions “prove” your

case

5

Implications

Must have an alt (testable) hyp No multiple testing No post hoc analysis Need multiple experiments

6

Silver standard argument

1. Collect data2. Data variations could be chance

(null)3. Are there “real” patterns in the

data?4. Use statistics to suggest

(unlikely) patterns5. Follow up findings with gold

standard work

7

Fishing: This is bad science

1. Collect lots of data– DVs and IVs

2. Data variations could be chance3. Test until a significant result

appears4. Report the tests that were

significant5. Claim the result is important

Statistical pit…

… is bottomless! Safe designs– One (or two) IV– Two (or three) conditions– One primary DV

Other stuff is not severely tested

8

Choosing a test

What’s the data type? Do you know the distribution? Within or between What are you looking for?

9

Seeing location

Boxplots Median, IQR, “Range” Outliers

10

11

Distributions

Theoretical stance Must have this! Not inferred from samples

12

13

Parametric tests

Normal distribution Two parameters Null = one underlying normal

distribution Differences in location (mean)

t-test: null vs alternate

14

t-test

Two samples Two means Are means showing natural

variation? Compare difference to natural

variation

15

set AB

Effect size

How interesting is the difference?– 2s difference in timings – Significance is not same as

importance Cohen’s d

16

sd AB

ANOVA

Parametric Multiple groups Why not do pairwise comparison? Get an F value Follow up tests

17

ANOVA++

Multiple IV– So more F values!

Within and between Effect size, η2

– Amount of variance predicted by IV

18

Non-parametric tests

Unknown underlying distribution Heterogeneity of variance Non-interval data Usually test location Effect size is tricky!

19

Basic tests

Mann-Whitney Wilcoxon Kruskal-Wallis Friedman No accepted two-way tests

20

Choosing a test

For your fantasy abstract, what test would you choose? Why?

Would you change your design?

21

Questions

Specific problems Specific tests Other tests?

22

Useful Reading

Cairns, Cox, Research Methods for HCI: chaps 6

Rowntree, Statistics Without Tears Howell, Fundamental Statistics for

the Behavioural Sciences, 6th edn. Abelson, Statistics as Principled

Argument Silver, The Signal and the Noise

23

Multivariate

Multiple DV Multivariate normal distribution– Normal no matter how you slice

MANOVA Null = one underlying (mv)

normal distribution

24

25

Issues

Sample size Assumptions Interpretation Communication

26

Monte Carlo

Process but not distribution Generate a really large sample Compare to your sample Still theoretically driven!

27

Example

Event = 4 heads in a row from a set of 20 flips of a coin

You have sample of 30 sets 18 events How likely?– Get flipping!

28