Quantitative Methods for Researchers

Paul Cairnspaul.cairns@york.ac.uk

Objectives

Statistical argument Comparison of distributions A fly-by of approaches

2

How are the abstracts?

Questions? Problems? Restarts?

3

4

Statistical Argument

Inference is an argument form Prediction is essential– Alternative hypothesis– “X causes Y”

No prediction – measuring noise

5

Gold standard argument

1. Collect data2. Data variation could be chance

(null)3. Predict the variations

(alternative)4. Statistics give probabilities5. Unlikely predictions “prove” your

case

6

Implications

Must have an alt hyp No multiple testing No post hoc analysis Need multiple experiments

7

Silver standard argument

1. Collect data2. Data variations could be chance

(null)3. Are there “real” patterns in the

data?4. Use statistics to suggest

(unlikely) patterns5. Follow up findings with gold

standard work

8

Fishing: This is bad science

1. Collect lots of data– DVs and IVs

2. Data variations could be chance3. Test until a significant result

appears4. Report the tests that were

significant5. Claim the result is important

Statistical inference

Model comparison:– Single distribution (null)– Multiple distributions (alternative)

From samples, which model is better?

From samples, is null likely?

9

What terms do you know?

The statistical zoo!

10

Choosing a test

What’s the data type? Do you know the distribution? Within or between What are you looking for?

11

Distributions

Theoretical stance Must have this! Not inferred from samples

12

13

Parametric tests

Normal distribution Two parameters Null = one underlying normal

distribution Differences in location (mean)

t-test models

14

t-test

Two samples Two means Are means showing natural

variation? Compare difference to natural

variation

15

set AB

Effect size

How interesting is the difference?– 2s difference in timings – Significance is not same as

importance Cohen’s d

16

sd AB

ANOVA

Parametric Multiple groups Why not do pairwise comparison? Get an F value Follow up tests

17

ANOVA++

Multiple IV– So more F values!

Within and between Effect size, η2

– Amount of variance predicted by IV

18

Non-parametric tests

Unknown underlying distribution Heterogeneity of variance Non-interval data Usually test location Effect size is tricky!

19

Wilcoxon test

See sheet

20

Seeing location

Boxplots Median, IQR, “Range” Outliers

21

22

Multivariate

Multiple DV Multivariate normal distribution– Normal no matter how you slice

MANOVA Null = one underlying (mv)

normal distribution

23

24

Issues

Sample size Assumptions Interpretation Communication

25

Your abstract

What sort of data will you produce?

Can you theorise about the distribution?

What sort of test do you think you will need?

26

Health warnings

Craft skill Simpler is better– Doing it – Interpreting it– Communicating it

Experiments as evidence Software packages are deceptively

easy27

Q & A

Any question about any aspect Very general or very specific Any research method!

28

Useful Reading

Cairns, Cox, Research Methods for HCI: chaps 6

Rowntree, Statistics Without Tears Howell, Fundamental Statistics for

the Behavioural Sciences, 6th edn. Abelson, Statistics as Principled

Argument Silver, The Signal and the Noise

29

Monte Carlo

Process but not distribution Generate a really large sample Compare to your sample Still theoretically driven!

30

Example

Event = 4 heads in a row from a set of 20 flips of a coin

You have sample of 30 sets 18 events How likely?– Get flipping!

31