Severe Testing: The Key to Error Correction

Deborah G MayoVirginia Tech

March 17, 2017 “Understanding Reproducibility and Error Correction in

Science”

2

Statistical Crisis of Replication

O Statistical ‘findings’ disappear when others look for them.

O Beyond the social sciences to genomics, bioinformatics, and medicine (Big Data)

O Methodological reforms (some welcome, others radical)

O Need to understand philosophical, statistical, historical issues

3

American Statistical Association (ASA):Statement on P-values

“The statistical community has been deeply concerned about issues of reproducibility and replicability of scientific conclusions. …. much confusion and even doubt about the validity of science is arising. Such doubt can lead to radical choices such as…to ban p-values” (ASA, Wasserstein & Lazar 2016, p. 129)

4

I was a philosophical observer at the ASA P-value “pow wow”

5

“Don’t throw out the error control baby with the bad statistics bathwater” The American Statistician

6

O The most used methods are most criticized

O “Statistical significance tests are a small part of a rich set of:

“techniques for systematically appraising and bounding the probabilities … of seriously misleading interpretations of data” (Birnbaum 1970, p. 1033)

O These I call error statistical methods (or sampling theory)”.

7

Error Statistics

O Statistics: Collection, modeling, drawing inferences from data to claims about aspects of processes

O The inference may be in error

O It’s qualified by a claim about the method’s capabilities to control and alert us to erroneous interpretations (error probabilities)

8

“p-value. …to test the conformity of the particular data under analysis with H0 in some respect:…we find a function T = t(y) of the data, to be called the test statistic, such that• the larger the value of T the more

inconsistent are the data with H0;• The random variable T = t(Y) has a

(numerically) known probability distribution when H0 is true.

…the p-value corresponding to any t0bs asp = p(t) = Pr(T ≥ t0bs; H0)”

(Mayo and Cox 2006, p. 81)

9

Testing Reasoning

O If even larger differences than t0bs occur fairly frequently under H0 (i.e.,P-value is not small), there’s scarcely evidence of incompatibility with H0 

O Small P-value indicates some underlying discrepancy from H0 because very probably you would have seen a less impressive difference than t0bs were H0 true.

O This indication isn’t evidence of a genuine statistical effect H, let alone a scientific conclusion H*

Stat-Sub fallacy  H => H*

10

O I’m not keen to defend many uses of significance tests long lampooned

O I introduce a reformulation of tests in terms of discrepancies (effect sizes) that are and are not severely-tested

O The criticisms are often based on misunderstandings; consequently so are many “reforms”

11

Replication Paradox (for Significance Test Critics)

Critic: It’s much too easy to get a small P-value

You: Why do they find it so difficult to replicate the small P-values others found?

Is it easy or is it hard?

12

Only 36 of 100 psychology experiments yielded small P-values in Open Science Collaboration on

replication in psychology

OSC: Reproducibility Project: Psychology: 2011-15 (Science 2015): Crowd-sourced effort to replicate 100 articles (Led by Brian Nozek, U. VA)

13

O R.A. Fisher: it’s easy to lie with statistics by selective reporting, not the test’s fault

O Sufficient finagling—cherry-picking, P-hacking, significance seeking, multiple testing, look elsewhere—may practically guarantee a preferred claim H gets support, even if it’s unwarranted by evidence

(biasing selection effects, need to adjust P-values)

Note: Support for some preferred claim H is by rejecting a null hypothesisO H hasn’t passed a severe test

14

Severity Requirement:

If the test procedure had little or no capability of finding flaws with H (even if H is incorrect), then agreement between data x0 and H provides poor (or no) evidence for H

(“too cheap to be worth having” Popper)

O Such a test fails a minimal requirement for a stringent or severe test

O My account: severe testing based on error statistics (requires reinterpreting tests)

15

This alters the role of probability: typically just 2

O Probabilism. To assign a degree of probability, confirmation, support or belief in a hypothesis, given data x0

(e.g., Bayesian, likelihoodist)—with regard for inner coherencyO Performance. Ensure long-run reliability of

methods, coverage probabilities (frequentist, behavioristic Neyman-Pearson)

16

What happened to using probability to assess the error probing capacity by the severity

requirement?

O Neither “probabilism” nor “performance” directly captures it

O Good long-run performance is a necessary, not a sufficient, condition for severity

 

17

O Problems with selective reporting, cherry picking, stopping when the data look good, P-hacking, are not problems about long-runs—

O It’s that we cannot say the case at hand has done a good job of avoiding the sources of misinterpreting data

Key to revising the role of error probabilities

18

A claim C is not warranted _______

O Probabilism: unless C is true or probable (gets a probability boost, is made comparatively firmer)

O Performance: unless it stems from a method with low long-run error

O Probativism (severe testing) unless something (a fair amount) has been done to probe ways we can be wrong about C

19

O If you assume probabilism is required for inference, error probabilities are relevant for inference only by misinterpretation False!

O I claim, error probabilities play a crucial role in appraising well-testedness

O It’s crucial to be able to say, C is highly believable or plausible but poorly tested

20

Biasing selection effects:

O One function of severity is to identify problematic selection effects (not all are)

O Biasing selection effects: when data or hypotheses are selected or generated (or a test criterion is specified), in such a way that the minimal severity requirement is violated, seriously altered or incapable of being assessed

O Picking up on these alterations is precisely what enables error statistics to be self-correcting—

21

Nominal vs actual Significance levels

Suppose that twenty sets of differences have been examined, that one difference seems large enough to test and that this difference turns out to be ‘significant at the 5 percent level.’ ….The actual level of significance is not 5 percent, but 64 percent! (Selvin, 1970, p. 104)

From (Morrison & Henkel’s Significance Test controversy 1970!)

22

O Morrison and Henkel were clear on the fallacy: blurring the “computed” or “nominal” significance level, and the “actual” level

O There are many more ways you can be wrong with hunting (different sample space)

23

Spurious P-ValueYou report: Such results would be difficult to achieve under the assumption of H0

When in fact such results are common under the assumption of H0

(Formally):

O You say Pr(P-value ≤ Pobs; H0) ~ Pobs small

O But in fact Pr(P-value ≤ Pobs; H0) = high

24

Scapegoating

O Nowadays, we’re likely to see the tests blamed

O My view: Tests don’t kill inferences, people do

O Even worse are those statistical accounts where the abuse vanishes!

25

On some views, taking account of biasing selection effects “defies scientific sense”

Two problems that plague frequentist inference: multiple comparisons and multiple looks, or, as they are more commonly called, data dredging and peeking at the data. The frequentist solution to both problems involves adjusting the P-value…But adjusting the measure of evidence because of considerations that have nothing to do with the data defies scientific sense, belies the claim of ‘objectivity’ that is often made for the P-value” (Goodman 1999, p. 1010)

(To his credit, he’s open about this; heads the Meta-Research Innovation Center at Stanford)

26

Technical activism isn’t free of philosophyBen Goldacre (of Bad Science) in a 2016 Nature article, is puzzled that bad statistical practices continue even in the face of the new "technical activism”:

The editors at Annals of Internal Medicine,… repeatedly (but confusedly) argue that it is acceptable to identify “prespecified outcomes” [from results] produced after a trial began; ….they say that their expertise allows them to permit – and even solicit –undeclared outcome-switching

27

His paper: “Make journals report clinical trials properly”

O He shouldn’t close his eyes to the possibility that some of the pushback he’s seeing has a basis in statistical philosophy!

28

Likelihood Principle (LP)

The vanishing act links to a pivotal disagreement in the philosophy of statistics battles

In probabilisms, the import of the data is via the ratios of likelihoods of hypotheses

P(x0;H1)/P(x0;H0)

The data x0 are fixed, while the hypotheses vary

29

All error probabilities violate the LP (even without selection effects):

Sampling distributions, significance levels, power, all depend on something more [than the likelihood function]–something that is irrelevant in Bayesian inference–namely the sample space (Lindley 1971, p. 436) 

The LP implies…the irrelevance of predesignation, of whether a hypothesis was thought of before hand or was introduced to explain known effects (Rosenkrantz, 1977, p. 122)

30

Paradox of Optional Stopping:

Error probing capacities are altered not just by cherry picking and data dredging, but also via data dependent stopping rules:

Xi ~ N(μ, σ2), 2-sided H0: μ = 0 vs. H1: μ ≠ 0.

Instead of fixing the sample size n in advance, in some tests, n is determined by a stopping rule:

31

“Trying and trying again”

O Keep sampling until H0 is rejected at 0.05 level

i.e., keep sampling until M 1.96 s/√nO Trying and trying again: Having failed to

rack up a 1.96 s difference after 10 trials, go to 20, 30 and so on until obtaining a 1.96 s difference

32

Nominal vs. Actual significance levels again:

O With n fixed the Type 1 error probability is 0.05

O With this stopping rule the actual significance level differs from, and will be greater than 0.05

O Violates Cox and Hinkley’s (1974) “weak repeated sampling principle”

33

O “The ASA (p. 131) correctly warns that “[c]onducting multiple analyses of the data and reporting only those with certain p-values” leads to spurious p-values (Principle 4)

O They don’t mention that the same p-hacked hypothesis can occur in Bayes factors, credibility intervals, likelihood ratios

34

With One Big Difference:

O “The direct grounds to criticize inferences as flouting error statistical control is lost

O They condition on the actual data,

O Error probabilities take into account other outcomes that could have occurred but did not (sampling distribution)”

35

How might probabilists block intuitively unwarranted inferences

(without error probabilities)?

A subjective Bayesian might say:

If our beliefs were mixed into the interpretation of the evidence, we wouldn’t declare there’s statistical evidence of some unbelievable claim (distinguishing shades of grey and being politically moderate, ovulation and voting preferences)

36

Rescued by beliefsO That could work in some cases (it still

wouldn’t show what researchers had done wrong)—battle of beliefs

O Besides, researchers sincerely believe their hypotheses 

O So now you’ve got two sources of flexibility, priors and biasing selection effects

37

No help with our most important problem

O How to distinguish the warrant for a single hypothesis H with different methods

(e.g., one has biasing selection effects, another, pre-registered results and precautions)?

38

Most Bayesians use “default” priors

O Eliciting subjective priors too difficult, scientists reluctant to allow subjective beliefs to overshadow data

O Default, or reference priors are supposed to prevent prior beliefs from influencing the posteriors (O-Bayesians, 2006)

39

O “The priors are not to be considered expressions

of uncertainty, ignorance, or degree of belief. Default priors may not even be probabilities…” (Cox and Mayo 2010, p. 299)

O Default Bayesian Reforms are touted as free of selection effects

O “…Bayes factors can be used in the complete absence of a sampling plan…” (Bayarri, Benjamin, Berger, Sellke 2016, p. 100)  

40

Granted, some are prepared to abandon the LP for model testing

In an attempted meeting of the minds Andrew Gelman and Cosma Shalizi say:

O “[C]rucial parts of Bayesian data analysis, such as model checking, can be understood as ‘error probes’ in Mayo’s sense” which might be seen as using modern statistics to implement the Popperian criteria of severe tests.” (2013, p.10).

O An open question

41

The ASA doc highlights classic foibles that block replication

“In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result” (Fisher 1935, p. 14)

“isolated” low P-value ≠> H: statistical effect

42

Statistical ≠> substantive (H ≠> H*)

“[A]ccording to Fisher, rejecting the null hypothesis is not equivalent to accepting the efficacy of the cause in question. The latter...requires obtaining more significant results when the experiment, or an improvement of it, is repeated at other laboratories or under other conditions” (Gigerentzer 1989, pp. 95-6)

43

The Problem is with so-called NHST (“null hypothesis significance testing”)

O NHSTs supposedly allow moving from statistical to substantive hypotheses

O If defined that way, they exist only as abuses of tests

O ASA doc ignores Neyman-Pearson (N-P) tests

44

Neyman-Pearson (N-P) tests:

A null and alternative hypotheses H0, H1 that are exhaustive

H0: μ ≤ 12 vs H1: μ > 12

O So this fallacy of rejection HH* is impossible

O Rejecting the null only indicates statistical alternatives (how discrepant from null)

45

P-values Don’t Report Effect Sizes (Principle 5)

Who ever said to just report a P-value?

O “Tests should be accompanied by interpretive tools that avoid the fallacies of rejection and non-rejection. These correctives can be articulated in either Fisherian or Neyman-Pearson terms” (Mayo and Cox 2006, Mayo and Spanos 2006)

46

To Avoid Inferring a Discrepancy Beyond What’s Warranted:

large n problem.

O Severity tells us: an α-significant difference is indicative of less of a discrepancy from the null if it results from larger (n1) rather than a smaller (n2) sample size (n1 > n2 )

47

O What’s more indicative of a large effect (fire), a fire alarm that goes off with burnt toast or one so insensitive that it doesn’t go off unless the house is fully ablaze?

O [The larger sample size is like the one that goes off with burnt toast]

48

What About Fallacies of Non-Significant Results?

O They don’t warrant 0 discrepancy

O Use the same severity reasoning to rule out discrepancies that very probably would have resulted in a larger difference than observed- set upper bounds

O If you very probably would have observed a more impressive (smaller) p-value than you did, if μ > μ1 (μ1 = μ0 + γ), then the data are good evidence that μ< μ1

O Akin to power analysis (Cohen, Neyman) but sensitive to x0

49

O There’s another kind of fallacy behind a move that’s supposed improve replication but it confuses the notions from significance testing and it leads to “Most findings are false”

O Fake replication crisis.

50

Diagnostic Screening Model of Tests: urn of nulls

(“most findings are false”)

O If we imagine randomly select a hypothesis from an urn of nulls 90% of which are true

O Consider just 2 possibilities: H0: no effect H1: meaningful effect, all else ignored,

O Take the prevalence of 90% as Pr(H0 you picked) = .9, Pr(H1)= .1

O Rejecting H0 with a single (just) .05 significant result, Cherry-picking to boot

51

The unsurprising result is that most “findings” are false: Pr(H0| findings with a P-value of .05) > .5 Pr(H0| findings with a P-value of .05) ≠ Pr(P-value of .05 | H0)  Only the second one is a Type 1 error probability)Major source of confusion…. (Berger and Sellke 1987, Ioannidis 2005, Colquhoun 2014)

52

O A: Announce a finding (a P-value of .05)

O Not properly Bayesian (not even empirical Bayes), not properly frequentist

O Where does the high prevalence come from?

53

Concluding RemarkO If replication research and reforms are to lead to

error correction, they must correct errors: they don’t always do that

O They do when they encourage preregistration, control error probabilities & require good design

RCTs, checking model assumptions)

O They don’t when they permit tools that lack error control

54

Don’t Throw Out the Error Control BabyO Main source of hand-wringing behind the

statistical crisis in science stems from cherry-picking, hunting for significance, multiple testing

O These biasing selection effects are picked up by tools that assess error control (performance or severity)

O Reforms based on “probabilisms” enable rather than check unreliable results due to biasing selection effects

55

Repligate

O Replication research has pushback: some call it methodological terrorism (enforcing good science or bullying?)

O My gripe is that replications, at least in social psychology, should go beyond the statistical criticism

56

Non-replications construed as simply weaker effects

O One of the non-replications: cleanliness and morality: Does unscrambling soap words make you less judgmental?

“Ms. Schnall had 40 undergraduates unscramble some words. One group unscrambled words that suggested cleanliness (pure, immaculate, pristine), while the other group unscrambled neutral words. They were then presented with a number of moral dilemmas, like whether it’s cool to eat your dog after it gets run over by a car. …Chronicle of Higher Ed.

57

…Turns out, it did. Subjects who had unscrambled clean words weren’t as harsh on the guy who chows down on his chow.” (Chronicle of Higher Education)

O Focusing on the P-values ignore larger questions of measurement in psych & the leap from the statistical to the substantive.

HH*

O Increasingly the basis for experimental philosophy-needs philosophical scrutiny

58

The ASA’s Six PrinciplesO (1) P-values can indicate how incompatible the data are with

a specified statistical model

O (2) P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone

O (3) Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold

O (4) Proper inference requires full reporting and transparency

O (5) A p-value, or statistical significance, does not measure the size of an effect or the importance of a result

O (6) By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis

59

Test T+: Normal testing: H0: μ < μ0 vs. H1: μ > μ0 σ known (FEV/SEV): If d(x) is not statistically significant, then

μ < M0 + kεσ/√n passes the test T+ with severity (1 – ε)

 (FEV/SEV): If d(x) is statistically significant, then

μ > M0 + kεσ/√n passes the test T+ with severity (1 – ε)

  where P(d(X) > kε) = ε

60

ReferencesO Armitage, P. 1962. “Contribution to Discussion.” In The Foundations of Statistical

Inference: A Discussion, edited by L. J. Savage. London: Methuen.O Berger, J. O. 2003 'Could Fisher, Jeffreys and Neyman Have Agreed on Testing?' and

'Rejoinder,', Statistical Science 18(1): 1-12; 28-32.O Berger, J. O. 2006. “The Case for Objective Bayesian Analysis.” Bayesian Analysis 1 (3):

385–402. O Birnbaum, A. 1970. “Statistical Methods in Scientific Inference (letter to the Editor).”

Nature 225 (5237) (March 14): 1033.O Efron, B. 2013. 'A 250-Year Argument: Belief, Behavior, and the Bootstrap', Bulletin of the

American Mathematical Society 50(1): 126-46. O Box, G. 1983. “An Apology for Ecumenism in Statistics,” in Box, G.E.P., Leonard, T. and

Wu, D. F. J. (eds.), pp. 51-84, Scientific Inference, Data Analysis, and Robustness. New York: Academic Press.

O Cox, D. R. and Hinkley, D. 1974. Theoretical Statistics. London: Chapman and Hall. O Cox, D. R., and Deborah G. Mayo. 2010. “Objectivity and Conditionality in Frequentist

Inference.” In Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability, and the Objectivity and Rationality of Science, edited by Deborah G. Mayo and Aris Spanos, 276–304. Cambridge: Cambridge University Press.

O Fisher, R. A. 1935. The Design of Experiments. Edinburgh: Oliver and Boyd. O Fisher, R. A. 1955. “Statistical Methods and Scientific Induction.” Journal of the Royal

Statistical Society, Series B (Methodological) 17 (1) (January 1): 69–78.

61

O Gelman, A. and Shalizi, C. 2013. 'Philosophy and the Practice of Bayesian Statistics' and 'Rejoinder', British Journal of Mathematical and Statistical Psychology 66(1): 8–38; 76-80.

O Gigerenzer, G., Swijtink, Porter, T. Daston, L. Beatty, J, and Kruger, L. 1989. The Empire of Chance. Cambridge: Cambridge University Press.

O Goldacre, B. 2008. Bad Science. HarperCollins Publishers.O Goldacre, B. 2016. “Make journals report clinical trials properly”, Nature 530(7588);online

02Feb2016.O Goodman SN. 1999. “Toward evidence-based medical statistics. 2: The Bayes factor,”

Annals of Internal Medicine 1999; 130:1005 –1013.O Lindley, D. V. 1971. “The Estimation of Many Parameters.” In Foundations of Statistical

Inference, edited by V. P. Godambe and D. A. Sprott, 435–455. Toronto: Holt, Rinehart and Winston.

O Mayo, D. G. 1996. Error and the Growth of Experimental Knowledge. Science and Its Conceptual Foundation. Chicago: University of Chicago Press.

O Mayo, D. G. and Cox, D. R. (2010). "Frequentist Statistics as a Theory of Inductive Inference" in Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability and the Objectivity and Rationality of Science (D Mayo and A. Spanos eds.), Cambridge: Cambridge University Press: 1-27. This paper appeared in The Second Erich L. Lehmann Symposium: Optimality, 2006, Lecture Notes-Monograph Series, Volume 49, Institute of Mathematical Statistics, pp. 247-275.

O Mayo, D. G., and A. Spanos. 2006. “Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction.” British Journal for the Philosophy of Science 57 (2) (June 1): 323–357.

62

O Mayo, D. G., and A. Spanos. 2011. “Error Statistics.” In Philosophy of Statistics, edited by Prasanta S. Bandyopadhyay and Malcolm R. Forster, 7:152–198. Handbook of the Philosophy of Science. The Netherlands: Elsevier.

O Meehl, P. E., and N. G. Waller. 2002. “The Path Analysis Controversy: A New Statistical Approach to Strong Appraisal of Verisimilitude.” Psychological Methods 7 (3): 283–300.

O Morrison, D. E., and R. E. Henkel, ed. 1970. The Significance Test Controversy: A Reader. Chicago: Aldine De Gruyter.

O Pearson, E. S. & Neyman, J. (1930). On the problem of two samples. Joint Statistical Papers by J. Neyman & E.S. Pearson, 99-115 (Berkeley: U. of Calif. Press). First published in Bul. Acad. Pol.Sci. 73-96.

O Rosenkrantz, R. 1977. Inference, Method and Decision: Towards a Bayesian Philosophy of Science. Dordrecht, The Netherlands: D. Reidel.

O Savage, L. J. 1962. The Foundations of Statistical Inference: A Discussion. London: Methuen.

O Selvin, H. 1970. “A critique of tests of significance in survey research. In The significance test controversy, edited by D. Morrison and R. Henkel, 94-106. Chicago: Aldine De Gruyter.

O Simonsohn, U. 2013, "Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone", Psychological Science, vol. 24, no. 10, pp. 1875-1888.

O Trafimow D. and Marks, M. 2015. “Editorial”, Basic and Applied Social Psychology 37(1): pp. 1-2.

O Wasserstein, R. and Lazar, N. 2016. “The ASA’s statement on p-values: context, process, and purpose”, The American Statistician

63

AbstractIf a statistical methodology is to be adequate, it needs to register how “questionable research practices” (QRPs) alter a method’s error probing capacities. If little has been done to rule out flaws in taking data as evidence for a claim, then that claim has not passed a stringent or severe test. The goal of severe testing is the linchpin for (re)interpreting frequentist methods so as to avoid long-standing fallacies at the heart of today’s statistics wars. A contrasting philosophy views statistical inference in terms of posterior probabilities in hypotheses: probabilism. Presupposing probabilism, critics mistakenly argue that significance and confidence levels are misinterpreted, exaggerate evidence, or are irrelevant for inference. Recommended replacements–Bayesian updating, Bayes factors, likelihood ratios–fail to control severity.