CSE 5331/7331Fall 2011

P-Value and Statistical Significance

Margaret H. DunhamDepartment of Computer Science and Engineering

Southern Methodist University

.CSE 5331/7331 F‘11 1

Outline

Overview P-Value Statistical Significance Test Statistics Examples

2CSE 5331/7331 F‘11

Outline

Overview P-Value Statistical Significance Test Statistics Examples

3CSE 5331/7331 F‘11

Data Mining - Remember When we analyze data we may only be

dealing with a sample of the complete data (which may be infinite)

We often want to generalize our findings to the entire data population

Calculation of a P-value can help us to determine how likely our results are to apply to the entire population

How do we do this?

4CSE 5331/7331 F‘11

Normal Distribution

5CSE 5331/7331 F‘11

Will G. Hopkins, “A New View of Statistics; P Values and Statistical Significance,” 2002, http://sportsci.org/resource/stats/pvalues.html

But what are we comparing?

Two different results:– Employment rate for MSCS vs MBA– Scores on a standardized test between

male and female or USA and another country.

– Classification by two different classifiers Sometimes a correlation coefficient or a test

statistic whose distribution is known (such as chi squared) is examined.

6CSE 5331/7331 F‘11

Correlation Coefficient When we look at Statistical Significance we will be

comparing two values. The figure below shows the probability coefficient values in a sample of size 20 when the correlation is 0 (that is no correlation).*

7CSE 5331/7331 F‘11

* Will G. Hopkins, “A New View of Statistics,” 2002,

http://www.sportsci.org/resource/stats/pvalues.html

Statistics Assumptions

Sample data is representative of the entire population

Sample data is randomly chosen These assumptions often (usually) do not

hold for a given real world sample (training data). – Real world data format is unknown.– May wish to extrapolate to a larger population

8CSE 5331/7331 F‘11

Hypotheses

Alternative Hypothesis (HA): This is the relationship between the variables that you expect (hope) your experiments will demonstrate.

Null Hypothesis (H0). This is just the opposite relationship. There is no relationship between the variables.

In significance testing we really determine whether we should reject the null hypothesis.

9CSE 5331/7331 F‘11

Outline

Overview P-Value Statistical Significance Test Statistics Examples

10CSE 5331/7331 F‘11

P-Value

The probability that a variable has a value greater than the observed value

http://en.wikipedia.org/wiki/P-value http://sportsci.org/resource/stats/pvalue

s.html

11CSE 5331/7331 F‘11

P-Value “The probability that a variate would assume

a value greater than or equal to the observed value strictly by chance.”*

“the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.”**

Smaller the value the better (You want to reject the Null Hypothesis.

* Weisstein, Eric W. "P-Value." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/P-Value.html , 9/21/11.

** Wikipedia, “P-value”, http://en.wikipedia.org/wiki/P-value, 9/19/11.

/12CSE 5331/7331 F‘11

Finding P-value

May be able to calculate P-value directly or you may have to convert the data (by using a test statistic) into a value that can be used.

Example: Find correlation between two variables. The correlation coefficient can be used directly if we know its distribution.

However, in some cases a new statistic is used to provide that P-value.

13CSE 5331/7331 F‘11

Another Way to Look at It

P-value is a measure of how much evidence you have against the Null Hypothesis.

Null Hypothesis: Hypothesis of no change Critical Regions – values of statistics for

which, if they occur you will reject the Null Hypothesis

14CSE 5331/7331 F‘11

Confidence Intervals

Range of values that contains the true parameter.

Size depends on sample size and variance. Larger sample size, smaller interval If the variance is large, larger interval.

15CSE 5331/7331 F‘11

P-Value and Confidence Intervals Confidence Intervals and P-Values are related. However, 95% confidence interval

is not the same thing as P=0.05 It is the same as a P-value of 0.05

“that doesn’t overlap zero.” [3] “If the 95% CI includes no

difference between groups, then

the P values is > 0.05.” [1]

“If the 95% CI does not include

no difference between groups

then the P value is < 0.05.” [1]16CSE 5331/7331 F‘11

* Will G. Hopkins, “A New View of Statistics,” 2002,

http://www.sportsci.org/resource/stats/pvalues.html

95% confidence intervals

Outline

Overview P-Value Statistical Significance Test Statistics Examples

17CSE 5331/7331 F‘11

So …

So, if we obtain a data value from a population distribution that is normal AND we know that it occurs in one of the 0.2% wings of the distribution, than we should be somewhat convinced that that value is not as likely (although still possible) to have seen as the more commonly occurring data values.

How sure are you that the results you’ve found in the experiments are actually true???

18CSE 5331/7331 F‘11

Statistical Significance Way to assign a confidence value to a finding. Probability that a finding is true in the general

population and not a fluke (not due to chance or random).

A significance level is a measure as to how likely the result is due to chance.

P-value is a test or measure of statistical significance. What is probability that relationship between two

variables exists? What is the probability that this relationship (our

experiments seem to indicate exists) is due only to random chance?

19CSE 5331/7331 F‘11

Using Significance Test State Alternative Hypothesis State Null Hypothesis. Perform research Identify statistic and its distribution Decide on alpha threshold Calculate statistic Calculate P-Value Compare P-Value to threshold

– If lower, probability is small that the result was by chance therefore the finding is significant

– If higher, then finding is not significant

20CSE 5331/7331 F‘11

Probability of Error

Alpha Level (Threshold) Threshold value to make decision Rule of thumb: 0.05 Since you hope to reject the Null Hypothesis,

then you hope you’ll find p < 0.05. This means you are beyond (outside in the

distribution) the Alpha level and your chance of a Type I error is acceptable.

21CSE 5331/7331 F‘11

Significance Levels Rule of thumb: 0.95 – The result has a 95% chance

of being true. However, results are being stated in terms of not

being true. (i.e. P-value = 1 – Level )So then a 0.05 value is good.

So p<0.005 indicates a significant result. The smaller the p value the better. P=0.001 is better

than p=0.05. If we know what the distribution of the statistic is,

then we can estimate the extreme areas outside (furthest from the mean). This can tell us the significance

22CSE 5331/7331 F‘11

Sample P-Value Levels

Suppose your threshold is P=0.05

Table from GraphPad, “Interpreting statistical significance,” 1999, http://www.graphpad.com/articles/interpret/principles/stat_sig.htm .

23CSE 5331/7331 F‘11

P-Value Wording

> 0.05 Not Significant

0.01 to 0.05 Significant

0.001 to 0.01 Very Significant

< 0.001 Extremely Significant

DM &Significance Tests

In MOST data mining experiments we want to determine if the results we found in the sample can be generalized to the general population.

So the two variables being compared are the one found in the sample and the one that should be found in the population (if we could even get the entire population set).

24CSE 5331/7331 F‘11

Two-Tailed vs One-Tailed Whether the test assumes both sides of the

statistics distribution If direction of difference or relationship is

important than a one-tailed probability is include – otherwise two-tailed.

25CSE 5331/7331 F‘11

Will G. Hopkins, “A New View of Statistics; P Values and Statistical Significance,” 2002, http://sportsci.org/resource/stats/pvalues.html

Types of Errors Type I: Conclude relationship exists when in fact it

doesn’t and evidence shows that it doesn’t. – Null hypothesis should be accepted. – Alpha error

Type II: Conclude relationship does not exist when in fact it does and the evidence shows it does.– Null hypothesis should be rejected.– Beta error

To be on safe side, try to minimize Type I errors. – Use high Alpha probability– Alpha level is the probability of making an error that you are

willing to accept.– Often Alpha=0.05 or even 0.01.

26CSE 5331/7331 F‘11

Outline

Overview P-Value Statistical Significance

Test Statistics Examples

27CSE 5331/7331 F‘11

Test Statistics

So how do we calculate P-Value? We need to convert the raw data

generated by experiments into a single value which we know follows some known frequency distribution.

We can then calculate the P-Value based on that

These are called Test Statistics

28CSE 5331/7331 F‘11

Standard Test Statistics

Common Test Statistics

http://en.wikipedia.org/wiki/Statistical_hypothesis_testing T – Compares distributions of means of two groups

http://www.socialresearchmethods.net/kb/stat_t.php

http://mathworld.wolfram.com/Studentst-Distribution.html

http://www.stattools.net/tTest_Tab.php Conversion between

http://www.graphpad.com/quickcalcs/pvalue1.cfm

29CSE 5331/7331 F‘11

Chi Square Test Experiments may actually general nominal or

ordinal data. How do you convert these values into

numbers that can be evaluated? Often a Chi Square Test is used. Here the

collected data is summarized and then converted into a statistic that can be used to measure how the values compare to what would be expected randomly.

30CSE 5331/7331 F‘11

Chi Square Distribution

Not normal but approaches normal as degrees of freedom (number of indepdent variables) increases.

http://stattrek.com/lesson3/chisquare.aspx

31CSE 5331/7331 F‘11

Chi Square Test Process

Complete contingency table based on two subset types and possible values.

Even though actual values are not numeric, when we complete the table we then have counts (frequencies) which are numeric and can be analyzed.

The Chi Squared Statistic is calculated by comparing the observed frequency values to the expected frequency values.

This statistic is what is examined for significance. It follows a chi squared distribution. Note: http://epm.sagepub.com/content/52/1/57.short

32CSE 5331/7331 F‘11

Chi Squared Example Suppose we compare the hiring rate for recent MS

CS graduates to MBA graduates. We find that 150 out of 200 MS students have a job and 100 out of 200 MBA students have a job.

Hypothesis: MSCS students are hired at a higher rate than MBA students.

Null Hypothesis: MBA and MSCS students are hired at the same rate.

Suppose we use p=0.05 as significance level.

33CSE 5331/7331 F‘11

Chi Squared Example (cont’d)

250 students have jobs so 250/400=0.625 150 do not have jobs, so 150/400=0.375 We would expect these percentages to hold for each

population if the Null Hypothesis is true. 150/200=0.75 MSCS have jobs 100/200=0.5 MBA have jobs We would expect 0.625 * 200 = 125 in each group to

have jobs34CSE 5331/7331 F‘11

MSCS MBA Total

Job 150 100 250

No Job 50 100 150

Total 200 200 400

Chi Squared Example (cont’d)

Calculate Chi Square

http://www.graphpad.com/quickcalcs/contingency1.cfm Chi Square value is 26.67 How do we convert this to a P-Value? http://www.statsoft.com/textbook/distribution-tables/#chi P-Value is <0.05. Reject Null Hypothesis Yes it is significant

(Actually p<0.0001 so it is extremely significant)

35CSE 5331/7331 F‘11

http://course1.winona.edu/sberg/Equation/chi-squa.gif

Outline

Overview P-Value Statistical Significance Test Statistics

Examples

36CSE 5331/7331 F‘11

Example 1 – Iris Data

J48 Default Cross Validation We have the Kappa statistic but can’t use it to calculate P-value

without variance.

http://twiki.org/p/pub/Main/SigurdurRunarSaemundsson/Interrater_agreement.Kappa_statistic.pdf

37CSE 5331/7331 F‘11

=== Summary ===

Correctly Classified Instances 144 96 %Incorrectly Classified Instances 6 4 %Kappa statistic 0.94 Mean absolute error 0.035 Root mean squared error 0.1586Relative absolute error 7.8705 %Root relative squared error 33.6353 %Total Number of Instances 150

Example 1 – Iris Data

38CSE 5331/7331 F‘11

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure ROC Area Class 0.98 0 1 0.98 0.99 0.99 Setosa 0.94 0.03 0.94 0.94 0.94 0.952 Versicolor 0.96 0.03 0.941 0.96 0.95 0.961 VerginicaWeighted Avg. 0.96 0.02 0.96 0.96 0.96 0.968

=== Confusion Matrix ===

a b c <-- classified as 49 1 0 | a = Setosa 0 47 3 | b = Versicolor 0 2 48 | c = Verginica

View confusion matrix as contingency Table andCalculate Chi Squared

Example 1 (cont’d)

http://faculty.vassar.edu/lowry/newcs.html Chi Square = 266 P < 0.0001 Extremely statistically significant

39CSE 5331/7331 F‘11

Example 2

Nektarios Leontiadis, Tyler Moore and Nicolas Christin. "Measuring and Analyzing Search-Redirection Attacks in the Illict Online Prescription Drug Trade". 20th USENIX Security Symposium. August 10-12, 2011: San Francisco, Ca.

http://cs.wellesley.edu/~tmoore/usenix11.pdf

40CSE 5331/7331 F‘11

References

1. American College of Physicians- American Society of Internal Medicine, “Primer on 95% Confidence Intervals,” Effective Clinical Practice, September/October 2001, Vol 4, No 6, pp 229-231.

2. Will G. Hopkins, “A New View of Statistics,” 2002, http://www.sportsci.org/resource/stats/pvalues.html

3. M. A. Saint-Germain, “PPA 696 Research Methods,” 1/1/01, http://www.csulb.edu/~msaintg/ppa696/696menu.htm

41CSE 5331/7331 F‘11