Hypothesis Testing (Statistical Significance)

Hypothesis Testing

• Goal: Make statement(s) regarding unknown population parameter values based

on sample data

• Elements of a hypothesis test:

• Null hypothesis - Statement regarding the value(s) of unknown

parameter(s). Typically will imply no association between explanatory and

response variables in our applications (will always contain an equality)

• Alternative hypothesis - Statement contradictory to the null hypothesis

(will always contain an inequality)

• The level of significant (Alpha) is the maximum probability of committing a type I

error. P(type I error)= alpha

Definitions

Rejection (alpha, α) Region:

• Represents area under the curve that is used to reject the null hypothesis

Level of Confidence, 1 - alpha (a):

• Also known as fail to reject (FTR) region

• Represents area under the curve that is used to fail to reject the null hypothesis

FTR

H0

α/2α/2

1 vs. 2 Sided Tests• Two-sided test

• No a priori reason 1 group should have stronger effect

• Used for most tests

• Example

• H0: μ1 = μ2

• HA: μ1 ≠ μ2

• One-sided test

• Specific interest in only one direction

• Not scientifically relevant/interesting if reverse situation true

• Example

• H0: μ1 ≤ μ2

• HA: μ1 > μ2

Example: It is believed that the mean age of smokers in San Bernardino is 47.

Researchers from LLU believe that the average age is different than 47.

Hypothesis

H0: μ = 47

HA: μ ≠ 47

μ = 47

α /2 = 0.025Fail to Reject (FTR)

α /2 = 0.025

Three Approaches to Reject or Fail to Reject A Null Hypothesis:

1a. Confidence interval

Calculate the confidence interval

Decision Rule:

a. If the confidence interval (CI) includes the null, then the decision must be to

fail to reject the H0.

b. If the confidence interval (CI) does not include the null, then the decision

must be to reject the H0.

1b. Confidence interval to compare groups

Calculate the confidence interval for each group

Decision Rule:

a. If the confidence interval (CI) overlap, then

the decision must be to fail to reject the H0.

b. If the confidence interval (CI) do not include

the null, then the decision must be to reject

the H0.

2.Test Statistic Calculate the test statistic (TS) Obtain the critical value (CV) from the reference

table

Decision Rule:

a. If the test statistic is in the FTR region, then the decision must be to fail to reject the H0.

b. If the test statistic is in the rejection region, then the decision must be to reject the H0.

FTR

CV TS

Since the test statistic is in the rejection region, reject the H0

FTR

CV

Since the test statistic is in the fail to reject region, fail to reject the H0

TS CV

CV

3. P-Value

• Choose α

• Calculate value of test statistic from your data

• Calculate P- value from test statistic

Decision Rule:

a. If the p-value is less than the

level of significance, α, then the

decision must be to reject H0.

b. If the p-value is greater than

or equal to the level of

significance ,α, then the decision

must be to fail to reject H0.

FTR

CV TS

FTR

CV TS

P-value

P-value

Types of Errors!

Types of Errors

Truth

HypothesisTesting

Decision Based on

a Random Sample

1-α (Correct Decision)

Type II error (β)

Type I error (α) 1-β ( Power)(Correct Decision)

Fail to Reject H0

Reject H0

The Null Hypothesis(H0) is True

The Null Hypothesis(H0) is False

FTR

CV

H0 is True

ts

Since the H0 is true and we decide to accept it, we have thus made a correct decision

Correct Decision

FTR

ts

CV

H0 is True

Since the H0 is true and we decide to reject it, we have thus made an incorrect decision leading to Type I error

Alpha (α) Error

ts

FTR

CV

H0 is False

Since the H0 is False and we decide to reject it, we have thus made a correct decision

Power

FTR

ts

CV

H0 is False

Since the H0 is False and we decide to accept it, we have thus made an incorrect decision leading to type II error.

Beta, β, Error

Null Hypothesis

True

Fail to reject

Correct Decision

Reject

Type I

Error

False

Fail to Reject

Type II

Error

Reject

Correct

Decision

How to Reduce Errors

• Alpha error is reduced by increasing the confidence interval or reducing

bias

• Beta error is reduced by increasing the sample size

• Alpha and beta are inversely related

Example

ANOVA

GROUPS

763.000 2 381.500 31.918 .000

251.000 21 11.952

1014.000 23

Between Groups

Within Groups

Total

Sum ofSquares df Mean Square F Sig.

What type of error was possibly committed in the above example?

How would you reduce the error?