Hypothesis Testing. Outline The Null Hypothesis The Null Hypothesis Type I and Type II Error Type I and Type II Error Using Statistics to test the Null

Hypothesis Hypothesis TestingTesting

OutlineOutline

The Null HypothesisThe Null Hypothesis

Type I and Type II ErrorType I and Type II Error

Using Statistics to test the Null HypothesisUsing Statistics to test the Null Hypothesis

The Logic of Data AnalysisThe Logic of Data Analysis

Research Questions and Research Questions and HypothesesHypotheses

Research question:Research question: Non-directional:Non-directional:

No stated expectation about outcomeNo stated expectation about outcome Example:Example:

Do men and women differ in terms of Do men and women differ in terms of conversational memory?conversational memory?

Hypothesis:Hypothesis: Statement of expected relationshipStatement of expected relationship

Directionality of relationshipDirectionality of relationship Example:Example:

Women will have greater conversational memory Women will have greater conversational memory than menthan men

Grounding Hypotheses in Grounding Hypotheses in TheoryTheory

Hypotheses have an underlying rationale:Hypotheses have an underlying rationale: Logical reasoning behind the direction of the Logical reasoning behind the direction of the

hypotheses (theoretical rationale – hypotheses (theoretical rationale – explanation)explanation)

Why do we expect women to have better Why do we expect women to have better conversational memory? conversational memory?

Theoretical rationale based on:Theoretical rationale based on: 1. Past research1. Past research 2. Existing theory2. Existing theory 3. Logical reasoning3. Logical reasoning

The Null HypothesisThe Null Hypothesis

Null Hypothesis - the absence of a Null Hypothesis - the absence of a relationshiprelationship E..g., There is no difference between men’s and E..g., There is no difference between men’s and

women’s with regards to conversational women’s with regards to conversational memoriesmemories

Compare observed results to Null HypothesisCompare observed results to Null Hypothesis How different are the results from the null How different are the results from the null

hypothesis?hypothesis?

We do not propose a null hypothesis as We do not propose a null hypothesis as research hypothesis - need very large research hypothesis - need very large sample size / powersample size / power Used as point of contrast for testingUsed as point of contrast for testing

Hypotheses testingHypotheses testing

When we test observed results against null:When we test observed results against null: We can make two decisions:We can make two decisions:

1. Accept the null1. Accept the null No significant relationshipNo significant relationship Observed results similar to the Null HypothesisObserved results similar to the Null Hypothesis

2. Reject the null2. Reject the null Significant relationshipSignificant relationship Observed results different from the Null Observed results different from the Null

HypothesisHypothesis

Whichever decision, we risk making an errorWhichever decision, we risk making an error

Type I and Type II ErrorType I and Type II Error

1. Type I Error1. Type I Error Reality: No relationshipReality: No relationship Decision: Reject the nullDecision: Reject the null

Believe your research hypothesis have received Believe your research hypothesis have received support when in fact you should have disconfirmed support when in fact you should have disconfirmed itit

Analogy: Find an innocent man guilty of a crimeAnalogy: Find an innocent man guilty of a crime

2. Type II Error2. Type II Error Reality: RelationshipReality: Relationship Decision: Accept the nullDecision: Accept the null

Believe your research hypothesis has not received Believe your research hypothesis has not received support when in fact you should have rejected the support when in fact you should have rejected the null.null.

Analogy: Find a guilty man innocent of a crimeAnalogy: Find a guilty man innocent of a crime

Potential outcomes of Potential outcomes of testingtesting

DecisionDecision

Accept NullAccept Null Reject Reject NullNull

RR

EE No No

AA RelationshipRelationship

LL

II RelationshipRelationship

TT

YY

1 2

3 4


DecisionDecision


RR

EE No No


LL


TT

YY

Correct decision 2

3 4


DecisionDecision


RR

EE No No


LL


TT

YY

1 2

3 Correctdecision


DecisionDecision


RR

EE No No


LL


TT

YY

1 Type I Error

3 4


DecisionDecision


RR

EE No No


LL


TT

YY

1 2

Type II Error 4


DecisionDecision


RR

EE No No


LL


TT

YYType II Error Correct

decision

Type I Error Correctdecision

Function of Statistical Function of Statistical TestsTests

Statistical tests determine:Statistical tests determine: Accept or Reject the Null HypothesisAccept or Reject the Null Hypothesis

Based on probability of making a Type I Based on probability of making a Type I errorerror Observed results compared to the results Observed results compared to the results

expected by the Null Hypothesesexpected by the Null Hypotheses What is the probability of getting observed What is the probability of getting observed

results if Null Hypothesis were true?results if Null Hypothesis were true? If results would occur less than 5% of the If results would occur less than 5% of the

time by simple chance then we reject the time by simple chance then we reject the Null HypothesisNull Hypothesis

Start by setting level of Start by setting level of risk of making a Type I risk of making a Type I

ErrorError How dangerous is it to make a Type I Error:How dangerous is it to make a Type I Error:

What risk is acceptable?:What risk is acceptable?: 5%? 5%? 1%?1%? .1%? .1%?

Smaller percentages are more conservative in Smaller percentages are more conservative in guarding against a Type I Errorguarding against a Type I Error

Level of acceptable risk is called Level of acceptable risk is called “Significance level” :“Significance level” : Usually the cutoff - <.05Usually the cutoff - <.05

Conventional Significance Conventional Significance LevelsLevels

.05 level (5% chance of Type I Error).05 level (5% chance of Type I Error)

.01 level (1% chance of Type I Error).01 level (1% chance of Type I Error)

.001 level (.1% chance of Type I Error).001 level (.1% chance of Type I Error)

Rejecting the Null at the .05 level means:Rejecting the Null at the .05 level means: Taking a 5% risk of making a Type I ErrorTaking a 5% risk of making a Type I Error

Steps in Hypothesis Steps in Hypothesis TestingTesting

1. State research hypothesis1. State research hypothesis

2. State null hypothesis2. State null hypothesis

3.Set significance level (e.g., .05 level)3.Set significance level (e.g., .05 level)

4. Observe results4. Observe results

5. Statistics calculate probability of results 5. Statistics calculate probability of results if null hypothesis were trueif null hypothesis were true

6. If probability of observed results is less 6. If probability of observed results is less than significance level, then reject the nullthan significance level, then reject the null

Guarding against Type Guarding against Type I ErrorI Error

Significance level regulates Type I ErrorSignificance level regulates Type I Error

Conservative standards reduce Type I Conservative standards reduce Type I Error:Error: .01 instead of .05, especially with large .01 instead of .05, especially with large

samplesample

Reducing the probability of Type I Error:Reducing the probability of Type I Error: Increases the probability of Type II ErrorIncreases the probability of Type II Error

Sample size regulates Type II ErrorSample size regulates Type II Error The larger the sample, the lower the The larger the sample, the lower the

probability of Type II Error occurring in probability of Type II Error occurring in conservative testingconservative testing

Statistical PowerStatistical Power

The power to detect significant The power to detect significant relationshipsrelationships The larger the sample size, the more powerThe larger the sample size, the more power The larger the sample size, the lower the The larger the sample size, the lower the

probability of Type II Errorprobability of Type II Error

Power = 1 – probability of Type II ErrorPower = 1 – probability of Type II Error

Statistical AnalysisStatistical Analysis

Statistical analysis:Statistical analysis: Examines observed dataExamines observed data Calculates the probability that the results Calculates the probability that the results

could occur by chance (I.e., if Null was true)could occur by chance (I.e., if Null was true)

Choice of statistical test depends on:Choice of statistical test depends on: Level of measurement of the variables in Level of measurement of the variables in

question:question: Nominal, Ordinal, Interval or RatioNominal, Ordinal, Interval or Ratio

Logic of data analysisLogic of data analysis

Univariate analysisUnivariate analysis One variable at a time (descriptive)One variable at a time (descriptive)

Bivariate analysisBivariate analysis Two variables at a time (testing relationships)Two variables at a time (testing relationships)

Multivariate analysisMultivariate analysis More than two variables at a time (testing More than two variables at a time (testing

relationships and controlling for other relationships and controlling for other variables) variables)

VariablesVariables

Dependent variable:Dependent variable: What we are trying to predictWhat we are trying to predict E.g., Candidate preferenceE.g., Candidate preference

Independent variables:Independent variables: What we are using as predictorsWhat we are using as predictors E.g., Gender, Party affiliationE.g., Gender, Party affiliation

Testing hypothesis for Testing hypothesis for two nominal variablestwo nominal variables

VariablesVariables Null hypothesisNull hypothesisProcedureProcedure

GenderGender

Passing is notPassing is not Chi- Chi-squaresquare

related to genderrelated to gender

Pass/FailPass/Fail

Testing hypothesis for Testing hypothesis for one nominal and one one nominal and one

ratio variableratio variableVariablesVariables Null hypothesisNull hypothesis ProcedureProcedure

GenderGender

Score is notScore is not T-T-testtest

related to genderrelated to gender

Test scoreTest score

Testing hypothesis for Testing hypothesis for one nominal and one one nominal and one

ratio variableratio variableVariableVariable Null hypothesisNull hypothesis ProcedureProcedure

Year in schoolYear in school

Score is notScore is not

related to year inrelated to year inANOVAANOVA

schoolschool


Can be used when nominal variable has more than two Can be used when nominal variable has more than two categories and can include more than one independent categories and can include more than one independent variablevariable

Testing hypothesis for Testing hypothesis for two ratio variablestwo ratio variables

VariableVariable Null hypothesisNull hypothesisProcedureProcedure

Hours spentHours spent

studyingstudying Score is notScore is not

related to hoursrelated to hours Correlation Correlation

spent studyingspent studying


Testing hypothesis for Testing hypothesis for more than two ratio more than two ratio

variablesvariablesVariableVariable Null hypothesisNull hypothesis ProcedureProcedure

Hours spentHours spent

studying studying Score is positivelyScore is positively

related to hoursrelated to hours

Classes Classes spent studying andspent studying and Multiple Multiple

missed missed negatively relatednegatively related regression regression

to classes missedto classes missed


Commonality across all Commonality across all statistical analysis statistical analysis

proceduresprocedures Set the significance level:Set the significance level:

E.g., .05 levelE.g., .05 level Means that we are willing to conclude that there Means that we are willing to conclude that there

is a relationship if:is a relationship if: Chance of Type I error is less than 5%Chance of Type I error is less than 5%

Statistical tests tell us whether:Statistical tests tell us whether: The observed relationship has less than a 5% The observed relationship has less than a 5%

chance of occurring by chancechance of occurring by chance

Summary of Statistical Summary of Statistical ProceduresProcedures

Variables Procedure

Nominal IV, Nominal DV Chi-square

Nominal IV, Ratio DV T-test

Multiple Nominal IVs, Ratio DV

ANOVA

Ratio IV, Ratio DV Pearson’s R

Multiple Nominal IVs, Ratio DV with ratio covariates

ANCOVA

Multiple ratio Multiple Regression

Documents

Hypothesis Testing. Outline The Null Hypothesis The Null Hypothesis Type I and Type II Error Type I and Type II Error Using Statistics to test the Null