Embed Size (px)
Citation preview
1
1
Empirical Research Methods in Information Science
IS 4800 / CS 6350
Lecture 42 Correlation
Today
n Review logic of hypothesis testing n Review Chi-square goodness of fit n Review Chi-square test for independence n Correlation
n Pearson n Point-biserial n Spearman rank-order
2
2
A common scenario
3
IV
Software A vs.
Software B
DV
Efficiency (time to complete)
Sampling Error Measurement Error
Demand Effects etc.
• We measure difference A-B • Is this real? Or just chance? • We have very good models of randomness
(noise). • We can compute p(observed difference A-B is due only to chance variation) • When should we conclude A-B is real (H1)? • What can we conclude otherwise (H0)?
4
Basic Process of Hypothesis Testing
n H1: Research Hypothesis: n Population 1 is different than Population 2
n H0: Null Hypothesis: n No difference between Pop 1 and Pop 2 n The difference is “null”
n Compute p(observed difference|H0) n ‘p’ = probability observed difference is due to
random variation n If p<threshold then reject H0 => accept H1
n p typically set to 0.05 for most work n p is called the “level of significance”
3
5
Chi-Square for Goodness of Fit
n Assumes 1. You have a nominal variable
n Values are exhaustive & mutually-exclusive
2. You have an Expected Frequency table for the nominal variable
3. None of the expected frequencies are “too small” (≥5)
4. Random sampling
6
Chi-Square for Goodness of Fit
n Form of null hypothesis H0? n Observed frequency = Expected frequency n Populations (expected, observed) are
actually the same on the nominal measure of interest
n Form of hypothesis H1? n Observed frequency ≠ Expected frequency n Populations (expected, observed) are
different
4
7
Chi-Square Test for Independence
n Are two variables related, or are they independent?
n Assumptions n Both variables must be nominal. n Cannot be related in a ‘special’ way (i.e.,
repeated measures) n Random sampling assumed
8
Example from chapter
n Morning & night people using different modes of transportation.
n What kind of study is this?
Bus Carpool Own CarMorning 60 30 30Night 20 20 40
5
9
Expected frequencies if variables are independent
n E = (R x C)/N for each cell n R = row count n C = column count n N = total number in all cells
Bus Carpool Own CarMorning 60 30 30Night 20 20 40
10
Expected frequencies if variables are independent
n Step 1 – compute row & col totals
120 80
Bus Carpool Own CarMorning 60 30 30Night 20 20 40
80 50 70
6
11
Expected frequencies if variables are independent n Step 1 – compute row & col totals n Step 2 – ea cell = (R x C)/N
120 80
Bus Carpool Own CarMorning 60 30 30Night 20 20 40
80 50 70
(48)
(32) (30) (20)
(42) (28)
12
Formula
n Same as goodness-of-fit test.
n df = (NumRows-1)x(NumColumns-1)
∑−
=ΧEEO 2
2 )(
7
13
Group Exercise
n For each problem, write 1. What kind of study design is it? 2. Two populations being compared 3. Research hypothesis 4. Null hypothesis 5. Test criteria 6. Expected frequencies 7. Observed frequencies 8. Test results
n publication format and n English
df 1 2 3 4 5
cutoff 3.84 5.99 7.82 9.49
11.07
Cutoff for α=.05
Correlation!
15
8
Quiz
16
Measure A
Mea
sure
B
r = ______
Measure A
Mea
sure
B
r = ______
Measure A
Mea
sure
B
r = ______
Measure A
Mea
sure
B
r = ______
17
Correlation?
All r=0.81
9
19
Predictions Using Correlations
n Predictor vs. Criterion (Dependent) Variable
n Can you assume directionality?
n Depends entirely on your study design.
20
Pearson Correlation Coefficient
n Assumptions 1. Two interval (or ratio) measures. 2. Not an obviously curvilinear relationship. 3. Both populations normally distributed*.
*Unimodal and symmetric frequency distributions. Most important if doing a significance test.
10
21
Formula for Pearson correlation
1−=∑
NZZ
r YX
))(())([(
YX
YX
SSSSMYMX
r ∑ −−=
-OR-
Where
∑ −= 2)( XX MXSS
∑ −= 2)( YY MYSS
23
Procedure for Hypothesis Testing with Correlations
n Populations being compared: n Test: The population from which the
observed sample was drawn. n Comparison: A hypothetical population in
which the variables are unrelated, i.e., have a correlation of zero.
11
24
Procedure for Hypothesis Testing with Correlations
n Form of hypothesis H1? n The correlation in the observed population is
different from a population in which the correlation is zero.
n Unlikely we would have obtained a correlation this big if the variables actually were unrelated.
n Form of null hypothesis H0? n The correlation in the observed population is the
same as a population in which the correlation is zero.
25
Procedure for Hypothesis Testing with Correlations n Heuristic threshold for α=0.05:
n Exact form given in Aron.
Nr 2>
12
27
Procedure for Hypothesis Testing with Correlations
n R: n cor.test(v1,v2) n See if significance<threshold
n Yes => reject H0 n No => inconclusive
n Manually: n Compute r n Is
n If yes => reject H0 n If no => inconclusive
Nr 2>
28
Reporting results Where,
n sigthresh = pre-defined significance threshold n Note: if p<<sigthresh, can report that as well, e.g., “p<.
01”, “p=.001”
For example: If not significant, than use “n.s.” instead of
“p<…”.
r=val, p<sigthresh
r=0.82, p<.05
13
29
Example
Employee Sue Sam Sid Sal SierraProductivity 8 5 7 12 22Monitor Size 17 15 21 19 24
30
Example Continued
0
5
10
15
20
25
30
0 5 10 15 20 25
Productivity
Mon
itor S
ize
14
Example
31
Employee Sue Sam Sid Sal SierraProductivity 8 5 7 12 22Monitor Size 17 15 21 19 24
P = 10.8 (6.8), M = 19.2 (3.5)
-0.414 -0.858 -0.562 0.178 1.657-0.63 -1.202 0.515 -0.057 1.374Z scores
Z score products
0.260881 1.031666 -0.28968 -0.01016 2.276781
r = sum / (N-1) = +0.82 Nr 2>
32
Pearson Correlation coefficient
n Which of the following is it appropriate for? n Descriptive study designs n Demonstration study designs n Correlational study designs n Experimental study designs
15
33
Group Exercise
34
Group Exercise
n For each problem, write 1. Two populations being compared 2. Research hypothesis 3. Null hypothesis 4. Test criteria 5. Scatter plot 6. r (if appropriate) 7. Hypothesis test results
n publication format and n English
1−=∑
NZZ
r YX
Nr 2>
16
Pearson Correlation in R
#For vectors v1, v2
#Just ‘r’
cor(v1,v2)
#Hypothesis test (including r)
cor.test(v1,v2)
35
36
Example Correlation Matrix
17
37
Example Correlation Matrix Morrow, et al ’96, Medication Instruction Design
38
Comparing r’s
n If you want to make statements about how large one correlation is relative to another. n e.g. one is twice as large as another
n Don’t compare r’s directly… n Compare r^2 (“proportionate reduction
in error”)
18
39
Other measures of association
n Point-biserial n One numeric & one binary (nominal) measure n Just dummy code the nominal and use Pearson
correlation.
n Spearman Rank Order (rho) n Two ordinal measures (or for transformed numeric
measures if non-linear) n Replace each value with its rank order n Compute Pearson correlation with ranks n Measures degree of monotonicity
Two meanings of ‘correlation’
n Correlation statistic vs. n Correlational research model
40
19
Example: Leashes & Attachment
41
Example: Leashes & Attachment
n You want to see if toddlers who grow up leashed have better attachment scores.
n You recruit 30 parents of toddlers, and randomly give half of them leashes and sign contracts agreeing to leash their toddler every time they leave the house.
n After one year you administer the strange situation protocol to classify the toddler attachment as secure, avoidant, or resistant.
n What kind of study is this? n What statistic would you use to evaluate results? n What is df? n Assuming X^2(df) = 32.4, what would you conclude? n Assuming X^2(df)=0.2, what would you conclude?
42
20
Example: Net Latency & Satisfaction
n CCIS wants to save money by switching to slower wireless routers, and wants to assess the impact this will have on student satisfaction. You want to know how slow things have to get before students start complaining.
n You have the crew implement a program that randomly chooses a network latency (between 0s and 10s) every time a student logs into CCIS wireless, then adds that latency to every network access from them. After 10 minutes of use a web form pops up asking students to rate their degree of satisfaction with CCIS systems (10-item).
n What kind of study is this? n What statistic would you use to evaluate results? n What is df? n Assuming r = -0.8, p=.021, what would you conclude? n Assuming r = 0.1, p=.342, what would you conclude? 43
What do you do if your data is clearly not unimodal & symmetric OR there is a clear non-linear relationship?
44
21
Sample Data for Spearman rho
45
Monitor Size 21 24 17 19 15Productivity 3 1 2 5 0
0123456
12 17 22 27
rho = 0.3
46
Spearman measures degree of
monotonicity.
22
Parametric vs. Non-parametric Statistics
n non-parametric statistics that do not rely on data belonging to any particular distribution
n E.g., Pearson r is a parametric statistic (assumes underlying distributions are normal – can be described using parameters – mean & stddev)
n E.g., Spearman rho is non-parametric 47
48
Intro to Power & Effect Size
23
49
A Brief Note About Power
n The “power” of a statistical test is its ability to detect differences in data that are inconsistent with the null hypothesis. n p(rejecting H0|H1) n aka Concluding H1, given that H1 is
actually true.
50
Relationship between alpha, beta, and power.
Correct p = power
Type I err p = alpha
Type II err p = beta
Correct p = 1-alpha
H1 True H1 False
“The Truth”
Decide to Reject H0 & accept H1
Do not Reject H0 & do not accept H1
24
51
Effect size n The amount of change in the DVs seen. n Can have statistically significant test but small effect
size.
n IF your test yields significance THEN you are usually interested in how big the effect is. (If not significant, effect size is irrelevant.)
n Different for each statistical test. n “Cohen” provides qualitative categories
n small, medium, large n Example, for r: 0.1=small, 0.3=medium, 0.5=large
52
Power Analysis
n Power n Increases with effect size n Increases with sample size n Increases with alpha (decreases as you make the
criteria more stringent)
n Should determine number of subjects you need ahead of time by doing a ‘power analysis’
n Standard procedure: n Fix alpha and beta (power) n Estimate effect size from prior studies n Determine number of subjects you need
25
Power & Effect Size for Correlation
n Effect size = |r| n Power, see table 11-7, pg 465 Aron
n Usually, given n Expected effect size n Test criteria
n Desired significance level (usually 0.05) n Desired power (usually 0.8) n Directionality of test
n Want to determine how many samples (ie, bodies) you need
n What happens if you use fewer? 53
54
26
Table 11-8, Aron
55
Effect size
Medium (r=0.3)
Two-tailed…… 85
Approximate number of participants needed for 80% power for a study using the correlation coefficient (r) for testing a hypothesis at the .05 significance level
Small (r=0.1)
Large (r=0.5)
783 28
Effect Size & Power for X2
n Completely different formulas than for Pearson r.
n Dependent on df. n See Aron Tables 13-9, 13-10
56
27
57
58
Homework
n Read B&A Ch 10 to 300 n Read Aron Ch 3 & 4
n Or remedial stats: normal curve, 1-sample t-tests
n Start on Homework I11 – Study Design & Analysis
n Work individually on this one. n Part I – Write a complete study proposal n Part II – Analyze data and write report
