Mk Select Right Stat

8/10/2019 Mk Select Right Stat

1/34

Selecting Right StatisticsSelecting Right Statistics

HyungjinHyungjin Myra Kim,Myra Kim, Sc.DSc.D..

The University of MichiganThe University of Michigan


2/34

Choosing an Analytic Method (1)Choosing an Analytic Method (1)

First, analytic plan should be considered whileFirst, analytic plan should be considered while

planning the study.planning the study. What do you plan to study (or measure)?What do you plan to study (or measure)?

Primary outcome measure determines the typePrimary outcome measure determines the type

of dependent variableof dependent variable

Continuous (ex: hours of sleep)Continuous (ex: hours of sleep)

Dichotomous (ex: binge drinking or not)Dichotomous (ex: binge drinking or not) Ordinal (ex: depression diagnosis)Ordinal (ex: depression diagnosis)

Categorical (ex: choice of treatment)Categorical (ex: choice of treatment)

Time to event (ex: time to relapse)Time to event (ex: time to relapse)


3/34

Choosing an Analytic Method (Choosing an Analytic Method (22))

Sometimes, there is no dependent variableSometimes, there is no dependent variable

Factor analysisFactor analysis

Cluster analysisCluster analysis

HigherHigher--way contingency table analysesway contingency table analyses

Agreement (kappa)Agreement (kappa)

CorrelationCorrelation analysis (correlation coefficient)analysis (correlation coefficient)

AAccuracyccuracy ((sensitivity, specificitysensitivity, specificity))

(We will not discuss the above today.)(We will not discuss the above today.)


4/34

Choosing an Analytic Method (Choosing an Analytic Method (33))

Study designStudy design

Do you have a primary comparison?Do you have a primary comparison?

Determines the nature of the primary predictorDetermines the nature of the primary predictorvariable (independent variable)variable (independent variable)

(ex) 2 group or 3 group comparison?(ex) 2 group or 3 group comparison?

(ex) Evaluating the relationship between happiness(ex) Evaluating the relationship between happinessto ratio of leisure to work hoursto ratio of leisure to work hours

How often do you plan to measure?How often do you plan to measure? XX--sectional, longitudinal,sectional, longitudinal,xx--overover

DeterminesDetermines the number of dependent variablesthe number of dependent variables

(ex) pre/post has measurements twice per person(ex) pre/post has measurements twice per person


5/34

The choice of analysis will also depend onThe choice of analysis will also depend on

UnvariateUnvariate vs.vs. bivariatebivariate analysisanalysis

BivariateBivariate vs. multivariate analysisvs. multivariate analysis

Potential confounder?Potential confounder? Adjust for covariates?Adjust for covariates?

Data skewed or sample size small?Data skewed or sample size small? TransformationTransformation

Parametric vs. nonParametric vs. non

--parametriparametri

cc


6/34

Dependent Variable (Outcome)

Study Designs Continuous Binary (yes/no)

Pre/Post Effect of nightly exercise on hrsof sleep before/after in

insomniacs

Patient satisfaction before vs.

after color change in hospital

ward

Matched pairs Mastectomy vs. Lumpectomy onQOL in patients matched by age

& family history

Mastectomy vs. Lumpectomy on

survival in patients matched by

age & family history

1-group Cholesterol in diabetic patients:Is it higher than general public? Depression in substanceabusers

2-group Writing skill between teachingmethods A vs. B

Comparison of drugs A vs. B onrelapse to heavy drinking

2-group,pre/post

Weight before/after in exercisevs. no exercise group

Satisfaction before & afterbetween 2 skin products

3-group Comparing effectiveness of threedrugs on cholesterol

Pain reduction in three

different pain relief medication

ContinuousPredictor

Does pack-year of smoking

predict Cognitive deficit?

Is average nightly sleep

predictive of hair loss?


7/34

What Type of Analysis?What Type of Analysis?

DescriptiveDescriptive

NumericalNumerical tablestables of means, counts,of means, counts,proportionproportion

GraphicalGraphical -- histogramshistograms, box plots, s, box plots, scattercatter

plotsplots,, etc.etc.

InferentialInferential

EstimationEstimation Point estimates/ConfidencePoint estimates/ConfidenceIntervalsIntervals

Hypothesis TestsHypothesis Tests


8/34

Analytic MethodsAnalytic Methods

Dependent Variable (Outcome) Type

Study DesignContinuous

(multiple regression for

multivariate analysis)

Binary (yes/no)

(logistic regression for

multivariate analysis)Pre/Post Paired t-test McNemars test

Matched pairs Paired t-test McNemars test

1-group One-group t-test One proportion test

2-group* Two-group t-test Two proportion test orChi-square Test

2-group, pre/post* Analysis of Covariance or

multiple regression

Repeated measures

logistic regression

3-group* Analysis of Variance Chi-square test

Continuous

Predictor

Simple regression Logistic regression

** BivariateBivariate relationshipsrelationships


9/34

Binary Dependent VariableBinary Dependent Variable

Descriptive Statistics: ProportionDescriptive Statistics: Proportion

To estimate a proportion or prevalence, subjects mustTo estimate a proportion or prevalence, subjects mustbe a representative sample from the population.be a representative sample from the population.

Assuming the subjects are representative andAssuming the subjects are representative andindependent, the rate is estimated as:independent, the rate is estimated as:

p = n/Np = n/N

where n is the number of subjects with the attributewhere n is the number of subjects with the attributeand N is the total number of subjects tested (orand N is the total number of subjects tested (orstudied).studied).


10/34

Binary Dependent Variable (2)Binary Dependent Variable (2)

When Only One Group is of Interest:When Only One Group is of Interest:

TestTest

Proportion compared to a null valueProportion compared to a null value

one proportion testone proportion test

Ex)Ex) Are substance abusers more likely to bedepressed than general public?

Confidence Interval (95% CI: proportionConfidence Interval (95% CI: proportion

1.96*SE)1.96*SE)

Ex) Prevalence of depression in substance abusersEx) Prevalence of depression in substance abusers

Ex) Sensitivity and specificity of a new shortEx) Sensitivity and specificity of a new shortdepression instrument compared with thedepression instrument compared with thephysicianphysicians gold standard depression diagnosiss gold standard depression diagnosis

More on interpretation of 95% CI tomorrow.More on interpretation of 95% CI tomorrow.


11/34


When Comparing Two Independent Groups:When Comparing Two Independent Groups:

Ex) Comparing drugs A vs. B on relapse to heavydrinking

Essentially a 2 by 2 tableEssentially a 2 by 2 table

Comparative TestComparative Test

-- ChiChi--square testsquare test

-- Two proportion testTwo proportion test

Comparative Statistics (summary effect size)Comparative Statistics (summary effect size)

-- Absolute Difference in ProportionsAbsolute Difference in Proportions

-- Odds Ratios (OR)Odds Ratios (OR)

-- Relative Risks (RR)Relative Risks (RR)

For both OR and RR, 1 means no differenceFor both OR and RR, 1 means no difference

Can calculate 95% CI for any of the above (OR, etc.)Can calculate 95% CI for any of the above (OR, etc.)


12/34


When Comparing Two Independent GroupsWhen Comparing Two Independent Groups::

If sample size is smallIf sample size is small

rule of thumb = expected cell count < 5rule of thumb = expected cell count < 5 Comparative Test: FisherComparative Test: Fishers Exact Tests Exact Test

| A B | Total

---------------------------------------------

yes | 3 6 | 9

no | 9 2 | 11

--------------------------------------------Total | 12 8 | 20

Pearson chi-square test p-value = 0.028

Fisher's exact test p-value = 0.065


13/34

Continuous Dependent VariableContinuous Dependent Variable

Descriptive StatisticsDescriptive Statistics

Mean and Standard Deviation if data are symmetricMean and Standard Deviation if data are symmetric

Median and InterMedian and Inter--quartile range if data are skewedquartile range if data are skewed

MeanMean can be affected by one very large or one very smallcan be affected by one very large or one very smallvalue, and therefore isvalue, and therefore is sensitive to outlying valuessensitive to outlying values

MedianMedian isis robust to an outlying valuerobust to an outlying value because it is simplybecause it is simplythe value at the center when data are ranked in orderthe value at the center when data are ranked in order..

IfIf mean and median are very different, data are skewed.mean and median are very different, data are skewed. Always graphically explore the distribution (e.g., usingAlways graphically explore the distribution (e.g., using

histogram, box plot) and choose the appropriatehistogram, box plot) and choose the appropriatedescriptive statisticsdescriptive statistics

More on mean vs. median tomorrow.More on mean vs. median tomorrow.


14/34

Continuous Dependent Variable (2)Continuous Dependent Variable (2)


Test (One mean compared to a null value)Test (One mean compared to a null value)

One sample tOne sample t--testtest

Ex)Ex) Is cholesterol higher in diabetic patients comparedwith the general public?

Confidence Interval (95% CI = meanConfidence Interval (95% CI = mean

1.96*SE)1.96*SE)

Ex) Sample meanEx) Sample mean cholesterol = 124

Sample SD = 10, N = 200

95% CI for mean cholesterol = 124 1.96*10/sqrt(200)

= (122.6, 125.4)


15/34



When sample size is small (N


16/34



TestTestTwo independent group tTwo independent group t--testtest

Ex)Ex) Writing skill comparison between teaching methods A vs. B

Comparative Statistics: difference in meansComparative Statistics: difference in means

Ex) Difference in mean writing skill scores betweenEx) Difference in mean writing skill scores betweenthose who were taught with method A vs. method Bthose who were taught with method A vs. method B

Confidence Interval for Difference in MeansConfidence Interval for Difference in Means

95% CI = difference95% CI = difference

1.96*SE (of difference)1.96*SE (of difference)


17/34



If sample size is small (N < 25) or cannot assume that theIf sample size is small (N < 25) or cannot assume that the

dependent variable is interval and normally distributeddependent variable is interval and normally distributed

Use a NonUse a Non--parametric Test (Test of Median)parametric Test (Test of Median)

WilcoxonWilcoxon ranksumranksum test (tests equality of medians)test (tests equality of medians)


18/34

Graphical Methods to Compare Groups: Box PlotsGraphical Methods to Compare Groups: Box Plots

Resting

Heart

Rate

NoExercise

MildExercise

StrenuousExercise


19/34

Using Subjects as Their Own Controls:Using Subjects as Their Own Controls:

CrossCross--Over DesignsOver Designs

Same subject undergoes 2 or more treatmentsSame subject undergoes 2 or more treatments AdvantageAdvantage Maximizes powerMaximizes power fewest subjects neededfewest subjects needed

Limitations of reusing the same subjectLimitations of reusing the same subject May not be possibleMay not be possible

Carryover effectCarryover effect of treatmentof treatment need washoutneed washout

Length of experimentLength of experiment

Order effectOrder effect

Order should be randomized and balancedOrder should be randomized and balanced

Period effectPeriod effect


20/34

CrossCross--Over Designs (2)Over Designs (2)

ExamplesExamples

PrePre--post study (poor design, why?)post study (poor design, why?)

Ex) Weight before an exercise program and weightEx) Weight before an exercise program and weightafter a month of exercise programafter a month of exercise program

Traditional XTraditional X--over Studyover Study

Ex)Ex) AlternatingAlternating exposure to guided imagery procedureexposure to guided imagery procedurebetween stressful situation and a natural relaxingbetween stressful situation and a natural relaxingsituation on different days in random order andsituation on different days in random order and

assessing the effect on cravingassessing the effect on craving Stressful ImageStressful Image washout periodwashout period Relaxing ImageRelaxing Image

Relaxing ImageRelaxing Image washout periodwashout period Stressful ImageStressful Image

Ex) Drug A then cross over to BEx) Drug A then cross over to B


21/34

CrossCross--Over Designs (2)Over Designs (2)

Analytic MethodAnalytic Method

PrePre--post studypost study

Analyze changeAnalyze change--score or gainscore or gain--score and treat it asscore and treat it asa one sample problema one sample problem

Ex) change in weight within a person before andEx) change in weight within a person before and

after the exercise programafter the exercise program

Traditional XTraditional X--over Studyover Study

Analysis must first assess carryover effect, orderAnalysis must first assess carryover effect, ordereffect and period effect.effect and period effect.

If any effect, then must account for it.If any effect, then must account for it.


22/34

-level (significance level) the probabilityof claiming that there is a difference when

there is no true difference

Small

is good.

We usually set -level at 0.05. This means we allow 5% for making the

kind of error where we declare a

significant difference (reject the null

hypothesis) when the result happened by

chance (Type 1 error).

Multiple Comparison: Doing Many TestsMultiple Comparison: Doing Many Tests


23/34

WhenWhen 2 comparisons,2 comparisons,

(5%) should be reduced to(5%) should be reduced toadjust for the number of comparisons.adjust for the number of comparisons.

Suppose we are performing two independentSuppose we are performing two independentstatistical tests, then:statistical tests, then:

P(ofP(of rejecting the 1rejecting the 1stst when true) is 0.05when true) is 0.05

P(ofP(of rejecting the 2rejecting the 2ndnd when true) is 0.05when true) is 0.05 What is probability of rejecting at least one?What is probability of rejecting at least one?

P(ofP(of accepting 1accepting 1stst when true) is 0.95when true) is 0.95

P(ofP(of accepting 2accepting 2ndnd when true) is 0.95when true) is 0.95 Therefore,Therefore, p(ofp(of accepting both)accepting both)

= 0.95 x 0.95 = 0.9025= 0.95 x 0.95 = 0.9025

That is,That is, p(ofp(of rejecting at least one) = 0.0975rejecting at least one) = 0.0975

Multiple Comparison (2)Multiple Comparison (2)


24/34


Number ofNumber of

independent testsindependent tests

Probability of rejectingProbability of rejecting

null hypothesis,null hypothesis,

when truewhen true

11 0.050.05

22 0.09750.097533 0.1430.143

55 0.2260.226

1010 0.4010.401

If perform enough significant tests, you are sure toIf perform enough significant tests, you are sure to

find significant results by chance alone even whenfind significant results by chance alone even whennone exists.none exists.


25/34

For independent tests, one easy way of adjusting theFor independent tests, one easy way of adjusting the

level of significance is to use:level of significance is to use:

0.05/k0.05/k

where k is the number of tests to be performed.where k is the number of tests to be performed.

Therefore, instead of 0.05,Therefore, instead of 0.05,

When there are 5 tests, use 0.01When there are 5 tests, use 0.01

When there are 10 tests, use 0.005When there are 10 tests, use 0.005

Multiple Comparisons: What to do? (4)Multiple Comparisons: What to do? (4)


26/34


When testing a preWhen testing a pre--specified relationship, use aspecified relationship, use asignificance level of 5%.significance level of 5%.

When screening for interesting relationships,When screening for interesting relationships,use significance level of 1% so as not to identifyuse significance level of 1% so as not to identify

too many false relationships.too many false relationships.


27/34


28/34

MaleMale FemaleFemale

MajorMajor Number ofNumber of

ApplicantsApplicants

PercentPercent

AdmittedAdmitted

Number ofNumber of

ApplicantsApplicants

PercentPercent

AdmittedAdmittedAA 825825 62%62% 108108 82%82%

BB 560560 63%63% 2525 68%68%

CC 325325 37%37% 593593 34%34%

DD 417417 33%33% 375375 35%35%

EE 191191 28%28% 393393 24%24%

FF 373373 6%6% 341341 7%7%

TotalTotal 26912691 45%45% 18351835 30%30%

Weighted Average:Weighted Average: 39%39% 43%43%

Confounding (2)Confounding (2)


29/34

Example2Example2 :: Is psychiatric hospitalization rate differentIs psychiatric hospitalization rate differentin substance users versus nonin substance users versus non--users?users?

HospitalizationHospitalizationYes NoYes No

UserUser 2020 373373 5.1%

NonNon--useruser 66 316316 1.9%Substance useSubstance use looks to be associated with higherlooks to be associated with higher

psychiatric hospitalization ratepsychiatric hospitalization rate..

Separated bySeparated by Bipolar StatusBipolar Status

No BipolarNo Bipolar BBipolaripolar I/III/II

UserUser 33 176176 1.7%1.7% 1717 197197 7.9%7.9%NonNon--UserUser 44 293293 1.4%1.4% 22 2323 8.0%8.0%



30/34

Example 3Example 3: Smoking versus MI: Smoking versus MISmokerSmoker NonNon--SmokerSmoker

MIMI 5151 5454

No MINo MI 4343 6767

54%54% 44.6%44.6% OR = 1.47OR = 1.47

MaleMale FemaleFemale

MIMI 3737 2525 1414 2929

No MINo MI 2424 2020 1919 4747

61%61% 56%56% 48%48% 38%38%OR = 1.23OR = 1.23 OR = 1.19OR = 1.19

Smokers have higher MI rate, but the magnitude of theSmokers have higher MI rate, but the magnitude of the

relative likelihood of MI (measured as odds ratio (OR)) isrelative likelihood of MI (measured as odds ratio (OR)) islarger in the combined datalarger in the combined data..



31/34

Example 4:Example 4:1) Regression of Happiness on Smoker Group1) Regression of Happiness on Smoker Group

CoefCoef SESE pp--valuevalue

InterceptIntercept 65.0565.05 1.481.48 0.0000.000

SmokeSmoke 4.804.80 2.032.03 0.0200.020

2) Regression of Happiness on Age2) Regression of Happiness on Age

CoefCoef SESE pp--valuevalueInterceptIntercept 7.487.48 2.452.45 0.0030.003

AgeAge 1.851.85 0.070.07 0.0000.000

3) Regression of Happiness on Age and Smoke3) Regression of Happiness on Age and SmokeCoefCoef SESE pp--valuevalue

InterceptIntercept 2.652.65 2.072.07 0.2030.203

AgeAge 2.082.08 0.070.07 0.0000.000SmokeSmoke --5.255.25 0.700.70 0.0000.000



32/34


20

40

60

80

100

Happiness

Not Smoke

Without Considering Age, smokers appear to have higher mean

20

40

60

80

100

20 25 30 35 40 45Age

Y, Smoke == Not Y, Smoke == Smoke

by Smoking Status

Relationship between Happiness and Age

Increasing age is associated with greater happiness.Increasing age is associated with greater happiness.

Smokers tend to be older, making it look like smoking is associaSmokers tend to be older, making it look like smoking is associatedted

with greater happiness when not adjusting for age.with greater happiness when not adjusting for age. But smokers tend to be less happy than nonBut smokers tend to be less happy than non--smokers given same age.smokers given same age.


33/34

Developing

a

Statistical

Analysis

PlanDevelopingaStatisticalAnalysisPlan

Comparing two groupsComparing two groups

Continuous: tContinuous: t--testtest

Proportion: chiProportion: chi--square testsquare test Comparing multiple groups (continuous): ANOVAComparing multiple groups (continuous): ANOVA

Adjusted for other factors: ANCOVA, or regressionAdjusted for other factors: ANCOVA, or regression

Dichotomous outcome: Logistic regressionDichotomous outcome: Logistic regression Count outcome: Poisson regressionCount outcome: Poisson regression

Survival time outcome: Cox regressionSurvival time outcome: Cox regression

Watch for correlated data (repeated measures, clustersWatch for correlated data (repeated measures, clusters e.g., teeth in the mouthe.g., teeth in the mouth


34/34

To Keep in MindTo Keep in Mind

Typically, multiple appropriate methods are availableto analyze the same data that could yield legitimateanswers.

Try to use at least two different available methods toconfirm your results.

Always look at the raw data and display datagraphically, so learn to choose the right graphical

displays (ex: cross tabs, scatter plots, box plots)

It helps to make sample tables summarizing results

before you start the analysis.

Documents

Mk Select Right Stat