Generalized Linear Models (GLMs) & Categorical Data Analysis (CDA) in R

Hong Tran, April 21, 2015

Laboratory for Interdisciplinary Statistical Analysis

Collaboration:

Visit our website to request personalized statistical advice and assistance with:

Designing Experiments • Analyzing Data • Interpreting ResultsGrant Proposals • Software (R, SAS, JMP, Minitab...)

LISA statistical collaborators aim to explain concepts in ways useful for your research.

Great advice right now: Meet with LISA before collecting your data.

All services are FREE for VT researchers. We assist with research—not class projects or homework.

LISA helps VT researchers benefit from the use of Statistics

www.lisa.stat.vt.edu

LISA also offers:

Educational Short Courses: Designed to help graduate students apply statistics in their researchWalk-In Consulting: Available Monday-Friday from 1-3 PM in the Old Security Building (OSB) for questions <30 mins. See our

website for additional times and locations.

Outline

1.What is CDA?

2.Contingency Table

3.Measures of Association

4.Test of Independence

5.What is GLM? When should we use it?

6.How to evaluate the GLM models?

7.Logistic Regression

8.Poisson Regression

What is CDA?

Dependent Variable (Y)

Independent Variables (X)

Model

Continuous (Normal) Continuous Linear Regression

Continuous Categorical ANOVA

Continuous Mixed ANCOVA

Categorical Categorical CDA

Contingency Table

I rows for categories in X

J rows for categories in Y

Values in cell=possible outcomes

Example 1 of Contingency Table

the relationship between smoking and epidermoid/undifferentiated pulmonary carcinoma (cancer)

Cohort study conducted

2x2 contingency table

Does smoking increase the risk of having epidermoid/undifferentiated pulmonary carcinoma?

Generating Contingency Table in R

Input the 2×2 table in R as a 2×2 matrix

Change the matrix to table using the function as.table(), because some functions are happier with tables than matrices

Measure of Association

Continuous Variables-Pearson Correlation Coefficient

Ordinal Variables-Pearson Correlation Coefficient

Nominal Variables-Phi Coefficient and Cramer’s V

Pearson Correlation

Pearson Correlation Example 2

mtcars in R

1974 Motor Trend US magazine

mpg: miles per gallonwt: weightdrat: rare axle ratio

Phi Coefficient

measures the association between two binary variables.

Its value ranges from -1 to +1, where +1/-1 indicates perfect positive association/negative association, 0 indicates no association.

The square of the phi coefficient is related to the chi-squared statistic for a 2×2 contingency table.

Cramer’s V

Cramer’s V measures the association between two nominal variables.

It varies from 0 (no association) to 1 (complete association) and can reach 1 only when the two variables are equal to each other.

Measures of Association

Comments:

1, When the two variables are binary, Cramer’s V is the same as Phi Coefficient

2, In R, under library(psych), use function phi() for Phi Coefficient

3, In R, under library(vcd), use function assocstats() for Cramer’s V

Test of Independence

Large Sample Size

Chi-square Test

Small Sample Size

Fisher’s Exact Test

Test of Independence (Chi-square Test)

Back to Example 1

Cases Control Total

Smoke 18/313 13/313 31/313

Non-smoker 46/313 236/313 282/313

Total 64/313 249/313 1

Test of Independence (Chi-square Test)

Test of Independence (Fisher’s Exact Test)

When any of the expected counts fall below 5, Chi-square test is not appropriate. Instead, we use Fisher’s Exact Test.

Example 3: The following data are from a Stanford University study of the effectiveness of the antidepressant Celexain the treatment of compulsive shopping.

Worse Same Better

Celexain 2 3 7

Placebo 2 8 2

Test of Independence

Chi-Square Test

Use R function chisq.test()

Fisher’s Exact Test

Use R function fisher.test()

Generalized Linear Models

When the response variables are not continuous, not normally distributed

Count numbers: 1, 2, 3,…

Binary: 0 and 1

Comparison

General Linear Model

Generalized Linear Model

Special cases ANOVA, ANCOVA, MANOVA, MANCOVA,

linear regression, mixed model

Linear regression, logistic regression,

Poisson regression

Function in R lm glm

Typical method estimation

Least Square Maximum Likelihood

Ordinary Linear Regression Ordinary Linear Regression (OLR) investigates and models

the linear relationship between independent variables and dependent variables that are continuous.

The simplest regression is Simple Linear regression, which models the linear relationship between a single independent variable and a single dependent variable.

Simple Linear Regression Model:

Assumptions in OLRThe assumptions are:

The true relationship between x and y is linear.

The errors are normally distributed with mean zero and unknown common variance .

The errors are uncorrelated.

The possible approaches when the assumptions of a normally distributed dependent variable with constant variance are violated:

Data transformations

Weighted least squares

Generalized linear model (GLM)

GLM Model

𝑔 function is called the link function because it connects the mean and the linear predictor 𝜇 𝑥

Dependent variable’s distribution must come from the Exponential Family of Distributions

Includes Normal, Bernoulli, Binomial, Poisson, Gamma, etc.

3 Components

Random: Identifies dependent Y and its probability distribution

Systematic: Independent variables in a linear predictor function

Link function: Invertible function .that links the mean of the 𝑔dependent variable to the systematic component.

Response Distribution

Types of GLMs

GLM and OLR

Ordinary linear regression is a special case of GLM

In OLR, the 3 components for GLM are:

Random: the dependent variable is normally distributed with mean and variance 𝜇

Systematic: Independent variables in a linear predictor function

Link function: Identity link ( )=𝑔 𝜇 𝜇Therefore, the GLM model for Ordinary linear regression is

Model Evaluation: Deviance

Deviance: measures how close the predicted values from the fitted model match the actual values from the raw data.

Definition:

Deviance = -2[log-likelihood(proposed model)-log-likelihood(saturated model)]

A saturated model is a model that fits the data perfectly, so its log-likelihood is the maximum. It has as many parameters as observations and hence it provides no simplification at all.

The deviance has a chi-squared asymptotic null distribution.

The degree of freedom is n-p, where n is the number of observations and p is the number of model parameters.

Smaller deviance, the better the model

Inference in GLM Goodness of Fit test

─ The null hypothesis is that the model is a good alternative to the saturated model.

─ Deviance is the Likelihood Ratio Statistic

Likelihood Ratio test

- Allows for the comparison of one model to another model by looking at the difference in deviance of the two models.

-Null Hypothesis: the predictor variables in Model 1 that are not found in Model 2 are not significant to the model fit.

-Alternative Hypothesis: the predictor variables in Model 1 that are not found in Model 2 are significant to the model fit.

─ LRS is distributed as Chi-square distribution.

─ Simpler models have larger deviance.

Model Comparison in GLM

Two additional measures for model comparison are:

─ AkaikeInformation Criterion (AIC)

•Penalizes model for having many parameters

•AIC=-2logLikelihood+2*p where p is the number of parameters in the model

•The smaller AIC, the better the model

─ Bayesian Information Criterion (BIC)

•BIC=-2logLikelihood+ln(n)*p where p is the number of parameters in the model and n is the number of observations

•Usually stronger penalization for additional parameter than AIC

•The smaller BIC, the better the model

Summary

Setup of GLM

Inference in GLM

Deviance and Likelihood Ratio Test

─ Test goodness of fit for the proposed GLM model

─ Test the significance of a predictor variable or set of predictor variables in the model

Model Comparison in GLM

─ AIC

─ BIC

Logistic Regression

Logistic regression is a regression technique for predicting the outcome of a binary dependent variable.

Example: y=1-Success, 0-Failure

Random Component: the dependent variable follows a Bernoulli distribution

─ Probability of Success: 𝑝─ Probability of Failure: 1-𝑝─ The probability of obtaining y=1 or y=0 is given by Bernoulli Distribution:

─ Mean(Y): μ=𝑝

Logistic Regression

Logistic Regression

Steps for Logistic Regression in R

1.Create a single vector of 0’s and 1’s for the response variable.

2.Use the function glm() family=binomial to fit the model.

3.Test for goodness of fit and significance of predictors.

4.Interpretation

Poisson Regressions

Poisson regression is a regression technique for predicting the outcome of a count dependent variable.

Dependent variable measures the number of occurrences in a given time frame.

Outcomes equal to 0,1,2,…

Examples:

Number of penalties during a football game.

Number of customers shop at a grocery store on a given day.

Number of car accidents at an intersection during a period of time.

Poisson Regression

Poisson Regression

Steps for Poisson Regression in R

1.Input data where y is a column of counts.

2.Use the function glm() family=poisson to fit the model.

3.Test for goodness of fit and significance of predictors.