Considerations for Statistical Analysis in Observational Comparative Effectiveness

ResearchPrepared for:

Agency for Healthcare Research and Quality (AHRQ)www.ahrq.gov

This presentation will: Describe the key variables of interest with

regard to factors that determine appropriate statistical analysis

Propose descriptive analysis or graph according to treatment group

Propose the model that will be used for primary and secondary analytical objectives

Outline of Material

When observational data are used in comparative effectiveness research, careful and often complex analytic strategies are required to adjust for confounding.

Statistical considerations Descriptive Statistics/Unadjusted Analyses Traditional Multiple Regression Model Selection Model Assumptions Propensity Scores/Disease Risk Scores Instrumental Variables Missing Data Considerations Time-Varying Exposures/Covariates

Introduction

Descriptive statistics Continuous variables: measures of range,

dispersion, and central tendency Categorical variables: frequency (n) and

percentage Data distributions: Kaplan-Meier plots

Unadjusted analysis Conducted to identify covariates associated with

the exposure and/or study outcome (e.g., t-test) Can provide a broad picture of study subject

characteristics

Descriptive Statistics and Unadjusted Analysis

Traditional multivariable regression Control for potential confounding variables in

the estimation of treatment effects Useful when there is a sufficient number of

outcome events per covariate and exposure is not infrequent

If exposure is common and the outcome rare, consider propensity scores.

If exposure is infrequent, consider disease risk scores.

Adjusted Analysis

Model Selection  Number of Followup Measures and Time Intervals 

Single measureRepeated measure, fixed intervals

Repeated measure, variable intervals

Outcome measure No clustering Clustering (e.g.,

multisite study)    

Dichotomous

Logistic regression

Multilevel (mixed) logistic regression, GLMM, GEE, conditional logistic regression

Repeated measures ANOVA (MANOVA), GLMM, GEE

GLMM, GEE

Continuous Linear regression

Multilevel (mixed) linear regression, GLMM, GEE

Repeated measures ANOVA (MANOVA), GLMM, GEE

GLMM, GEE

Time to event

Cox proportional hazards regression

Variance-adjusted Cox model or shared frailty model

   

Time to event (aggregate or count data)

Poisson regression

Multilevel (mixed) Poisson regression    

Propensity scores measure the probability of receiving treatment (or exposure) conditional on observed covariates.

Propensity scores are favorable in studies with a common exposure and rare or multiple outcomes.

They can be used in subclassification or stratification, matching, and weighting.

They include covariates that are true confounders or at least related to study outcome.

Propensity Scores

Disease risk scores (DRSs) measure the estimated probability or rate of outcome occurrence as a function of covariates.

Estimation approaches: Fit regression model for entire cohort, adjusting for

exposure Fit regression model for unexposed/referent group Compute fitted values assuming unexposed/referent

group for all study subjects DRSs are favorable in studies having a common

outcome and rare exposure. DRSs are useful for effect modification by disease risk.

Disease Risk Scores

Instrumental variables are measures that are causally related to exposure but unrelated to outcome and study covariates.

They are useful for adjusting for potential unmeasured confounders. However, it may be difficult to identify a high-

quality instrument. They can be used in conjunction with

traditional multiple regression and propensity score matching.

Instrumental Variables

Observational studies commonly have missing data.

Missingness can be characterized by using exploratory data analyses. Complete-case analysis for subjects with no

missing data: Can reduce sample size, limiting efficiency Can result in potential bias if missingness is

differential between groups Imputation for missing completely at random

or missing at random

Missing Data Considerations

Time-dependent Cox regression models can account for time-varying exposures and covariates. However, difficult issues arise when both treatment

and confounding variables vary over time. Inverse-probability-of-treatment weighting can be

used to estimate a marginal structure model. This approach is a generalization of propensity score

weighting to the time-varying treatment context. Intent-to-treat analysis can be conducted in which

exposure status is assumed throughout followup when treatment adherence is low.

Time-Varying Exposures/Covariates

Observational comparative effectiveness studies are often strongly affected by confounding.

Thoughtful application of statistical approaches can adjust for confounding and improve causal inference.

An appropriate analytical technique is based on assumptions.

Consider the effect of missing data on analyses.

Sensitivity analyses can address residual confounding.

Conclusion

Summary Checklist (1 of 3)

Guidance Key ConsiderationsDescribe the key variables of interest with regard to factors that determine appropriate statistical analysis

• Independent variables (when are they measured, fixed, or time-varying; e.g., exposures, confounders, effect modifiers)

• Dependent variables or outcomes (continuous or categorical, single or repeated measure, and time to event)

• State if there will be a “multilevel” analysis (e.g., looking at effects of both practice-level and patient-level characteristics on outcome)

Summary Checklist (2 of 3)

Guidance Key ConsiderationsPropose descriptive analysis or graph according to treatment group

• Should include the available numbers per group, n missing for all key covariates, and distributions or graphs that are needed to decide if transformation of data is needed or determine an accurate functional form of the final model

• Should include all potential confounders and effect modifiers to assess initial covariate balance by study group

Summary Checklist (3 of 3)

Guidance Key Considerations

Propose the model that will be used for primary and secondary analytical objectives

• Should take into account the design (independent vs. dependent observations, matched, repeated measurement, clustered), objectives, functional form of model, fixed/time-varying followup period, fixed and time-varying exposure and other covariates, assessment of effect modification/heterogeneity, type of outcome variables (categorical, ordinal, or continuous), censored data, and the degree of rarity of outcome and exposure

• Should propose a suitable approach for adjusting for confounding (e.g., multiple regression model, propensity scores, instrumental variables [could be secondary or main analysis])