LEVELS OF EVIDENCE FROM DIABETES REGISTRIES

Registry-based Epidemiology?

John M. Lachin

Professor of Biostatistics, Epidemiology and Statistics

The Biostatistics Center

The George Washington University

EuBIRO-D vs. USA

Ciao Fabrizio e Massimo

No regional or national healthcare program

No national or regional registriesHMO network

Translating Research into Action for Diabetes

Comparative Effectiveness ResearchAgency for Healthcare Quality and Research: patient satisfaction, quality of life

National Institutes of Health: Clinical outcomes

GRADE study

Science and Uncertainty

Jacob Bronowsky:

All information is imperfect. We have to treat it with humility... Errors are inextricably bound up with the nature of human knowledge…

The degree of uncertainty is controlled through the application of the scientific method,

and is quantified through statistics.

Statistical Test of an Hypothesis

Null Hypothesis (H0): The hypothesis to be disproven

The hypothesis of no difference.

Alternative Hypothesis (H1): The hypothesis to be proven

The hypothesis that a difference exists.

Two types of errors:Type I: False positive, probability Type II: False negative, probability

Power = 1 -

Factors that Affect and Power

Selection and Observational/Experimental Bias

Poor study design or executionMissing dataReproducibility (precision) of assessments

Missing DataThe Fundamental Issue - BIAS

Numerators and denominators may be biased

Estimates of population parameters, differences between treatments or exposure groups may be biased.

Statistical analyses, p–values and confidence limits may be biased.

p = 0.05 may mean a false positive error rate () much greater than 0.05;

N=800, 20% missing in treated/exposed, true ≈ 0.40.

Can’t Statistics Handle This?Not definitively.The magnitude of the bias can not be

estimated, no correction possible.Analyses can be conducted under certain

assumptions.But there is no way to prove that the

assumptions apply.Best way to deal with missing data is to

prevent it.

Sample Size Adjustments

Can adjust sample size to allow for losses-to-follow-up and missing data, e.g. increase N by 10% if expect 10% losses

BUT, this adjusts only for the loss of information,

NOT for any bias introduced by missing data.

Precision or Reliability of Measures

Reliability coefficient = proportion of total variation between subjects due to variation in the true values.

1 - = proportion of variation due to random errors of collection, processing and measurement.

Reliability ()

Po

we

r

Impact of ReliabilityPower decreases as decreases.

Impact of ReliabilityIf N is the sample size needed for a

precise measure then N/ is needed for an imprecise measure.

1.0 0.9 0.8 0.7 0.6 0.5

1/ 1.0 1.11 1.25 1.43 1.67 2.0

Impact of ReliabilityMaximum possible correlation between Y

and X is a function of the respective reliabilities: Max(R2) = x y

x y Max(R2)

1.0 0.9 0.90

0.9 0.9 0.81

0.9 0.7 0.63

0.9 0.5 0.45

0.7 0.7 0.49

0.7 0.5 0.35

Impact of Misclassificationsm = fraction of treatment or exposure

misclassifications, or fraction of outcomes misclassified

N/(1-2m)2 is needed

m 0 0.1 0.8 0.7 0.6 0.5

1/(1-2m)2 1.0 1.56 2.78 6.25 25.0 ∞

Randomized Clinical Trial Randomization:

•Subjects assigned to each treatment independently of patient characteristics

•No selection bias. Treatment groups expected to be similar for all variables measured and unmeasured.

•No confounding of the experimental treatment with other uncontrolled factors

•May infer a cause – effect relationship between treatment and the outcome, provided the trial is of good quality.

Randomized Clinical Trial Precisely defined population

Precisely defined exposure (the treatments)

Precisely defined outcome measure

Results clearly interpretable

Observational Study

Many types, e.g. case-control study

Prospective cohort study

No randomized controls

Maybe a precisely defined population

Maybe a precisely defined exposure (the treatments)

Maybe a precisely defined outcome measure

Observational Study

Many potential biases

Selection bias – composition of groups

Confounding with other factors

Statistical adjustments substituted for randomization

Observational Study

Necessary in settings where a randomized study is impossible

Smoking and lung cancer

Generally describe an association between the exposure factor and an outcome that may not represent a causal relationship.

Difficult to establish causality, though possible with replication of a highly specific association.

Observational Evidence

The essential issues with observational evidence is the degree to which an observed relationship can or can not be explained by

•other variables,

•other mechanisms, or

•biases

– even after statistical adjustment

Confounding

When the study factor (groups) are associated with another (confounding) factor that is a direct cause of the outcome.

Coffee consumption and cancer.

Coffee consumption confounded with smoking.

Higher fraction of smokers among coffee drinkers.

Statistical Adjustment for Confounding

Regression or stratification model including the study factor and the possible confounding factor(s)

Assumes that the operating confounding factors have been identified and measured.

Assumes that the regression model specifications are correct.

Statistical Adjustment for Confounding

Estimates the association of the factor with the outcome IF the confounding factor were equally distributed among the groups.

Difference in cancer risk between coffee drinkers and non-drinkers IF the fraction of smokers was the same among drinkers and non-drinkers.

Coffee drinking and smoking are alterable. Thus, the results would have a population interpretation.

Statistical Adjustments

NOT all covariate imbalances introduce bias, in which case adjustment itself introduces bias.

Gender inherently confounded with body weight

Gender adjusted for body weight estimates the gender difference if males and females had the same weight distribution.

Statistical Adjustments

Adjustment for weight provides a biased estimate of the overall male:female difference in risk in the population

But weight-adjusted estimate describes the additional male:female difference in risk, if any, that is associated with gender differences other than weight

Of mechanistic interest.

Omitted Covariates

Observational study can only adjust for what has been measured.

Adjustment for observed factors can not eliminate bias due to imbalances in unmeasured covariates.

Inappropriate CovariatesAnalysis should follow the prospective

history of covariates

Statistically invalid to define a covariate over a period of exposure that goes beyond the observation of an event.

Example, mean HbA1c over 5 years as a predictor of outcomes observed during the 5 years.

Rather, use the mean HbA1c up to the time of each successive event.

Confounding by Indication

In some cases, however, exposure to a factor (e.g. drug) may be confounded with the indications leading to the exposure.

Example: statins indicated in the presence of hyperlipidemia.

Recent data suggests that statin use may also increase risk of T2D in IFG/IGT.

But is the increased risk due to the statin use or the prior history of hyperlipidemia?

Confounding by Indication

In other cases an adjusting factor (e.g. dose) may likewise be confounded with an indication.

Example: Hemkens et al. analysis of the association of insulin glargine vs. human insulin with cancer in a German claims database.

14% decrease in age, gender adjusted risk.

But substantial dose imbalance.

14% increase in risk when also adjusted for dose.

Reasons for Dose ImbalanceConfounding by indication, or allocation

bias.

High or low glargine (or human insulin) dose may be determined by unmeasured patient factors that are differentially distributed within groups.

e.g. high glargine dose only administered to severely ill patients.

Impossible to statistically adjust for such confounding

Adjusted analysis results are biased.

Registries

Many types:

100% population captured, e.g. public health care system

Non-random subsample, e.g. insurance provider or hospital based

In latter case, registry population may not represent the full population of interest

Inherently prospective

But no standardized follow-up schedule

Registries

Relies on data capture in conjunction with the administration of medical care

No specific exposure of interest when established, in epidemiological sense

No specific outcome measure of interest.

Rather medical status and treatment recorded (possible exposures) and other major morbidities and mortality recorded (possible outcomes).

Registries

Epidemiologic analyses may be attempted.

But, difficult to precisely define exposure to a factor:

When is a subject

First at risk of being exposed (e.g. when is a drug introduced to the market?)

Actually first exposed (e.g. starts drug)

Removed from exposure (e.g. off drug)

Confounding by indication often an issue

Registries

Coding, classification of events may not be standardized

Often no adjudication

May be difficult to determine whether or exactly when an outcome event occurred, e.g. macroalbuminuria is “interval-censored”

May be difficult to determine when subject no longer at risk (right censored)

Incidence may be difficult to assess reliably.

Registries - Uses

Prevalence

Distribution of patient status or conditions in the population

Cross-sectional associations

If “representative” but not proportionally, weighted analyses can provide estimates in the broader population.

Disadvantaged populations (poverty, uninsured) may not be represented

Registries - Epidemiology

Exposure to a factor and outcomes

Open to many biases.

Statistical adjustments may be inadequate.

But, a registry can be the foundation for first-rate epidemiologic studies.

Registries - Epidemiology

Nested case-control studies

Sub-sample of possible cases that is carefully adjudicated

Sub-sample of possible controls (matched by follow-up time) also verified.

Exposure (risk) and confounding factors also verified.

Registries - Epidemiology

Prospective cohort studies

Identify eligible subjects -- representative of the registry (general) population

Formally enroll subjects (consent) with a systematic follow-up schedule

Careful characterization of exposure (risk) and confounding factors

Specific outcome reporting (assessments) with adjudication.

Registries - EpidemiologyEmbedded cohort study

Identify eligible subjects

Enroll subjects (consent)

Establish a schedule of assessments to be conducted as part of routine care

Send notices to patients when visits due

Capture exposure (risk) and confounding factors

Identify possible outcomes through medical reports, with subsequent adjudication.

Registries - Epidemiology

A hybrid

Establish an embedded cohort study.

Also implement a formal prospective study in a sub-sample.

The latter can serve as a quality check on the former.

Registries - Epidemiology

LARGE Sample Size

N needed to detect a rare outcome (e.g. fulminant hepatotoxicity, or angioedema)

If risk is 1 in 10,000, need N = 29,956 to be 95% confident that at least one case will be observed.

If wished to have 85% power to detect a 50% increased risk, at least 75 events required.

N = 836,000 followed for 1 year!!

Conclusions

Registry can provide superior descriptions of quality of care and distribution of factors in broad population of interest.

Not as rigorous as a formal prospective epidemiologic study, but can form the basis for such studies.

Affords opportunities for large sample sizes needed to detect rare outcomes.