Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch –...

Biomarkers: the Good, the Bad and the AmbiguousEnrique F. Schisterman, PhD

Epidemiology Branch – DESPR – NICHD

In Memory of My Friend Sholom Wacholder

“I never feel I truly understand something if I can’t explain it to someone else, orally or in writing,”

Outline

Background

Limit of Detection

Pooling Biomarkers – Hybrid Design

Calibration Curves

Lipid Standardization (if we have time)

Conclusions

Background

Biomarker: A specific physical trait used to measure or indicate the effects or progress of a disease or condition

Newly developed laboratory methods expand the number of biomarkers on a daily basis Cost Measurement Error Causal Link to Disease

Motivation

Preliminary analysis of salivary concentrations of cortisol from the LIFE study

P=0.04Shipment 1 n Mean SDMichigan 85 0.40 0.19Texas 142 0.57 0.79

Cortisol by Site & Plate6

Ship1 Ship2 Ship1 Ship2 Ship3

1 2 3 4 5 1 2 3 1 2 3 4 5 1 2 1 2 3 4 5 6 7 8

Michigan Texas

Graphs by Site

ROC curve—background

Before a marker is used its discriminating accuracy needs to be evaluated

Most commonly used tool - Receiver Operating Characteristic (ROC) curve.

We consider only continuous markers

Distribution of marker in Healthy and Disease

* ROC curve – graph of (1-p,q) for all c.

ROC(1-p)=1-FD(FH-1(p))

* ROC defined over all possible thresholds gives the entire range of possible sensitivities and specificities.

* Diagnostic accuracy evaluated using measures including the area under the curve (AUC AUC = Prob(XH<XD)) and Youden’s index (J)

ROC curve example Complete Separation

1 – p(c)0 1

c q(c)p(c)

q(c) = 1 and p(c) = 1 for some c

ROC curve example Complete Overlap

q(c) = 1- p(c), for all c

1 – p(c)0 1

c q(c)p(c)

ROC curve example Partial Separation

Sensitivity by 1- Specificity

P( True Pos.) by P( False Pos.) across all c

1 – p(c)0 1

Chance

lineDf

c q(c)p(c)

1 – p(c)0 1

ROC: AUC

Area Under the Curve Overall discriminatory ability Area under the ROC curve

dxdyygxf

YXPAUC

Do Common Laboratory Practices Affect our Estimates of Risk?

Limit of Detection

Measurement Error

Calibration Curves

Lipid Standardization

Outline

Background

Limit of Detection

Calibration Curves

Conclusions

Reporting of Biomarker Data

ID Z1 3.12 1.53 8.44 0.85 5.46 3.27 2.08 5.89 13.410 2.511 1.912 6.1

• Reporting threshold is equal to 2.2

ID Z1 3.12 ND3 8.44 ND5 5.46 3.27 ND8 5.89 13.410 2.511 ND12 6.1

Report values < threshold as ‘not detected’

ID Z1 3.12 1.13 8.44 1.15 5.46 3.27 1.18 5.89 13.410 2.511 1.112 6.1

Report values < threshold as one half the value of the threshold

Conventional Determination of the Limit of Detection (LOD)

6.15)3( blanksblanksLOD

BLANK SERIES• 10.0• 5.0• 8.1• 7.1• 4.0• 11.3• 12.0• 8.0• 7.7• 7.0Mean = 8.02Std Dev = 2.53

0 5 10 15 20

Example of LOD left-censored data

0 5 10 15 20 25 30 35

Blanks

“True” biomarker

Better LOD?

Example of LOD left-censored data

0 5 10 15 20 25 30 35

Blanks

“True” biomarkerObserved biomarker (samples)

CONTROLS

Why is this a problem?Comparisons of PCBs in cases and controls

Controls—mean PCB Cases—mean PCB

Effect size

Blanks

Approaches for LOD/ missing data

Simplest approach is substitution Under certain circumstances yield minimal bias Conventionally, values below the LOD are usually

1. replaced by zero, LOD, LOD/2, LOD/√22. excluded 3. retained

Model based approaches Likelihood models (Perkins et al., AJE 2007)

Multiple imputation (Harel et al., Epidemiology 2010)

Schisterman EF, Vexler A, Whitcomb BW, Liu A. AJE 2006

CONTROLS

Why is this a problem?Comparisons of PCBs in cases and controls

Impute what? 0 LODLOD/2

LOD Simulation

Purpose: To evaluate the effect of the handling of values below the LOD on risk estimates

Simulated data from a normal and log normal distribution and varied:

Effect size Variance of PCBs in the exposure group LOD level Measurement error mean and variance

Effect of Handling of Values < LOD on %Bias

*LOD “low” indicates 1.6 SDs below the mean of controls, resulting in imputed values for a small number of data points. LOD “high” indicates 1 SD above the mean of the controls, resulting in imputed values for a large number of both controls and cases

Effect size = .25 SD

Method for values < LOD LOD high LOD low 1. Replace by a. Zero -59.0 -25.1 b. LOD -187.1 -40.8

c. LOD/2 -19.3 -18.1 d. LOD/2 -17.1 -15.9

2. Exclude (truncated) -314.2 -265.3 3. Retain (observed) -11.5 -11.7

LOD—Conclusions

Choice of how to handle values below the LOD can result in a loss of accuracy in estimating risk

Retaining observed values below the LOD produces the least biased estimates

Substitution of LOD/√2 for values below the LOD produces not terribly biased estimates

How the LOD Affects the ROC Curve?26

LOD: ROC curve

Properly accounting for the mass yields a ROC curve bias toward the null that is identical for any choice of a<d.

* Schisterman, et al.(2005)

MLE to correct for LOD

Normal Case

MLE’s of dist.parameters

jYYYYjY

1)(log)()(

log)|,(log

YYY d /)( where

* Gupta(1952)

Normal Case

MLE’s of ROCmeasures

InvarianceProperty

22 ˆˆ

ˆˆˆ

Normal Case

CI’s for ROCmeasures

2nd Deriv. and Fisher Info.

NzCUA A

)(1)ˆ(ˆ2

][)ˆ,ˆ()ˆ(

ICovCov

Gamma Case

,)(log)(log)1(

log)(log)|,(log

knjYYY

knjjYjYY

YYYYYY

dyeyG y

* Harter and Moore(1967)

dtttCUA XY

ˆ )()ˆ()ˆ(

)ˆˆ(ˆ

Gamma Case

InvarianceProperty

);1()ln(

)()1()(

);1(21

);1();1();1())(ln(

dhdpII

dhddqdhd

Gamma Case

CI’s for ROCmeasures

2nd Deriv. and Fisher Info.

NzCUA A

12111][)ˆ,ˆ()ˆ(

IIICovCov θ

LOD: ROC curve

Corrected!

* Schisterman, et al.(2005)

Evaluation of MLE

Example

PCB114 levels of 28 cases 51 controls classifying women with and without endometriosis.

Empirical 0.701 (0.576, 0.827)

MLE 0.777 (0.655, 0.899)

*assuming gamma distributions and d=0.005

Outline

Background

Limit of Detection

Calibration Curves

Conclusions

What is pooling?

Physically combining several individual specimens to create a single mixed sample

Pooled samples are the average of the individual specimens

Random Sample of Biospecimens

RANDOM SAMPLE

Randomly select 20 samples

FULL DATA

N = 40 Individual Biospecimens

Pooling Biospecimens

POOLED DATA

40 samples in groups of 2

FULL DATA

N = 40 Individual Biospecimens

Un-pooled

-4 -3 -2 -1 0 1 2 3 4

Effect of Pooling on Markers

Pooled

-4 -3 -2 -1 0 1 2 3 4

Un-pooled

-4 -3 -2 -1 0 1 2 3 4

Pooled

-4 -3 -2 -1 0 1 2 3 4

0 2 4 6 8 10

# of Pooled Samples

Number of assays required to reach equivalency on ROC Curves

Limit of Detection and Pooling

x if x LODZ

N A if x LOD

Unpooled Specimens

Pooled Specimens

( ) ( )( )

, if LOD

N/A, if LOD

p pp i i

Un-pooled

-4 -3 -2 -1 0 1 2 3 4

Effect of Pooling on Markers Affected by an LOD

Pooled

-4 -3 -2 -1 0 1 2 3 4

Un-pooled

-4 -3 -2 -1 0 1 2 3 4

Pooled

-4 -3 -2 -1 0 1 2 3 4

Estimation Based on Pooling Using Likelihood Function

XfdXkNk

)(Pr)!()!1(

);ˆ,ˆ(log,0

ˆ);ˆ,ˆ(log

Differentiating with respect to

To obtain estimates of

Schisterman et al. Bioinformatics 2006

Efficiency of the Mean and Variance

Variance of Estimated Mean

Variance of Estimated Variance

FULL DATA POOLED RANDOMFULL DATA POOLED RANDOM

LOD below Mean

LOD above Mean

Pooling and Random Sampling

Pooling advantages Reduces the number of assays we need to test Efficiently estimates the mean Cost-effective

Random sampling advantages Reduces the number of assays we need to test Efficiently estimates the variance Cost-effective & easy to implement

Hybrid Design: Pooled—Unpooled

Creates a sample of both pooled and unpooled samples Takes advantage of the strengths of both the pooling and

random sampling designs

Reduces number of tests to perform Cuts overall costs Gains efficiency (by using pooling technique) Accounts for different types of measurement error without

replications

– Pooling error– Random measurement error– LOD

Unpooled: X1,…,X5

Pooled: Z1,…,Z15

Hybrid Sample S: X1,…,X5,Z1,…,Z15

Setup of Hybrid Design

Unpooled: X1,…,X[αn]

Pooled: Z1,…,Z[(1-α)n]

In GeneralHybrid Sample S: X1,…,X[αn],Z1,…,Z[(1-α)n]

α is the proportion of unpooled samples

Maximum Likelihood Estimators

Random Sampling Pooling

In order to estimate the variance, α cannot be zero.

Schisterman EF et al, Stat Med 2010

ˆ)1(ˆ

ˆ)1(ˆˆ

xzx nn

)ˆ(ˆ

ln)1(2

)(ln),,(

Hybrid Design: Pooled—Unpooled

Create a sample of both pooled and un-pooled samples that takes advantage of the strengths of the pooled and random sample designs Reduce number of tests to perform Cut overall costs Gain efficiency (by using pooling technique) Accounts for different types of measurement

error without replicationsPooling errorRandom measurement error Limit of detection

Mathematical Setup

),(~ 2xxi NX NiX i 1

),0(~ 2p

pijii eX

xi pNZ

Individual Samples

Pooled Samples

p is the pooling group size

Average of p individual samples pooled together

Pooling error

Maximum Likelihood Estimators

( )( )( , , ) ln (1 ) ln

i xi xi ni n

x x p x zx z

Random Sampling Pooling

ˆ)1(ˆ

ˆ)1(ˆˆ

xzx nn

)ˆ(ˆ

2(1 )2

ˆ( )ˆ

ˆ(1 )

i xi n x

In order to estimate the variance, α cannot be zero.

Add in Measurement Error

),0(~ 2M

Individual Samples + ME

Pooled Samples + ME

p is the pooling group size

Measurement error

* 2 2~ ( , )i x x MX N

2* ,~ Mp

Hybrid Design Example: IL-6

Measured IL-6 on 40 MI cases and 40 controls

Biological specimens were randomly pooled in groups of 2, for the cases and controls separately, and remeasured

We want to evaluate the discriminating ability of this biomarker in terms of AUC

Hybrid Design Example: IL-6

n αx αy AÛC Var(AÛC)

Empirical 40 1.00 1.00 0.640 0.0036

Hybrid design: Optimal α 20 0.40 0.35 0.621 0.0049

Random sample: α=1 20 1.00 1.00 0.641 0.0071

Hybrid design reduced the variability of Var(AÛC) by 32% as compared to taking only a random sample

Summary—Hybrid Design

Hybrid design is a more efficient way to estimate the mean and variance of a population

Cost-effective

Yields estimate of measurement error without requiring repeated measurements

Here we focus on normally distributed data, but can be applied to other distributions as well

Outline

Background

Limit of Detection

Calibration Curves

Conclusions

Measurement of G-CSF

Chemiluminescence assays 96-well plate Antibody against the biomarker of interest Set of standards of known biomarker concentration

included in each batch Set of samples (concentration unknown) Light emitting molecule binds to bound biomarker

Measurement of Cytokines

Cytokines are not measured directly Antibodies against analyte(s) coat wells

Samples added, analyte binds to antibodies Unbound proteins are washed away

A ‘tag’ is added to the assay that binds to the protein – antibody complex that produces color

The intensity of the color is measured

ELISA/Multiplex Layout

Step 1: prepare antibodies mixture and add to plate Step 2: prepare calibrators, add to plate Step 3: prepare unknowns, add to plate

Use of Chemiluminescence Assays for Measuring Protein Concentrations

Use calibration to convert relative measures to the desired unit of concentration From optical density in relative fluorescence units (RFU)

to concentration in pg/mL

Current practice is per assay calibration Results in potentially large calibration datasets used only

minimally in current practice

Calibrating the Assay: The Standard Curve

The human G-CSF standard curve is provided only for demonstration

A standard curve must be generated each time an assay is run, utilizing values from the Standard Value Card included in the Base Kit

Potential variation in the relation between relative fluorescence and concentration Chromophore potentially affected by temperature, humidity,

G-CSF and Miscarriage in the CPP

Case-control study nested in the Collaborative Perinatal Project study cohort 462 miscarriage cases 482 non-miscarriage controls

Serum biospecimens from early pregnancy, prior to miscarriage onset

For n = 944, 24 assays were used

Unadjusted model Adjusted model OR [95% CI] OR [95% CI] Factor G-CSF 0.84 [0.72, 0.99] 0.78 [0.64, 0.95]

This estimate is based on the conventional batch specific approach

Objective

Question: Is the current practice of standard batch-specific calibration the best use of information?

To evaluate the effect of different approaches for calibration models on risk estimation

To assess bias associated with different approaches

Data from the calibration experiments

24 batches, each with 7 known concentrations measured in replicate Batches varied by

Shape Location Agreement between replicates Presence of outliers

l ogod

l ogcon

0 1 2 3 4

Batch 1 Calibration Curve – G-CSF

Standard 1 – undiluted (conc = 6000 pg/mL)

Fixed ‘known’ concentration

*All calibration data (in log10)

l ogod

l ogcon

0 1 2 3 4

Standard 2 – 1/3rd dilution (conc = 2000 pg/mL)

l ogod

l ogcon

0 1 2 3 4

l ogod

l ogcon

0 1 2 3 4

l ogod

l ogcon

0 1 2 3 4

l ogod

l ogcon

0 1 2 3 4

l ogod

l ogcon

0 1 2 3 4

l ogod

l ogcon

0 1 2 3 4

l ogod

l ogcon

0 1 2 3 4

l ogod

l ogcon

0 1 2 3 4

All Calibration Curves Collapsed – G-CSF

l ogod

l ogcon

0 1 2 3 4

s i mpl e l i near

l ogod

l ogcon

0 1 2 3 4

Effect of Calibration Method on Logistic Regression Results

Calibration models As observed Outliers removed aOR 95%CI aOR 95%CI Forward Collapsed Linear 0.34 (0.13, 0.90) 0.27 (0.10, 0.73) Batch specific Linear 0.73 (0.46, 1.17) 0.60 (0.33, 1.10) Mixed model Linear 0.67 (0.39, 1.14) 0.56 (0.30, 1.06) Collapsed Curvilinear 0.21 (0.05, 0.84) 0.25 (0.09, 0.71) Batch specific Curvilinear 0.81 (0.57, 1.16) 0.98 (0.93, 1.02) Mixed model Curvilinear ~ ~ ~ ~ Reverse Collapsed Linear 0.37 (0.15, 0.91) 0.28 (0.11, 0.73) Batch specific Linear 0.63 (0.37, 1.11) 0.58 (0.32, 1.07) Mixed model Linear 0.43 (0.19, 0.94) 0.53 (0.27, 1.02) Collapsed Curvilinear 0.37 (0.15, 0.91) 0.29 (0.11, 0.74) Batch specific Curvilinear 0.86 (0.53, 1.41) 0.67 (0.38, 1.16) Mixed model Curvilinear 0.50 (0.27, 0.95) ~ ~

Simulation Study

1. Generate dataset with:

True biomarker concentration True effect on risk Overall relation between concentration and RFU Batch variability Occasional outliers

2. Simulate calibration experiments to estimate RFU – concentration relation according to each approach

3. Assess bias and variance of estimators from risk models

Simulation Study:The Biomarker

Biomarker: exp(X ~ N(5,1))

Miscarriage risk: OR = 1.05, 1.15 or 1.65 β={0.05, 0.14, 0.50}

Conc. and OD: OD determined througha single function

Summary of simulation study resultsComparison of shape, model for β = 0.14

CollapsedMixedBatch-specific

Linear Curvilinear Linear CurvilinearFORWARDS REVERSE

Whitcomb et al, Epidemiology 2010

Conclusions

Underestimation of effects due to calibration approach has broad implications

Use of conventional batch-specific approaches performed poorly Greatest bias to estimates in simulations Most prone to loss of data for batches with failure of

some calibration points

Outline

Background

Limit of Detection

Calibration Curves

Lipid Standardization (later if we have time)

Conclusions

Outline

Background

Limit of Detection

Calibration Curves

Conclusions

Do Common Laboratory Practices Affect our Estimates of Risk?

Limit of Detection Request the observed values Design away using hybrid methods and overcome cost, LOD and ME

Calibration Curves Study Design should include a calibration curve plan

Standardization Don’t do it!

YES!91

Questions?

Thank you!

Future Direction

Limit of detection: Linking the CV, AUC and calibration.

),(~ 2xXNX ),0(~ 2

Reliability index could be applied such that

Coefficient of variation is used by labs during calibration

For normally distributed biomarkers, we can use these two summaries such that

Limit of detection: Linking the CV, AUC and calibration.

),(~ 2xXNX ),0(~ 2

Reliability index could be applied such that

Coefficient of variation is used by labs during calibration

For normally distributed biomarkers, we can use these two summaries such that

We can investigate the relation between these factors easily because R and CV are relative measures.

This is of particular importance because for certain types of measurement processes, the Limit of detection is based on the CV and a more appropriate cut-point might be better generated using ROC methods for differentiating between signal and noise.

modeling calibration data

dependent/independent variables

Assay yields OD for observations in the main dataset

‘Forward’ (regression calibration) Estimate & use calibration equations in the same order, i.e.:

‘Reverse’ Estimate calibration equations, flip for calibration, i.e.:

)(ODfionConcentrat

)( ionconcentratfOD

treatment of batch

Approach to batch, OD as independent variable

use calibration data to model:

conijk = α + βODijkCollapsed

conijk = αk + βkODijk Batch-specific effects

conijk = α* + β*ODijk + aj + bjODijkMixed effects

i = standard 1 – 7

j = replicate 1 – 2

k = batch 1 – 24

treatment of batch

Approach to batch, concentration as dependent variable

use calibration data to model:

ODijk = α + βconijkCollapsed

ODijk = αk + βkconijk Batch-specific effects

ODijk = α* + β*conijk + ak + bkconijkMixed effects

i = standard 1 – 7

j = replicate 1 – 2

k = batch 1 – 24

Data calibration

Parameters estimated for linear and curvilinear models

For concentration as dependent Unknown concentrations calibrated from estimated f(OD)

)(ˆˆ ODionconcentrat

Data calibration

For concentration as dependent Unknown concentrations calibrated from estimated f(OD)

221 )(ˆ)(ˆˆ ODODionconcentrat

Data calibration

For OD as dependent Unknown concentrations calibrated as:

)(ˆˆ ionconcentratOD

Data calibration

For OD as dependent Unknown concentrations calibrated as:

ˆ)ˆ(4ˆ

221 )(ˆ)(ˆˆ conconOD

Simulation study the calibration relation

7 known concentrations in replicate/batchMeasured OD for each true concentration

True overall relation by 4 parameter logistic

at slope2)(at

Simulation study the calibration relation

7 known concentrations in replicate/batchMeasured OD for each true concentration

True overall relation by 4 parameter logistic

Random batch variability Random failures with frequency 0.01

Calibration approaches

Calibration models varied as regards:1. Dependent/independent variable

– Measured OD and fixed known amount

2. Shape (linear or quadratic in log)3. Treatment of batch

Disregard (collapsed) Model as fixed effect (batch-specific) Model as random effect

Results

Calibration models Plots of cross-classification

Effects on risk estimation Simulation study

s i mpl e l i near

l ogod

l ogcon

0 1 2 3 4

All batches combined – outliers in the calibration set

0 1 2 3

s i mpl e l i near

l ogod

l ogcon

0 1 2 3 4

All batches combined – outliers in the calibration set

0 1 2 3

Bat ch number of assay 1 2 3 4 5 6 7 89 10 11 12 13 14 15 16

17 18 19 20 21 22 23 24

A A A B B B C C C D D D E E E F F F G G G H H HI I I J J J K K K L L L M M M N N N O O O P P PQ Q Q R R R S S S T T T U U U V V V W W W

l ogod

l ogcon

0 1 2 3 4 5

s i mpl e l i near

17 18 19 20 21 22 23 24

l ogod

l ogcon

0 1 2 3 4

All calibration data (in log10), with batch-specific regression

All batches combined outliers in the calibration set

17 18 19 20 21 22 23 24

l ogod

l ogcon

0 1 2 3 4 5

s i mpl e l i near

17 18 19 20 21 22 23 24

l ogod

l ogcon

0 1 2 3 4

All calibration data (in log10), with batch-specific regression

All batches combined outliers in the calibration set

Impact of calibration on risk estimationoutput from logistic regression Assay response data were calibrated to convert OD

to concentration according to each of the previously described methods

Logistic regression relating miscarriage risk to log10 levels of GCSF were performed

Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch –...

Documents

¡Salud! con leche y el amiguito Cepillito - NICHD

Early Childhood Education and School Readiness - NICHD

Protocol for the NICHD Neonatal Research Network SU

olatile ncertain omplex Ambiguous

Ambiguous Serenade: Ginastera & Davidovsky

Ambiguous Universality

The 2018 NICHD Strategic Planning Process · • Why and Why Now? • Values and Questions • Visioning vs. Strategic Planning • NICHD Strategic Planning: Purpose • Alignment

6 The Sine Law - The Ambiguous Casemkamber.weebly.com/.../9/37695755/6_the_sine_law_-_the_ambiguo… · THE AMBIGUOUS CASE WITH THE SINE LAW Ambiguous means "having more than one

Dreger Ambiguous Sex

Protocollo NICHD Protocollo per l’intervista investigativa ai minori …nichdprotocol.com/Protocollo NICHD versione italiana.pdf · 2013-05-03 · Raccontami tutto quello che è

SMART Experimental Designs for Developing Adaptive Treatment Strategies S.A. Murphy NIDA DESPR February, 2007

NICHD/FDA Newborn Drug Development Initiative Workshop I

Spatial Network Links to Maternal and Child’s Health New ... · • National Institute of Health (NICHD K01-HD093863) • Penn State University's Population Research Institute (NICHD

AMBIGUOUS PARTISANSHIPS; - OpenEdition

Ambiguous Case in Fact

Ambiguous genitalia

NICHD Controlled Trail

Disambiguating Highly Ambiguous Words

Jackson Ambiguous Sexuality

Ambiguous Map Problem (application of the Ambiguous Case of the Law of Sines)