Metrological Experiments in Biomarker Development (Mass Spectrometry—Statistical Issues)

Metrological Experiments inBiomarker Development (Mass Spectrometry—Statistical Issues)

Walter Liggett Statistical Engineering Division

Peter BarkerBiotechnology Division

National Institute of Standards and Technology

Biomarker(Clinical Pharmacology & Therapeutics, 2001)

A characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacologic responses to a therapeutic intervention.

Two parts of a biomarker– Execution of measurement protocol– Interpretation of measured response

Metrology

• Development and evaluation of a measurement protocol, the first part of a biomarker

• Diverse lessons learned from varied applications• Focus on general purpose protocols which may be

adequate for a particular purpose• The use of metrology in biomarker development is

the subject of this talk

Metrological Experiments

• Experimental units (specimens)– Knowledge of their characteristics– Relation to unknowns of future interest

• Response– Univariate—interval-scale variable– Multivariate/Functional

• Protocol parameters—parameter design• Cost of experimental runs—high throughput?

Outline

• Alternative statistical formulations– Classification based on cases and controls– Measurement of an interval-scale variable

• Aspects of protocol development– Property of interest– Realization of protocol

• Multivariate and functional measurements

Statistics for Classification

• Assume gold standard for disease status• Evaluate marker on training data

– Sensitivity—true positive rate– Specificity—1 – false positive rate

• Continuous test result—ROC curves• Multivariate test result—classification,

discriminant analysis

Pepe, et al., J. National Cancer Institute, 2001Specimen Selection

1. Wide spectrum of tumor and non-tumor tissue2. Serum from cases and controls in a target

screening population3. Apparently healthy subjects monitored for

development of cancer4. Cohort from a population that might be targeted5. Subjects randomly selected from populations in

which the screening program is likely

Thinking Outside the Box

• Bottom line is prediction of disease status• Definitive gold standard may not be

available• Including laboratory sources of error in

training data is a problem• There are metrological experiments that do

not require a gold standard

The Role of Science

• Given valid training data, statisticians can proceed without scientific knowledge

• In the classification approach, scientific thought must go into specimen selection

• In the metrological approach, focus is on a property to be measured

• Scientific thought must go into the relation of the metrological property to biomarker goals

Statistics for Metrology

• Focus (as best one can) on the property to be measured, an interval- or ratio-scale variable

• Specify a baseline measurement protocol• Experiment with realizations of alternative

protocols• Optimize repeatability (at least) and then ask if the

measurement protocol is adequate for the purpose

Framework of Metrology

• Relation between property and protocol obtained scientifically or through realization

• Metrology explores faithfulness of realization before adequacy for the purpose

Property

Realization Protocol

Some Metrological Experiments

• Protocol development through classes of units known to differ in the property of interest

• Protocols linked to a scientific definition of the property of interest in such a way that all sources of error can be assessed (definitive methods)

• Sets of protocols that measure the same property but are based on different scientific principles (independent methods)

Aspects of Performance

• Repeatability• All manner of reproducibility

– Operator, equipment– Inter-laboratory

• Noise factors, effect of sample matrix• Calibration• Measurement assurance• Uncertainty components, type A and type B

uncertainties

Experimental Units(Reference Materials)

• Homogeneity (solution versus particles)• Quantity (cost)• Adaptable to high-throughput experiments• Known value of the property of interest• Classes with different values of the property

of interest

From Univariate to Functional

• Carryover has been done for classification• Extending measurement performance

concepts to multivariate and functional responses is still a challenge

• Chemometrics is the key word for much of the literature in this area

Functional Principal Components Analysis (Ramsay and Silverman)

• Metrologists like to look at the spread of a batch of measurements (outliers, more than one mode)

• For functional measurements, functional PCA provides a way to look at the spread

• Consider results of functional PCA on Petricoin’s Lancet…/Normal Healthy (SPLUS, Ramsay’s software)

• Main purpose is to illustrate metrological thinking

1800 1900 2000 2100 2200 2300

M/Z

05

1015

20

INTE

NS

ITY

Lancet ... Normal Healthy SELDI-TOF Mass Spectra

2200 2220 2240 2260 2280 2300

M/Z

0.00

0.05

0.10

0.15

0.20

0.25

VA

LUE

OF

PC

CU

RV

E

Functional Principal Components Analysis

2200 2220 2240 2260 2280 2300

M/Z

-0.2

-0.1

0.0

0.1

VA

LUE

OF

PC

CU

RV

E

2200 2220 2240 2260 2280 2300

M/Z

-0.1

0.0

0.1

0.2

0.3

VA

LUE

OF

PC

CU

RV

E

2200 2220 2240 2260 2280 2300

M/Z

-0.1

00.

000.

100.

20

VA

LUE

OF

PC

CU

RV

E

Functional Principal Components Analysis

2200 2220 2240 2260 2280 2300

0.0

0.5

1.0

1.5

2.0

2.5

3.0

INTE

NS

ITY

+++++

+++

++++++++++

++++

+++++

++

+++++++++

++++++

++++++

+

+

+

+

+

+

+

++++

+

+

++

+

+

+++

+++

+++++++++++++++++

+++++++++++

++++++

++++

++++

++++

+++

+++

+

++

--------

----------

------

---

--

---------

------

-----

---

-

--

-

-

-----

----

------

--------

------

-----------

----------

----

----

----

------

-

--

M/Z

Rotated Functional Principal Components Analysis

2200 2220 2240 2260 2280 2300

0.0

0.5

1.0

1.5

2.0

INTE

NS

ITY

+++++++

+

++++++++++

++++

++

+++

++

+++++++++

++++++

+++

++++

+

+

+

+

+

+

+

+++

+

+++

+

+++

+

++++++

+++++

+++++

+++++++

++++++++

++++++

++++

++++

+

++

+

++

+

++

+

+

++

------

--

----------

----

--

---

--

---------

------

---

----

-

-

-

-

-

-

----

-

-

--

-

-

---

-----

----

--------

---------

-----

-----

-----

----

-

--

-

--

-

--

-

-

--

M/Z

2200 2220 2240 2260 2280 2300

0.0

0.5

1.0

1.5

2.0

INTE

NS

ITY

+++++

+++

+++++++

+++++++

++

+++

++

+++++++++

++++++

+++

++++

+

+

+

+

+

+

++++

+

+

++

+

++++

++++++

++++

++++

+++++

++++++++++++

++++++

++++

++++

+

++

+

++

+

+++

+

++

--------

----------

----

--

---

--

--------

----

---

---

----

-

-

-

-

-

-

----

-

-

--

-

---

-

-----

----

-------

-------

--------

-----

-----

----

-

--

-

--

-

--

-

-

--

M/Z

2200 2220 2240 2260 2280 2300

0.0

0.5

1.0

1.5

2.0

INTE

NS

ITY

+++++++

+

++++++++++

++++

++

+++

++

+++++++++

++++++

+++

++++

+

+

+

+

+

+

++++

+

+

++

+

+++

+

+++++

+++++

+++++++

++++++++++++++

+++++

+++++

+++

++

++

+

++

+

++

+

+

++

-----

---

----------

----

--

---

--

---------

------

---

----

-

-

-

-

-

-

----

-

-

--

-

---

-

-----

----

--------

------

------

----

--------

----

-

--

-

--

-

--

-

-

--

M/Z

2090 2100 2110 2120 2130

PCA function 1 (Percentage of variability 92.2 )

01

23

45

6

INTE

NS

ITY

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++

++++++++++++++++++++++++++++++++++++++++++++

+++++++++

+

--------------------------------------------------------------------

------

------

--------------------------------------

----------

M/Z

Functional Principal Components Analysis (Not Rotated)

2090 2100 2110 2120 2130


1.0

1.5

2.0

2.5

3.0

3.5

INTE

NS

ITY

++++++++++++++++++++

++++++++++++++++

++++++++++

+++++++++++++++

+++++++++

++++++

++++++++

++++++++++++++

++++++++

+++++++

+++++

+++++++

++

+

------------------

-----

-----------------------

-------------

--------

--------

------

--------------------------

-----------------

---

-

M/Z

2090 2100 2110 2120 2130


1.0

1.5

2.0

2.5

3.0

INTE

NS

ITY

+++++++++++++++++++++++++++++++++

++++++++++++++

++++++++++++++

+++++++++++

+++++++

++++++++++++++++++++++++

+++++++

+++++++

++++

+++

+++

+

-----------------------------------

-----------

------------------

-----

---

-

----

--

-

------

------------

-

-

-----

--

------

----

----

----

--

-

M/Z

2090 2100 2110 2120 2130


1.0

1.5

2.0

2.5

3.0

INTE

NS

ITY

+++++++++++++++++++++++++++++++++++

++++++++++

+++++++++++

+++

++++++++++

+++++++++

+++++

+++++++++++++++++++++

++++++

+++++++

++++

+++

+

++

+

------------------------

---------

--------------

---------

-----

----------

-----

---

-

----

-------

-----

----------

-

------

----

----

---

---

-

M/Z

Conclusion

• Producing large data sets has become easier except perhaps for selecting individuals with a particular disease status

• With scientific and statistical reasoning, the advances in experimentation technology can be used to speed biomarker development

• Statisticians have a role in formulating overall experimental strategy, allocating effort among different approaches

Documents

Metrological Experiments in Biomarker Development (Mass Spectrometry—Statistical Issues)