March 17, 2015 ENCLOSURE 1

1

CONSOLIDATED AGENCY COMMENTS ON ERM DOCUMENT

“Data Quality Objectives Framework OU-1 Phase 1B RI”

Dated February 20, 2015

NOTE: These comments on the draft Phase 1B DQO document reflect discussions and

clarifications reached during scoping meetings held in Salt Lake City on March 11-12, and

reflect the main changes that the Agencies feel are necessary to ensure that the revised DQO

document is in accord with the discussions at the meetings. EPA reserves the right to provide

additional comments on the revised draft of the DQO document.

Major Comments

1. The DQO document uses an inconsistent and confusing nomenclature, often misusing the

term “COPC.” For example, stating that samples will be analyzed for COPCs is incorrect,

because contaminants of potential concern have not yet been selected. The DQO document

should be re-written to clearly state that all samples collected in Phase 1B will be analyzed

for the full list of chemicals provided in Worksheet #15 of the Phase 1A SAP. The list of

COPCs for the inner PRIs will be determined based on the results obtained from the samples

collected during the Phase 1B program.

2. As written, the DQO document implies that data sets of size 10-15 will provide UCL values

that will be adequate. This is not correct, and the text must be revised to clarify that UCL

values calculated from the data sets of this size MAY OR MAY NOT be adequate for well-

informed risk management decision making. Accordingly, the goals of the DQO presented

in Table 11-2 should be revised as follows:

Obtain sufficient data to allow for:

reliable identification of COPCs (human and ecological) in each PRI

calculation of 95% UCL values to support an initial assessment of human and

ecological risks

initial characterization of the nature and extent of site-related contamination

Do not use a phrase such as “reliable” or “acceptable” UCLs. All 95% UCLs (if calculated

correctly) are reliable and acceptable, in that they identify a value that has a 95% probability

of exceeding the true mean. Whether any specific UCL value is adequate for well-informed

risk-based decision-making can only be determined by performing the initial risk calculations

and identifying cases in which the risk conclusion based on the UCL is substantially different

from the conclusion based on the sample mean. In the absence of this process, the only way

to ensure that UCLs are adequate for decision making is to substantially increase the number

of samples collected.

March 17, 2015 ENCLOSURE 1

2

Accordingly, and in accord with discussions at the scoping meetings, the sample size for each

PRI should be revised to be a minimum of 14. The statistical justification for this sample

size should be revised to reflect the original COPC-based statistical assessment used in the

Phase 1A SAP, rather than an assessment of cost effectiveness (“diminishing return” in the

margin of error (MOE) between the UCL and the sample mean).

While not required, the DQO may also include an evaluation of the expected MOE between

the UCL and the sample mean for data sets of size 14, as a function of the expected range of

CVs. If presented, this should be based on an assumption that most data sets will be skewed

rather than normal. This is necessary because the historic data clearly demonstrate that

nearly all data sets from the inner PRIs are skewed, and many are well-approximated by a

lognormal distribution. A simple calculation of the expected MOE is not sufficient, because

this ignores the fact that both the sample mean and the sample standard deviation are random

variables, and hence the MOE is also a random variable. For example, if the true distribution

is assumed to be lognormal with a mean of 100 and a standard deviation of 250 (CV = 2.5),

the expected MOE is 2.94. However, if sampling variability is accounted for, the actual

distribution of MOE values is as shown in Figure 1. As seen, actual MOE values may be

much higher than the expected value. The rate of false positive decision errors is also quite

high for any COPC with a true mean greater than 0.1 times the RBC, as shown below:

Mean/RBC False Positive Rate

0.1 16%

0.2 37%

0.3 54%

0.4 66%

0.5 75%

0.6 82%

These calculations demonstrate and emphasize why data sets of size 14 may or may not be

adequate to support reliable risk management decision making, depending on the actual ratio

of the mean to the RBC and of the actual standard deviation.

3. The revised DQO document should include an independent 7-step DQO for a background

study of selected COPCs in soil. The goal of the study should be to identify one or more

upland and wetland habitat locations that are suitable for characterization of abiotic and

biotic concentration values of naturally-occurring or ubiquitous anthropogenic chemicals that

would be expected at the site in the absence of site-related contamination. The DQOs should

include a clear presentation of the null hypothesis (site > bkg), the statistical approach that

will be used to test the null hypothesis, and a demonstration that the sample size selected will

be sufficient to minimize the risk of decision errors. EPA recommends that the guidance

provided in EPA (2002) be followed. EPA expects that it should be possible to complete the

March 17, 2015 ENCLOSURE 1

3

text of DQO Steps 1-6 based on the discussions at the meeting. While Step 7 cannot present

final decisions about sampling locations, the text of Step 7 should discuss how suitable

candidate locations will identified and evaluated.

Also note that a background study of water (surface water and groundwater) is also needed.

EPA has agreed to provide initial thoughts on the DQOs and design of such a study during

one or more calls between the EPA and ERM toxicologists and during upcoming risk

assessor’s conference calls. A background water study should proceed according to a

timeline that informs Phase 2 collection of data in 2016 but implementation schedules are

still being contemplated.

Minor Comments

1. Page 3, middle paragraph. Use of the maximum value or the 95%UCL (whichever is lower)

as the EPC is no longer Agency policy. Selection of EPC values must follow current EPA

recommendations (see Section 1.10.2 of ProUCL Version 5.0.00 Technical Guide).

2. Page 3, last full paragraph. The following sentence is incorrect: “The existing set of

historical (pre-CERCLA) Inner PRI data have been judged by USEPA as not adequate for

use in risk assessments, including selection of COPCs and calculating 95UCLs.” While

EPA has determined that the historic data (alone) cannot be used for COPC selection or

UCL calculations, EPA has stated that, if historic data are found to be sufficiently

statistically similar to data collected in Phase 1B, it may be appropriate to combine the data

to increase sample size and decrease uncertainty. If so, then the historic data may be

included in the data set used for COPC selection and risk assessment.

3. Page 6, Section 11.3.2. When discussing the use of the LRMS method, include a brief

description (presumably detailed in the SOP) that summarizes when the method is expected

to be appropriate, and the triggers for re-analysis by HRMS (if needed). As per discussions

during the scoping meeting, criteria for reanalysis should be based on reliable quantification

of risk drivers, not high concentration low toxicity analytes. Refer to EPA’s letter of April

11, 2014, for EPA’s recommendations.

4. Page 8, Section 11.6. The use of the word “precision” to characterize the relation between a

sample mean and the 95% UCL of the sample mean is not correct. Precision is a measure of

the reproducibility between repeat analyses, while the relation between UCL and sample

mean is a measure of uncertainty. The document should be revised to use the term

“uncertainty” rather than “precision” to describe this relation.

March 17, 2015 ENCLOSURE 1

4

SUPPLEMENTAL COMMENTS ON THE PHASE 1A/B SAP AND ASSOCIATED SOPS

Although the following comments are not specifically relevant to the draft DQO submitted by

ERM, these comments are relevant to the Phase 1A/B SAP and associated SOPs that will be

required to achieve the DQOs.

1. When sampling sediment from water-covered locations in the lagoons, every effort must be

made to ensure that samples will be obtained from the target locations. Moving a sample to a

location that is accessible from the shoreline requires EPA approval.

2. The SAP for this project must establish a schedule that will ensure that all samples specified

in the DQOs for both the inner PRI study and the background study for soil and water (if

appropriate) will be collected, analyzed, and validated in time to allow thoughtful assessment

of the data in order to inform the requirements on any Phase 2 sampling or studies that may

be required in 2016.

CITATIONS

EPA. 2002. Guidance for Comparing Background and Chemical Concentrations in Soil

for CERCLA Sites. U.S. Environmental Protection Agency, Office of Emergency and Remedial

Response. 540-R-01-003, OSWER 9285.7-41. September 2002.

ProUCL Version 5.0.00 Technical Guide

http://www.epa.gov/osp/hstl/tsc/ProUCL_v5.0_tech.pdf

March 17, 2015 ENCLOSURE 1

5

FIGURE 1

DISTRIBUTION OF MOE VALUES FOR DATA SETS OF SIZE 14

DRAWN FROM A LOGNORMAL DISTRIBUTION WITH

MEAN 100 and STANDARD DEVIATION 250