Environmental Protection Agency

Statistical Sampling Plans: Collaborative Efforts between Statisticians

and Subject Matter Experts

by Marla Smith

ICES-IIIJune 2007

22

Overview

• Our program and its objective

• Surveys in our program

• Multi-disciplinary teams

• Survey development

• Lessons learned and examples

33

1970 President Nixon creates EPA

1972 Clean Water Act amendments shifts: -- Reliance on violations of water and quality standards as the primary enforcement tool, to-- Establishment of specific technology-based effluent guidelines that are enforceable as permit conditions

Milestones -- Environmental Protection22 June 1969 Cuyahoga River Fire

(Cleveland, OH)– Time Magazine:

“Some river! Chocolate-brown, oily, bubbling with subsurface gases, it oozes rather than flows.”

44

What are Effluent Guidelines (EG)?• Restrictions on the types and amounts of

pollutants that can be discharged into navigable waters of the U.S. by various industries

• Based upon best available process and/or wastewater technologies and performance

• Separately implemented through permits

55

• After collecting data from industry, EPA analyzes the data to identify or refine:– Technology Options and the related pollutant

reductions and incremental compliance costs– Economic Impacts– Industry subsectors– Environmental Benefits

Industry Identified Sampling,

Industry Survey, etc.Formulation of Optionsand Decision-making

Analysis of Data (including public comments)

Why Do We Conduct Surveys?

66

EG Surveys

• One-time surveys

• Duty to respond is mandatory under the Clean Water Act

77

EG Teams

• Each team has its own personality– Statistician has to adapt accordingly

• Typical team includes:– Team Leader– One or Two Engineers– One Economist– One Environment Assessor– One Statistician

• Everyone supports multiple projects– Some more part-time than others

• Each team member has contract resources– Additional level of coordination

• Management and lawyers provide oversight

88

Survey Development: EG Team

• Each project team expends considerable effort to:– Learn about each industry– Determine the data needs for each

type of analysis– Refine study objectives– Find a sample design that will satisfy

all requirements– Develop the survey instrument

99

Survey Development:Statistician

• Leads survey design work• Translates team objectives into

specifications for statistics contractor• Reviews statistics contractor products

– Often reviews multiple versions before sharing one with the team

• Discusses sample design concerns with management

• Documents and defends sample design as part of government approval process

• Reviews final draft of questionnaire

1010

Examples from 3 Surveys

• Airport Deicing EG– Airlines and airports perform deicing and

anti-icing of aircraft and airfield pavement• Can result in environmental impacts

– Survey Design includes two industries:• Airports: Single phase:

– Detailed questionnaire sent to stratified sample

• Airlines: Two-phase:– Screener sent to census of relatively active

airlines selected airports

– Detailed questionnaire sent to subset of deicers

1111

Overview: Examples (cont)• Drinking Water EG

– Residuals are created when drinking water is treated. Sometimes the residuals are discharged back into the river, stream, etc.

– Survey Design: Stratified sample

• Biosolids – not an EG– Biosolids result from the treatment of

domestic sewage in a treatment facility (different than drinking water treatment)

– Survey Design: Stratified sample– Team aspects similar to EG

1212

Lessons Learned: General

• Listen– Learn as much as possible– Need to consider what isn’t said– Participate in non-survey meetings

• Ask questions– Ask the same question at different meetings– Keep asking until there’s an answer

• Revisit and confirm previous decisions– May get different answers at different times

• Leave the room last at a meeting– Listen to what is (still) being discussed– Allow time for questions about statistics and data

• Keep it simple– Make sure everyone understands– Bottom line is more important than how we got there

1313

Lessons learned: Target Pop.

• Concept can be difficult for non-statisticians– Target population versus available data– Not everyone needs or should get a questionnaire

• Sub-populations need to be defined– Narrows down the target population– Provides insight to study objectives– Reduce eligible population

• Each pick means a loss of a pick elsewhere

– Verify that legal authority encompasses all members of the target population

– Decide if oversampling is appropriate

• Write down the definition– Team needs to review and agree

1414

Target Pop. – Examples

• Airport Deicing– Not everyone should get a questionnaire:

• Military airports were excluded from survey– Questionnaire not the right vehicle for collecting data– Affordability questions are not relevant

• Still included in EG development

• Biosolids– Target population definition:

• Two alternatives: facilities existing today or in 1988– Team knew a lot about the 1988 facilities– Existing data indicated differences in population over

time• Management wanted estimates of current conditions

– Subpopulation review:• 0.2 percent of the facilities generate 32 percent of the

total nationwide flows• Team decided to oversample

1515

Target Pop. – Examples

• Drinking Water– Target population definition:

• Difficult to describe (series of criteria)• Important during sample draw review

– Subpopulation considerations:• Excluded very small systems

– Approximately 98 percent of all 160,000 drinking water systems are very small

– Collectively, they have little environmental impact– Exclusion greatly simplified sample frame issues

• Oversampled certain systems – Team concluded they were most likely to have

biggest impact in terms of engineering, economics, and environment assessment

– Ratios based upon engineering judgment

1616

Lessons Learned: Design

• Recognize that using a sample is a leap of faith for non-statisticians– Sometimes need to involve management– Estimate resources for different sample sizes

• Keep stratification under control– Too many variables will complicate the analyses and

unnecessarily increase sample sizes• Accommodate ‘favorite’ picks early on

– Team will reject sample draws without them– Provides additional insight into study objectives– Often can conclude these facilities are different

enough that they can only represent themselves• Prepare to develop multiple designs as the

team goes through its thought process• Keep management involved

1717

Design – Examples

• Airport Deicing– Listen to what’s not said:

• Proposed design: Stratified by four hub sizes– Hub sizes readily available in sample frame– But, team wasn’t planning to stratify by size in any

analyses• Final design: Stratified by two size categories

– This approach will simplify our data analyses

• Drinking Water– Keep management involved

• Proposed two-phase design– Management questioned sample size

• Final design– One stage with reduced stratification– Less detailed, shorter, questionnaire

– Keep stratification under control• Removed one variable to reduce sample size

1818

Lessons Learned: Draw

• Include time in the schedule for a second draw and design tweak– Team generally concludes that the first draw ‘will

never work’• Reasons are often valid• Discussion may reveal additional study objectives not

previously recognized by the team

– Design tweaks are generally very close to original design

– Statistical adjustments to responses are possible• Team needs to compare analysis complications

versus redrawing sample

• Also recognize that no draw will be perfect

1919

Draw – Examples

• Drinking Water: No draw will be perfect– Team wanted to reject certain systems based upon

non-frame data– Team retained the draw after further review

• Concerns about bias• Judgment that statistically valid adjustments can be

made later• Strong management support for statistics arguments

• Biosolids: Second draw was necessary– Contact information not available for small facilities

• Tend to have part-time staffing• Difficult to find person with the key

– Team redefined target population to exclude smalls– Team able to move forward with the second draw

2020

Conclusion

• Listening and asking questions is a critical part of developing survey designs

• Flexibility is important in redesigning and reevaluating to find a workable design