Office of the Science AdvisorOffice of the Science Advisor

Peer Review2006 Peer Review Policy and

3rd edition of the PR Handbook

Barbara KlieforthOffice of the Science Advisor, ORD

2

Long History of Peer Review at EPA

Long History of Peer Review at EPA

1993 Peer Review Policy1998 1st Guidance Handbook2003 epa.gov/si Science Inventory2006 New Policy and 3rd edition Handbook;

http://epa.gov/osa/spc/peerreview/

3

2004 OMB Peer Review BulletinKey Features

• Highly influential scientific assessments, influential scientific information

• Peer review expectations• Public comment on peer review plans• Annual reports to OMB

4

Scientific Information

Factual inputs, data, models, analyses, technical information, or scientific assessments

5

Influential Scientific Information (ISI)

Will have or does have a clear and substantial impact on important public policies or private sector decisions

Same definition as used in the Information Quality Guidelines• Major per the old Peer Review Handbook• Supports top Agency actions (including economically significant

actions)

6

Scientific AssessmentScientific AssessmentEvaluation of a body of scientific or technical

knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties.Includes: state-of-the-science reports, technology assessments, weight-of-evidence analyses; meta-analyses; health, safety, or ecological risk assessments; toxicological characterizations of substances; integrated assessment models; hazard determinations; or exposure assessments

7

Highly Influential Scientific Assessment (HISA)

Highly Influential Scientific Assessment (HISA)

• Could have a potential impact of more than $500M in any year

• Is novel, controversial, or precedent-setting, or has significant interagency interestOverlaps with definition of old “major”

8

All EPA work products vs those considered ISIs or HISAs

All EPA work products vs those considered ISIs or HISAs

All work products EPA may generate or use

Influential Scientific Information

Highly Influential Scientific Assessments

9

Peer Review Expectations for Other Information

Peer Review Expectations for Other Information

• Most products in Science Inventory are not ISI or HISA

• Should determine if product would benefit from peer review and, if so, what type of review to conduct

• Should also consider peer input

ID work product & put in SI

Planning a Peer ReviewPlanning a Peer ReviewPlanning a Peer Review

Is it ISI or HISA?

Would it still warrant PR?

Is PR exemptor waived?

Does it consist only of science previously PR’ed?

Peer Review needed:•ID charge•ID key staff•Create PR record•Ensure funding•Determine time frame

PR Not Needed

no

yes

no

yes

no

no

Conducting a Peer ReviewConducting a Peer Review

Complete PR & Obtain comments

Work ProductReady for review

Develop Charge

Select Mechanism

Set Timelines

Select Peer Reviewers

Send Materials &Conduct Review

(Add all to PR Record)

Document how used &Put in PR record

Receive Comments from Peer Reviewers

Completing a Peer ReviewCompleting a Peer Review

Brief Decision-Maker, if appropriate

Are PR comments usedin Work Product?

Document why not &Put in PR record

Brief Decision-Maker on approach to handling

comments

Finalize Work Product

Update PR record

DONE!

yes

no

13

Science Inventory/Peer Reviews

Science Inventory/Peer Reviews

• Science Inventory• 1995: Approximately 100 products• 2005: >5,000 activities and products

• Peer Reviews in 2004• Over 2,000 (includes journal articles)• Estimated annual cost: $10M

14

Contents of Peer Review PlansContents of Peer Review Plans• Title/subject/purpose, EPA contact, if HISA or ISI• Timing of the review (including deferrals)• If panel or letter review (or through alternative procedure)• If and how public will be asked to comment on work

product• If EPA will provide public comments to peer reviewers• # of reviewers (3 or fewer, 4-10, >10)• Description of primary disciplines or expertise needed• Whether reviewers will be selected by EPA or outside

organization• If the public (including scientific/professional societies) will

be asked to nominate reviewers

15

Roles/ResponsibilitiesRoles/Responsibilities• AAs/RAs• Science Policy Council• ORD/OSP• Peer Review Coordinators• Peer Review Leaders

16

AA/RAAA/RA

• Is ultimately responsible for implementing policy• Determines ISI/HISA• Identifies other products needing peer review• Makes peer review decisions• Allocates resources for peer review

• Ensures that her/his office’s science activities and products are captured in the Science Inventory

17

Science Policy Council (SPC)Science Policy Council (SPC)• Develops EPA science policy and provides

oversight

• Peer Review Advisory Group• Assists the SPC with implementing policy

Agency-wide• Responsible for revising Peer Review Handbook

18

Peer Review CoordinatorPeer Review Coordinator• Serves as the main peer review contact for

his/her organization• Provides training on peer review process and

roles/responsibilities• Coordinates the entry and update of information in the

Science Inventory• Serves as a liaison with ORD and SPC• Assists with preparing Annual Report to OMB

19

Peer Review LeaderPeer Review LeaderOrganizes/oversees peer review for a specific work

product• Selects peer review mechanism• Develops charge• Considers public comments if HISA/ISI• Documents the peer review process and results• Responds to findings• Maintains the peer review records• Works with the Coordinator on the Science

Inventory

20

In SummaryISI – What is new?

In SummaryISI – What is new?

• Expanded definition of peer review products• Offer and consider public comment on peer review

plans• Consider asking public to nominate potential peer

reviewers• Preparing Peer Review Report and making it (and

related materials) available on the internet• Discussion in preamble and certification in the

administrative record

21

Major Products(per old Peer Review Handbook)

Major Products(per old Peer Review Handbook)

• Establishes a significant precedent, model or methodology

• Addresses significant controversial issues• Focuses on significant emerging issues• Has significant cross-Agency/inter-agency implications• Involves significant investment of Agency resources• Considers an innovative approach for a previously

defined problem/process/methodology• Satisfies a statutory or other legal mandate for peer

review

22

HISA- What is new?HISA- What is new?In addition to those listed for ISI:• When feasible, soliciting public comment on

work product and allowing public to make presentation to peer reviewers

• Providing peer reviewers with significant public comments that pertain to science issues

• Preparing and posting Agency response to peer review report