1

CSR and NIH Peer ReviewMaintaining Highest Quality & Fairness

René Etcheberrigaray, M.D.

Deputy Director, CSR

Health Research Alliance Members’ Meeting

September 18, 2014 - Durham, NC.

National Institutes of Health

We seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability.

2

National Institutes of Health

Much of the biomedical research in the United States is supported by the Federal Government, primarily the National Institutes of Health of Health (NIH)

Why Has NIH Been So Successful?

Peer Review

• Focus is on funding ideas not institutions.

• Ideas spring from local researchers.

• Researchers must compete – like entrepreneurs – for funding.

• Scientific experts do the judging.

• Institutions receive funds only when their scientists submit successful applications.

• NIH program and review staff are separated.

• Scientists manage the peer review process.

3

Spending at NIH

NIH Extramural & Intramural FundingFY 2012 Enacted: $30.9 Billion

83%

17%

Spending Outside NIH$25.7 B

– Supports over 300,000 Scientists & Research Personnel

– Supports over 2,500 Institutions

– $3.4 B Intramural Research – $1.5 B Research Management & Support – $0.3 B Buildings and Facilities, Other

$5.2 B

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Suc

cess

Rat

e

Fiscal Year

Grant Success RatesFY 1978-2013

* FY 2013 success rate is a preliminary estimate

4

NIH Program Level in Appropriated Dollars and Constant 1998 Dollars

0

20,000

40,000

60,000

80,000

100,000

120,000

App

licat

ions

Actual Fiscal Year

ARRA included

ARRA excluded

*2013 estimate based on applications received through July

Number of Applications Received by Fiscal Year

5

Scientific Research & Development Spending Change from 2012 to 2013

“To Maintain Our Edge . . . ”

“we’ve got to protect our rigorous peer review system and ensure that we only fund proposals that promise the biggest bang for taxpayer dollars . . . that’s what’s going to maintain our standards of scientific excellence for years to come.”

Remarks by the President on the 150th Anniversary of the National Academy of Sciences, April 29, 2013

6

Takes Final Action

National Institutes of Health

Center for Scientific Review

Study Section

Institute

Advisory Councils and Boards

Institute Director

Assigns to IC & IRG/Study Section

Reviews for Scientific Merit

Evaluates for Relevance

Recommends Action

ResearchGrant Application

School or OtherResearch Center

InitiatesResearch Idea

ConductsResearch

Allocates Funds

Submits Application

Review Process for a Research Grant

Core Values of NIH Peer Review

• Expert Assessment

• Transparency

• Impartiality

• Fairness

• Confidentiality

• Integrity and Ethical Conduct

7

Center for Scientific Review

To see that NIH grant applications receive fair, independent, expert, and timely reviews – free from inappropriate influences – so NIH can fund the most promising research.

CSR Mission

8

CSR Peer Review – Fiscal Year 2013

• 84,000 applications received by CSR

• 73% of NIH grant applications reviewed by CSR

• 173 standing study sections

• 236 Scientific Review Officers

• 1,500 review meetings

• 17,000 reviewers

Goals of CSR

• Improve continuously:

– Fairness of review

– Quality of review

– Efficiency of review

– Morale of staff and reviewers

• Create a science of peer review

9

Separation of Peer Review and Grant Award

NIH Peer Review System for Grant Applications

First Level of Review

Scientific Review Group (Study Section)

Second Level of ReviewNIH Institute/Center Council

About 80,000 applications and 18,000 reviewers

10

Scientific Review Officer

The overall peer review process is managed by a Designated Federal Official, who has doctoral-level scientific expertise relevant to the field

The Best Reviewers

11

Review by experts: Recruiting the Best ReviewersAcademic Rank of CSR Standing Reviewers

0%

10%

20%

30%

40%

50%

60%

70%

80%

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

PROFESSOR ASSOCIATE PROFESSOR ASSISTANT PROFESSOR

Recruiting the Best Reviewers

o Move a meeting a year to the (west coast) other citieso Additional review platformso Expansion of no submission deadlines

o No submission deadlines for chartered members of study sections & frequent reviewers

o Provide flexible time for reviewerso Choice of 3 times/year for 4 years oro 2 times/year for 6 years

Year Regular Freq. Flyer Total Unique

12/2009 4799 1090 5889

12/2010 5068 3317 8385

12/2011 4978 1853 6831

12/2012 4941 1856 6797

12

The Study Section Meeting

The Study Section Meeting

Summary Statement

• Summary of Review Discussion• Essentially Unedited Critiques• Budget Recommendations• Administrative Notes• Priority Score and Percentile Ranking

Results are documented by SRO in a summarystatement and forwarded to the PI and the assigned NIH Institute or Center, where a funding decision is made.

The Summary Statement Contains

13

Confidentiality!

Review materials, application reviewers and the content of review are confidential!

• Protects you and applicant

• Do not discuss or share applications with anyone outside of the review meeting.

• If you need more expert advice to evaluate an application, discuss it with your SRO

• If an applicant asks you a question or wants to discuss review, ask them to talk to the SRO; after review they can talk to their assigned program officer.

14

Speed of Award

Days to 90% of Summary Statement

AIDS: 102

Non-AIDS (R01): 154

Days to 90% of Award

AIDS: 363

Non-AIDS (R01): 414

AIDS (All)

Non-AIDS (R01)

Group

SRG Assign+

Award

SS Release

Days to

IRG Assign

Meeting

15

Current Timeframe for R01s -- Submission to Award

Three Main Overlapping Cycles per Year

http://grants1.nih.gov/grants/funding/submissionschedule.htm

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Cycle I

Cycle II

Cycle III

AIDSAIDSAIDS

R&R Assign Review Meeting SS Council Award

Council Award R&R Assign Review Meeting SS

Review Meeting SS Council Award R&R Assign

Fairness in peer review

16

Probability of NIH R01 award by race and ethnicity, FY 2000 to FY 2006 (N = 83,188)

D K Ginther et al. Science 2011;333:1015-1019

Concerns of Reviewer Bias

Award Probabilities by NIH Funding Rank and Race/Ethnicity

17

Ginther et al. (2011) Findings

• African American applicants were 10 percentage points less likely to receive NIH research funding compared to Whites

• Suggested explanations:

– Bias in peer review

– Deficits in applicants’ grant writing abilities

• Applications with strong priority scores were equally likely to be funded regardless of race

− This suggests that problems occur at the peer review stage or earlier

NIH Response

ACD Working Group on Diversity in the Biomedical Research Workforce met throughout 2012 and developed recommendations

• Create a subcommittee on peer review

• Research to examine causes of award disparities

– Implicit Bias

– Cumulative advantage or disadvantage

• Exploration of interventions to reduce possibility of bias

18

CSR Early Career Reviewer Program Goals

• Train and educate qualified scientists to become critical, competent and fair reviewers

• Provide peer review experience to help improve applicant competitiveness

• Enrich the existing pool of NIH reviewers

Early Career Reviewer Program (ECR)

3524 ECRs have been accepted into the program

1069 have served on at least one study section

30 percent of ECRs who have served are from under-represented groups

Software developed to allow for online applications

ECR video for outreach

Outreach webinars for R15 schools

19

How to Apply for the Early Career Reviewer Program

• Instructions are at www.csr.nih.gov/ECR

• If eligible, your name will be placed into our ECR database

• You will be invited to serve as an ECR when your expertise is needed for particular applications

Methods for strengthening reviewer training methods to maximize fairness and impartiality in peer review with regards to gender, race/ethnicity, institutional affiliation, area of science, and/or amount of research experience of applicants.

Launched on May 5, 2014. Submission deadline June 30, 2014 Awards September 30, 2014

• Prizes

First - $10,000

Second - $5,000

Challenge #2 - Strategies to Strengthen Fairness and Impartiality in Peer Review

20

Detection of Bias:Text Analysis of Reviewer Comments

Task 1: Development of lexicon/thesaurus for text analysis

Task 2: Test thesaurus for ability to detect text differences by applicant race and gender

Task 3: Full Scale study of bias in reviewer comments due to investigator characteristics

Contract to be awarded Summer 2014

Bias Detection:Alter Identity of Grant Applicants

• Grant applications from White Males will be altered by:

– Gender (male vs. female)

– Race (black vs. white)

– Institutional affiliation (low vs. high research intensive)

• Comparison of scores

• Analysis of written and verbal comments of reviewers

21

America COMPETES Challenges

Challenge #1 - New Methods to Detect bias in Peer Review

How to detect bias among reviewers due to gender, race/ethnicity, institutional affiliation, area of science, and/or amount of research experience of applicants.

Launched on May 5, 2014 Submission deadline June 30, 2014 Awards September 30, 2014

Judging criteria Best Empirically Based Submission Most Creative Submission

Prizes First - $10,000 Second - $5,000

Investigator Perceptions: Survey of New Investigators and Focus Groups

Survey launched May 28, 2014

Topics

Perception of fairness of NIH peer review

Stereotype threat

Self-Efficacy for Grant Writing

Access to mentors

Institutional supports and resources

Interaction with NIH

Response to critiques

22

Pilots and Studies

• Post meeting survey pilot

• More intra-IRG ranking studies

• Innovation and conformity study

• Post meeting ranking pilot

• Anonymization pilot

• Reliability

• Replicability

This Is CSR

23

Key NIH Review and Grants Web Sites

NIH Center for Scientific Review

http://www.csr.nih.gov

NIH Office of Extramural Research

http://grants.nih.gov/

Overall Impact of Research Applications