Center for Scientific ReviewNational Institutes of HealthDepartment of Health and Human Services

Toni Scarpa

CSR Where we are and where we would like to be

PRAC, August 2006

Changes in CSR Operations Our Vision for Peer Review

This is CSR

1. Increased Communication and Transparency

2. Increase Uniformity

3. Increase Efficiency

Changes in CSR Operations

Changes in CSR Operations

1. Increased Communication and Transparency

Within CSR– Biweekly meetings with SRAs and other staff– New comprehensive communications plan

With NIH and other Agencies

With the Scientific Community– Expanded Peer Review Notes newsletter– Six Open House Meetings with leaders of scientific societies

and disease groups

Changes in CSR Operations

2. Increase Uniformity

Summary Statements – Post all within 1 month of meeting (97.3%)– Post new investigator summary statements within 1 week– Produce complete and structured resumes

Unscoring– Unscore 50% (49.7%)

Best Practices Committee Assessments– Appropriate Rosters– Types of Meetings– Structure of Summary Statements

Changes in CSR Operations

3. Increase Efficiency

Electronic Submission Text Fingerprinting, Artificial Intelligence Software

– Assigning applications to Integrated Review Groups or Study Sections

Major pilot to directly assign applications to Integrated Review Groups and/or Study Sections will begin in October, with implementation by February 2007.

Scoring – Current Situation

• Compression of scoring• “Approach is King” and “Significance” has been

downgraded• Reasons for scoring often not clear in summary

statements• Scoring of A2 applications may be

inappropriately influenced by reviewers knowing A1 score

IRG and Study Sections Realignment

• Increasing concern from applicants, societies and diseases groups about the variance in the breadth of science covered by study sections.

Monitoring IRGs and Study Sections

1. Every IRG is being assessed by all senior CSR staff every 2 years in addition to the current 5-year assessments involving all stakeholders.

2. Emergent problems addressed by working groups with leaders from the extramural community, and senior NIH/CSR staff.

3. Substantive issues/changes reviewed by the NIH Peer Review Advisory Committee.

4. Most study sections visited by CSR’s Director and senior staff at least once a year.

5. All retiring study section chairs called by CSR’s Director to learn about problems and possible improvements.

6. All summary statements read by CSR’s Director.7. Multiple outreach efforts now gather fresh input.

IRG Review Schedule

Scheduled 2007 -- 9 IRGs

Biology of Development and Aging (BDA)

Infectious Diseases and Microbiology (IDM)

Biobehavioral and Behavioral Processes (BBBP)

Cell Biology (CB)

Musculoskeletal, Oral and Skin Sciences (MOSS)

Oncological Sciences (ONC)

Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB)

Respiratory Sciences (RES)

Renal and Urological Sciences (RUS)

Scheduled 2006 --14 IRGs

Biological Chemistry and Macromolecular Biophysics (BCMB)

Cardiovascular Science (CVS)

Bioengineering Sciences and Technologies (BST)

AIDS and Related Research (AARR)

Risk, Prevention, and Health Behavior (RPHB)

Genes, Genomes and Genetics (GGG)

Digestive Sciences (DIG)

Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR)

Brain Disorders and Clinical Neuroscience (BDCN)

Integrative, Functional and Cognitive Neuroscience (IFCN)

Molecular, Cellular and Developmental Neuroscience (MDCN)

Hematology (HEME)

Immunology (IMM)

Health of the Population (HOP)

Monitoring IRGs and Study Sections

1. Every IRG is being assessed by all senior CSR staff every 2 years in addition to the current 5-year assessments involving all stakeholders.

2. Emergent problems addressed by working groups with leaders from the extramural community, and senior NIH/CSR staff.

3. Substantive issues/changes reviewed by the NIH Peer Review Advisory Committee.

4. Most study sections visited by CSR’s Director, Deputy Director, and senior staff at least once a year.

5. All retiring study section chairs called to learn about problems and possible improvements.

6. Multiple outreach efforts now gather fresh input.7. Open house meetings to be held.

Broad Scientific Areas

• Biomolecular (4): Biological Chemistry and Macromolecular Biophysics (BCMB); Bioengineering Sciences and Technologies (BST); Cell Biology (CB); Genes, Genomes and Genetics (GGG)

• Integrated Biological (5): Immunology (IMM); Hematology (HEME); Cardiovascular Sciences (CVS); Respiratory Sciences (RES); Biology of Development and Aging (BDA)

• Integrated Biological (4): Digestive Sciences (DIG); Musculoskeletal, Oral and Skin Sciences (MOSS); Renal and Urological Sciences (RUS) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR)

• Disease-based (4): AIDS and Related Research (AARR); Infectious Diseases and Microbiology (IDM); Oncological Sciences (ONC);  Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB)

• Neurological (3): Brain Disorders and Clinical Neuroscience (BDCN); Integrative, Functional and Cognitive Neuroscience (IFCN); Molecular, Cellular and Developmental Neuroscience (MDCN)

• Behavioral/Social (3): Biobehavioral and Behavioral Processes (BBBP); Health of the Population (HOP); Risk Prevention and Health Behavior (RPHB)

Our Agenda

1. Shorten the review cycle

2. Address concern that clinical research is not properly evaluated

3. Improve the assessment of innovative, high- risk/high-reward research

4. Do more to recruit and retain more high-quality reviewers

Our Agenda

1. Shorten the review cycle

Shortening the NIH Review Cycle, Initial Steps

Pilot study with ~600 new investigators in 40 study sections who may revise and resubmit for the very next review cycle 4 months earlier than before

(Effective Feb 06)

Summary statements posted within one month after the study section meeting

Short Review Cycle Pilot of New Investigator R01 Applications

New Investigator R01 Applications Number Percent

Reviewed in Pilot 631 100%

Amended and Submitted for the Next Round 86 14%

R01 A1 Resubmissions in the Next Council Round

0

20

40

60

80

100

120

January 2005 May 2005 October 2005 January 2006 May 2006 October 2006

Original Council

R01 A1 Resubmissions in the Next Round

Our Agenda

1. Shorten the review cycle

2. Address concern that clinical research is not properly evaluated

17.5617.56

22.0722.07

M. Martin, CSR/NIH/DHHS

Our Agenda

1. Shorten the review cycle

2. Address concern that clinical research is not properly evaluated

3. Improve the assessment of innovative, high- risk/high-reward research

Our Agenda

1. Shorten the review cycle

2. Address concern that clinical research is not properly evaluated

3. Improve the assessment of innovative, high- risk/high-reward research

4. Do more to recruit and retain more high-quality reviewers

0

20,000

40,000

60,000

80,000

1998 1999 2000 2001 2002 2003 2004 2005 2006

Fiscal Year

Nu

mb

er

of

Ap

plic

ati

on

sApplications Received for all of NIH and Applications

Referred for CSR Review, FY 1998 - 2006

NIH Applications

Applications for CSR Review

1.05

1.1

1.15

1.2

1.25

1.3

1.35

1.4

1.45

1980

1982

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

Number of Research Grant Applications/Applicant

Growth of R01 Applications Reviewed at CSR vs. Other ICs

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

2001 2002 2003 2004 2005 2006

CSR Other

Growth of R01 Applications Reviewed at CSR vs. Other ICs

Growth of R21 Applications Reviewed at CSR vs. Other ICs

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

2001 2002 2003 2004 2005 2006

CSR Other

Growth of R21 Applications Reviewed at CSR vs. Other ICs

Institutes and Centers Use R21s Differently

• R21s fund ~14 types of efforts, such as—

– Exploratory/developmental research – Junior investigators– Phase I/II clinical trials– International research planning – High-risk/payoff research

• R21s have broad parameters—

– Award period: 1-3 years– Funding allowed: $100K-$450K– Research Plan: 10-20 pages

0

2

4

6

8

10

12

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Regular

SEP

Average Number of Applications Per Reviewer

October Council Only

Academic Rank of All CSR ReviewersStanding and Ad Hoc Combined

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

1998 1999 2000 2001 2002 2003 2004 2005 2006

FY of IRG

PROFESSOR ASSOCIATE PROFESSOR ASSISTANT PROFESSOR OTHER

Academic Rank of All CSR Reviewers Standing and Ad Hoc Combined

Reviewers – Current Situation

• Far too many reviewers on study sections– Broader science– Decrease in reviewer load – Unnecessary

• Too many ad hoc reviewers• Different processes for enlisting chartered and

ad hoc members• Often too many telephone reviewers

Challenges to Recruiting Reviewers

• Manage the need for 20,000 reviewers• Decrease the number of reviewers and

increase the quality• Increase the number of applications

reviewed without extra workload• Recruit and retain the best reviewers• Have smaller study sections with fewer

ad hoc reviewers

Possible Approaches for Recruiting and Retaining the Best Reviewers

• Require less travel by using electronic review modes

Expanding Peer Review Platforms

Electronic Reviews• Telephone Enhanced Discussions• Video Enhanced Discussions• Asynchronous Electronic Discussions

Our Goal: 10% of all reviews to be electronic in 2007

0

500

1000

1500

2000

2500

2004 2005 2006

Electronic Review Methods - Number of Applications Reviewed by Year

Virtual

Teleconferences

Electronic Review Methods Number of Applications Reviewed by Year

Possible Approaches for Recruiting and Retaining the Best Reviewers

• Require less travel by using electronic review modes

• Have shorter meetings• Use Various Review Platforms• Unscore 40% of the F32s (postdoctoral

fellowships)• Shorten Applications

Size of Grant Applications

• RO1 • Will increase number of applications reviewed by

reviewers• Will decrease the number of reviewers in a study section• May be combined with a change in format of the

application, more consonant to review criteria• Strong support by councils and scientific leadership• May result in better focus on significance (and

innovation)

Trans-NIH Committee to Shorten the Application

Chairs

Robert Finkelstein, Ph.D. (Co-Chair), NINDS

Donald Schneider, Ph.D. (Co-Chair), CSR

Members

Mary Custer, Ph.D.CSR

Ann A. Hagan, Ph.D.NIGMS

Craig A. Jordan, Ph.D. NIDCD

Sherry Mills, M.D., M.P.H.OER

Philip F. Smith, Ph.D. NIDDK

Barbara Spalholz NCI

Betsy Wilder, Ph.D.NIDDK

Alan L. Willard, Ph.D.NINDS

Special Thanks