Rules of the Road: Successful Navigation through the NCI/NIH Peer Review Process

Jeannette F. Korczak, Ph.D.Scientific Review Officer (SRO)

Resources and Training Review BranchDivision of Extramural Activities

National Cancer Institute

NIH’s “Enhancing Peer Review”

• Initiative began in early 2009 with a phased-in implementation

• Goals include:– Improving reliability of scoring– Providing quantitative feedback on

ALL applications– Easing administrative burden on

applicants, reviewers, and NIH staff

NIH’s “Enhancing Peer Review”

• Peer review emphasis is on IMPACT, rather than methodological details

• 9-point scoring system used to assess Overall Impact and each of 5 Core Review Criteria:– Only applications discussed at the

meeting receive an Overall Impact score• Other applications are “ND” (Not Discussed)

– All applications receive scores for the 5 Core Review Criteria

9-Point Scoring System

SCORE DESCRIPTOR IMPACT1 EXCEPTIONAL HIGH2 OUTSTANDING HIGH3 EXCELLENT HIGH4 VERY GOOD MODERATE5 GOOD MODERATE6 SATISFACTORY MODERATE7 FAIR LOW8 MARGINAL LOW9 POOR LOW

Scoring Process• Prior to review meeting, reviewers:

– Prepare critiques and assign scores from 1 to 9 to Preliminary Overall Impact and 5 Core Review Criteria

• At meeting, reviewers assign Final Impact Score to Discussed applications– Scores are averaged and multiplied by 10– Final Impact Scores: 10 (best) to 90 (worst)– Not discussed applications are scored ND

• After meeting, reviewers update criterion scores and critiques

Impact and 5 Core Review Criteria

• Definition of “Impact” varies across Funding Mechanisms:– For R01/R21/R03, etc., “Impact” is

likelihood for project to exert a sustained, powerful impact on the field

• in consideration of “pertinent” review criteria– For K01/K07/K99, etc., “Impact” is

likelihood for candidate to achieve scientific independence

• in consideration of “pertinent” review criteria

Impact and 5 Core Review Criteria

• 5 Core Review Criteria vary across Funding Mechanisms

• For R01/R21/R03, etc., they are:– Significance; Investigator(s); Innovation;

Approach; and Environment• For K01/K07/K99, etc., they are:

– Candidate; Career Development Plan; Research Plan; Mentor(s), Co-mentor(s), Consultant(s), Collaborator(s); and Environment and Institutional Commitment to Candidate

•

Scoring Based on Balance of Strengths and Weaknesses

• Reviewers list strengths/weaknesses for each Criterion and for Overall Impact, and assign scores based on:– Number of strengths/weaknesses – Whether strengths/weaknesses are

major, moderate, or minor – Examples:

• Score = 2: Extremely strong with negligible weaknesses

• Score = 5: Strong but > 1 moderate weakness

Maximizing Your Scores

• ALWAYS consider the guidelines for REVIEWERS when preparing your application

• See SECTION V: APPLICATION REVIEW INFORMATION of the Funding Opportunity Announcement for the review criteria guidelines

Section V for “K” Applications• Candidate:

– Potential to become independent, productive researcher?

– High quality academic, clinical, and research record?

– Commitment to becoming independent investigator in research?

– Letters of reference from well-established scientists who address review criteria and attest to candidate’s potential for independent research?

Section V for “K” Applications

• Career Development Plan:– Will contribute significantly to

candidate becoming scientifically independent?

– Appropriate content, scope, phasing, and duration, given candidate’s background and objectives?

– Adequate plans to monitor candidate’s research and career development progress?

Section V for “K” Applications

• Research Plan:– Significant scientific and technical merit

of research question, design, methods?– Relevant to candidate’s research career

objectives?– Appropriate to candidate’s stage of

research development and future goals?– Adequate plans to include both genders,

minorities and children, as appropriate, for human subjects research?

Section V for “K” Applications

• Mentors, Co-mentors, Consultants, Collaborators:– Mentor has appropriate research

qualifications?– Adequate description of quality and

extent of mentor’s role in advising candidate?

– Previous experience of team in mentoring independent investigators?

– Productive research team with peer-reviewed support?

Section V for “K” Applications

• Environment and Institutional Commitment to Candidate:– Assurance of > 75% of candidate’s effort

for “K” project?– Evidence that candidate will be integral

part of institution’s research enterprise?– Appropriate facilities, resources, and

training opportunities, including faculty?– High quality environment for candidate’s

scientific and professional development?

Other Components, as Applicable

• Additional Review Criteria - can affect the Impact Score:– Protections for Human Subjects; Inclusion

of Women, Minorities, and Children; Vertebrate Animals; Biohazards

• Additional Review Considerations -do not affect Impact Score:– Training in the Responsible Conduct of

Research; Budget and Period of Support; and Resource Sharing Plan

Protections for Human Subjects

• If research is exempt under 45 CFR Part 46, you MUST describe:– Justification for the exemption and

sources of materials– Involvement of human subjects and their

characteristics

Protections for Human Subjects• If research is NOT exempt under 45

CFR Part 46, you MUST describe:– Risk to subjects– Adequacy of protection against the risks– Potential benefits to subjects and others– Importance of the knowledge to be gained– Data and safety monitoring for clinical

trials– Plans for inclusion of minorities, members

of both genders, and children, as appropriate, for clinical research

Vertebrate Animals/Biohazards• For live vertebrate animals, describe:

– Proposed use of animals, and species, strains, ages, sex, and numbers

– Justification for the use of animals– Adequacy of veterinary care– Limitation of discomfort, distress, pain, and

injury– Methods of euthanasia

• For biohazards, describe:– Materials/procedures that are hazardous to

research personnel and/or environment and protections to be used

Training in the Responsible Conduct of Research (RCR)

• See NOT-OD-10-019 and NOT-OD-11-059• “K” applicants have 1 page to describe

RCR training in regard to:– Format: must include face-to-face training– Subject Matter: e.g., conflict of interest,

collaborative research, research misconduct– Faculty Participation: should include

mentors and training faculty– Duration of Instruction: > 8 contact hours– Frequency of Instruction: at least once every

career stage/every 4 years

Budget / Resource Sharing Plan

• Budget and Period of Support:– Requested funds must be adequately

explained in Budget Justification– Make sure Period of Support and Indirect

Costs are correct• Resource Sharing Plan:

– Data Sharing Plan– Model Organism Sharing Plan– Genome-Wide Association Studies

(GWAS)

Restructured Applications

• See NOT-OD-10-002 and NOT-OD-11-027• Key components of “K” applications

total < 12 pages and include:– Candidate’s Background– Career Goals & Objectives– Career Development/Training Activities

During Award Period– Research Strategy

Restructured Applications• Other sections with page limits:

– Project Description (Abstract) – 1 page– Specific Aims – 1 page– Training in Responsible Conduct of

Research – 1 page– Statements by Mentor, Co-Mentors,

Contributors – 6 pages– Description of Institutional Environment – 1

page– Commitment to Candidate’s Research

Career Development – 1 page– Biographical Sketches – 4 pages/person

Restructured Applications• Sections without page limits:

– Budget Justification– Protection of Human Subjects– Inclusion of Women and Minorities– Inclusion of Children– Vertebrate Animals– Appendixes

• Sage advice: Avoid the temptation to “Overstuff” your application!– e.g., by including Research Strategy in

Budget Justification or Vertebrate Animals sections

Biographical Sketches

• Biographical Sketches (“Biosketches”) – 4 pages/person:– Include a biosketch for you, your

mentor(s), and all collaborators– Biosketch contains a Personal Statement

which should be relevant to each person’s role on the project

– Recommended (not required!) that it includes full citation for up to 15 of most relevant publications

• Also indicate total number of publications after “Selected Peer-Reviewed Publications”

Appendixes• See NOT-OD-07-018• Allowable appendixes include:

– Up to 3 of the following:• Accepted but not yet published

manuscripts and/or abstracts• Published manuscripts/abstracts if free,

online, publicly available journal link not available

• Patent materials relevant to project– Surveys, questionnaires, data collection

instruments, clinical protocols, informed consent documents

More Recent NIH Policies on Grants• Resubmission applications (A1) must

be received within 37 months of the initial application receipt date– See NOT-OD-10-140

• No more 2-day “error correction window” – all corrections must be complete by due date– See NOT-OD-10-123

• No more 5-day grace period for Letters of Reference – must be in by due date – See NOT-OD-11-036

More Recent NIH Policies on Grants• Additional restrictions on submission

of late grant application materials– See NOT-OD-10-115

• Policy on late submission of grant applications: e.g., up to a 2-week extension for submitting an application under certain conditions– See NOT-OD-11-035