VA Research & Development · 2017-01-31 · VA Research & Development Health Services Research and Development Scientific Merit Review Board CyberSeminar: Reviewer Update and Orientation

VA Research & Development Health Services Research and Development

Scientific Merit Review Board

CyberSeminar:

Reviewer Update and Orientation

February 1, 2017

February 2017

Orientation to the March 2017 Scientific Merit Review Board (SMRB)

• Welcome and Outline of today s CyberSeminar

• Introductions of HSRD SPMs and Staff Transitions

• Broad Overview of HSR&D Priorities

• Poll question

• Peer Review core values / Review Panels

• Reviewer responsibilities

• Preparing Critiques / Using the Rating Scale for Scoring

• Preparing for the Meeting

• Activities during the Meeting, and Follow-up after

• CyberSeminar feedback survey VETERANS HEALTH ADMINISTRATION 1

Available on the HSR&D Website:

http://www.hsrd.research.va.gov/funding/Priorit

yDomains2017.pdf?utm_source=enews&utm_

medium=email&utm_campaign=Jan17

HSR&D Updated Priority Domains

Major Priority Domains

Healthcare Access

Healthcare Equity and Health Disparities

Healthcare Informatics

Aging, Long-Term Care, and Caregiving

Mental and Behavioral Health

Women s Health

Implementation Science

Patient-centered Care, Care Management, and Health Promotion

Health Care Systems

Cross-cutting Methods Domains

1. Non-VA services use data to

better track Veteran care

2. Engagement of Veterans,

providers, and other

stakeholders in research

methods, outcomes, and

implementation

3. Business case and policy

analysis, including systematic

evaluations of programs/policies

pertinent to Veteran and VA care

VETERANS HEALTH ADMINISTRATION 2

http://www.hsrd.research.va.gov/funding/PriorityDomains2017.pdf?utm_source=enews&utm_medium=email&utm_campaign=Jan17





http://www.hsrd.research.va.gov/funding/Priorit

Poll

• Have you ever received funding from HSR&D?

- *Yes

- *No


On which review panel(s) are you serving?

When you finish your participation in this CyberSeminar,

please be sure to indicate your review panel(s) on the

survey. These data, together with the feedback you

provide, will help us to provide the most useful

information for reviewers, this round and in the future.


HSR&D Review panels; SROs

Panel Panel Description SRO Cathie Plouzek, PhD, PMP HSR1 Health Care and Clinical Management

HSR2 Behavioral, Social, and Cultural Determinants of Health and Care Robert Small

Healthcare Informatics Miho Tanaka, PhD HSR3

HSR4 Mental and Behavioral Health Robert W. O'Brien, PhD

HSR5 Health Care System Organization and Delivery Jeffrey P. Haibach, PhD, MPH

HSR6 Post-acute and Long-term Care Cathie Plouzek, PhD, PMP and

George Fitzelle, PhD

Previously John Holden, PhD

CDA Career Development Award Robert Small

NRI Nursing Research Initiative Robert Small

HS1A Learning Healthcare System Initiatives (LHI) Cathie Plouzek, PhD, PMP

Provider Behavior Miho Tanaka, PhD HS3A Data and Measurement Sciences Jeffrey P. Haibach, PhD, MPH HS5A

HS8A Learning Healthcare System Initiative (LHI) Melissa Braganza, MPH

VETERANS H EA L RaTHndomi ADMIzNed ISTPrRAogrTIOam N Evaluation

5

Application Types

• IIR – Investigator Initiated Research

• PPO – Pilot Project Opportunity

• LHI – Learning Health Research Initiative

– Provider Behavior

– Data and Measurement Sciences

– Randomized Program Evaluation

• CDA – Career Development Award

• NRI – Nursing Research Initiative


Applications Reviewed and Funded

VETERANS HEALTH ADMINISTRATION

98

108

132 131 137

20, 20.4%

19, 17.6%

17, 12.9%

23, 17.6%

27, 19.7%

50

39 34

46 45

9, 18%

8, 20.5%

6, 17.7%

8, 17.4%

9, 20%

0

20

40

60

80

100

120

140

160

Summer 2014 Winter 2015 Summer 2015 *includes LHI proposals

Winter 2016 *includes LHI proposals

Summer 2016 *includes LHI proposals

Number of IIR Projects Reviewed

Number of IIR Projects Funded

Number of Pilot Projects Reviewed

Number of Pilot Projects Funded

7

Peer Review Core Values

• Fairness - Standard review and scoring criteria for each application.

• Transparency - Only published criteria used for evaluation.

• Expert Assessment - Scientific expertise to evaluate potential impact.

• Impartiality - COI, bias and predisposition must be managed for all participants in process (SRO, reviewers, applicants, observers), to avoid inappropriate influence.

• Highest Ethical Standards - Confidentiality of all discussions, application materials, other aspects. - Potential misconduct (very rare) is assessed by HSR&D Central Office.


Invitation to eRA Commons Internet Assisted Review (IAR) website

• Indicate concurrence with confidentiality agreement.

• Check all applications for conflict of interest (COI).

• Check your assigned applications for appropriateness of

review assignment.

• If new COI or assignment questions arise, discuss with your

Scientific Review Officer (SRO) as soon as possible.


Key Criteria Considered in Scoring

• Significance

• Approach

• Impact and Innovation

• Investigator Qualifications, Facilities and Resources

Similar criteria categories are used for Pilots and LHI applications.


Consider the Significance and Expected Impact

Focus on the significance and overall impact of the specific project (if executed successfully), not that of the field or the condition being investigated.

• Significance refers to scientific importance or value of the project, and its value to Veterans health care and health outcomes. While science is incremental, is the advancement in science large enough to be meaningful and generalizable?

• Impact and innovation refers to the ability of the project to yield results that can be expected to change practice, influence scientific methods, disrupt the previous order of things, or introduce novel ideas or methods.


11

Scoring Scale 2017

• 1.0 – 1.5: OUTSTANDING- Exceptionally strong with negligible weaknesses; ready for execution "as is."

• 1.6 – 1.9: EXCELLENT- Strong but with weaknesses that should be addressed prior to execution. Re-review not necessarily required.

• 2.0 – 2.3: VERY GOOD - Strong but with weaknesses that should be addressed in a resubmission.

• 2.4 – 2.8: GOOD - Some strengths, but also key weaknesses that require re-working.

• 2.9 – 3.4: FAIR - Major weakness that requires substantial revision before resubmission.

• 3.5 – 5.0: POOR- Major weaknesses that discourage resubmission.


Other Criteria and Summaries

• If Multiple Principal Investigators , need MPI Leadership Plan (see MPI eligibility policy in !dditional Guidelines document in Meeting Materials)/

• Response to prior review.

– Note: Each round of review is independent.

• Protection of human research participants.

• Inclusion of women and minorities.

• Overall Impression: Summarize your evaluation in a paragraph that includes the key factors that determine your overall priority score.

• Key Strengths and Weaknesses: List/describe the key strengths and weaknesses of the application, with brief supporting statement for each.


Support Letters

• HSR&D encourages a broad range of research relevant to VHA.

– Partnered research in collaboration with VHA healthcare system executives, clinicians, and Veterans.

– Highly innovative research that is responsive to VHA scientific priorities, but is not yet considered a high priority by a partner.

• Letters of support may be appropriate when research is connected to a healthcare system partner s goals/


Unscored Criteria

Criteria that should not be factored into the score of scientific merit.

Budget appropriate to scope of work.

Data Management and Access Plan (DMAP)

• Reviewers comment if Data Sharing Plan is reasonable.


Preparing Your Written Review

• Updated critique templates for IIRs, Pilots, and LHI.

– Hyperlinks to criteria descriptions are in template appendix.

• One overall score (no individual criterion scores).

• Substantive narratives expected for major review criteria. – As appropriate, identify your comments as Major or Minor.

• Evidence-based: critiques supported by evaluation of the application and understanding of the state of the science.

• Each round of merit review is independent; it is fair to raise new questions about a revised application.


Critique Comments

• Include full statements/descriptions of Key Strengths, Key Weaknesses in the last sections of your written critique.

• Evaluative (not descriptive) with supporting statements.

• Clear, concise narrative.

• Focus on specific strengths and weaknesses; avoid direction.

• Summarize your evaluation in a paragraph that includes the key factors that determine your overall priority score.

• Use abbreviations and acronyms sparingly

• Express criticism constructively as it will become part of the Summary Statement used by HSR&D and the applicant.


Preparing for Review Meeting

• Check for conflicts, read assigned application(s), and prepare written critique & prelim. score for each assigned proposal.

• Submit reviews on the IAR website (by February 28, 11:59pm).

• During I!R Read Phase (starting March 1), read the critiques submitted by other reviewers, in order to facilitate discussion.

• Prepare assigned reviewer presentations to be given at meeting.

• Be familiar with unassigned applications. • Participate in review discussions and scoring at meeting;

upload final scores in IAR by last day of meeting. • After meeting, revise written critiques as needed, and upload by end of the I!R Edit Phase (11.59pm ET, March 13).


Preparation for Review Discussion

• Prepare assigned reviewer presentation, focusing on the factors/criteria driving your score; critique should not be read verbatim.

• Primary reviewer prepares brief description of the proposal (not included in the written critique) to orient the panel.

• Include the major strengths and weaknesses for each of the key criteria

• Additional reviewers (secondary, tertiary, other) add only new comments based on their own expertise, and/or highlight key points of agreement or disagreement.

• Assigned reviewer presentation should reflect any change in evaluation that may have occurred based on review of other critiques.


Activities During the Meeting

• Introductions of reviewers (including telephone reviewers, if available).

• Opening remarks by SRO and Chair.

• Consider candidate applications for Streamlining. Any non-conflicted panel member may request application be removed from the list and discussed.

• Discussion of applications (~20 minutes per IIR application).


Streamlining

• Streamlining (not discussing) – Meeting management tool to increase efficiency, focus time on more meritorious proposals.

– Written critiques are sent to all applicant, but streamlined applications lack the Key Points from those discussed.

• Criteria for Streamlining – All initial scores in the range of 3.0 to 5.0 and/or

– Applications that rank in the lower quartile of all the applications on the panel

based on the preliminary scores.

• Panel Process: – Applications that are streamline candidates will be presented one at a time by the

SRO at the beginning of the meeting.

– Any non-conflicted panel member may indicate he/she wants application discussed by simply raising hand (or saying yes if on the phone)/

– Conflicts may be present, so no other comment or discussion is appropriate. Discussion would occur later in the meeting.


Discussion and Scoring

• Goal of discussion is to identify the overall impact & major strengths/weaknesses of an application, so that each member of the panel can assign an appropriate score to the application.

• Consensus is not the overall goal; discussion can include dissent and a range of opinions.

• Each panel member assigns a final score based on her or his independent understanding of the strengths and weaknesses of the application as expressed in the discussion and the application.


Review Order

• Application is introduced; reviewers in conflict leave.

• Assigned reviewers identified, asked to announce their preliminary scores (may differ from those entered in IAR).

• Presentations by the primary, secondary and tertiary reviewers (plus others, if additional assignments).

• All reviewers discuss the application and the critiques.

• The Chair summarizes major strengths, weaknesses.

• Key summary points are reviewed (IIRs only).


Review Order (Continued)

• After key points are finalized: Primary reviewer may make motion to recommend disapproval (due to research being unethical -- very rare); panel members would then vote.

• The Chair calls for assigned reviewers to provide their final scores, reminding them to use guidelines.

• The Chair asks if anyone is going to score outside the range; if so, panel member must state this, with a brief reason.

• The scores are recorded (on paper and in IAR Final Scoring).

• Budget considerations are not a factor in scientific merit.


Scoring Out of Range

• Range is set by the final scores of assigned reviewers.

• If unassigned reviewers score outside the range , they must declare their scoring;

• The exact score does not need to be stated, but the direction outside the range and a reason must be indicated.

• SRO will note names of panel members who score outside range.

• Additional review comments can be sent to the SRO, especially if the score is significantly outside the range.

• Difference could be based on:

• a scientific difference of opinion

• different weighting of the review criteria

• a perceived mismatch between the comments/score provided by the assigned reviewers.

• Dissent should be transparent and is NOT discouraged. VETERANS HEALTH ADMINISTRATION

Scoring Factors

• Score the application as submitted, not what it could be.

• Very important to follow HSR&D Scoring Guidelines, found in the Reviewer guidelines in the IAR Meeting Materials folder – Chairs will summarize strengths & weaknesses from discussion.

– Scores should be consistent with review comments (strengths, weaknesses) and with the scoring guidelines.

• Overall scores are generated by the entire panel.

– Reviewers score all proposals for which they are present for the discussion. Ideal is uniform reviewer cohort scoring all proposals.

– Phone reviewers participate in as much of meeting as possible.


Meeting Follow-up

• Reviewers are required to revise their critiques, as needed, so that they are consistent with their final scores.

• Reviewers are responsible for editing and uploading their final critiques to IAR (by end of Edit phase).

• Reviewers are asked to revise the critiques of not discussed applications. This is especially important if the preliminary evaluation was more enthusiastic than other reviewers and would lead to unclear feedback.

• Program staff in HSR&D will perform additional programmatic and administrative review of applications that are candidates for funding.


Reviewer Essentials

• Bring your own laptop to the meeting and/or have computer access if teleconferencing.

• Wireless internet access is available in the meeting room.

• Adobeconnect will be used to share information throughout the meeting.

• Download essential items (your written critiques) prior to the meeting just in case there are any issues with the wireless access being interrupted.

• If you are traveling to the meeting, hotel reservations deadline is February 12, 2017.

• Pre-COIs are required for access to eRA Commons (Ethics also required for non-VA).

• eRA Commons links to RFAs work only with VA access because they are attempting to access an intranet source. Use the pdf of RFAs in the meeting materials.


Important Dates

• February 28, 2017 11:59 pm ET

• March 1, 2017 begins

• March 7, 2017

• March 7-8, 2017

• March 8, 2017

• March 8-9, 2017

• March 9, 2017

• March 10, 2017

• 3 hours after each meeting ends

• March 13, 2017 11:59 pm ET

Preliminary Critiques due in eRA

Read period - ends before meeting

HSR2

HSR1/HS1A, HSR3/HS3A, HSR4/HS4A

HS8A

HSR5/HS5A

HSR6, CDA

NRI

Final Scores due in IAR

Final Edits Critiques due


Insider s Guide to Peer ReviewAdapted from NIH/CSR

• Get started right away to discover if you have a conflict with an application.

– It is cumbersome to reassign the application late in the game. You want to make sure every application gets due diligence.

• Remember you’re not expected to know everything.

– Peer review is a group process with multiple individuals providing special expertise and different points of view.

• Be clear.

– If you feel there is an immutable and fatal flaw, say so in your review. Otherwise, the investigator may needlessly make changes that do not address the issue.


Insider s Guide to Peer ReviewAdapted from NIH/CSR

• Recognize that applicants can’t provide all the details/

– 0 in the shorter format / page limits/ �onsider if the investigators have sufficient command of the material to make it likely the project will succeed.

• Don’t try to rewrite an application.

– You are a critic, not a mentor. Your comments should be generous when possible and polite at all times.

• Talk about what matters.

– Review meeting time is valuable, so it is vital that the oral presentations focus on issues that help people arrive at a score. Focus on major strengths and weaknesses. Limit the description of the research- dont try to enumerate every minor flaw/


Top 10 Things Reviewers Should Not Say in a Review !dapted from NIH

• “This application has 2 great aims and 1 bad one. I would recommend deleting !im 3, and I can give it a good score.”

– The application needs to be evaluated and scored as a whole, as it is submitted.

• “This application is not in my area of expertise . . . “

– If youre assigned an application you feel uncomfortable reviewing, you should tell your Scientific Review Officer (SRO) as soon as possible before the meeting.

• “This application was scored a 245 the last time it was reviewed . . . .”

– Do not mention the previous score an application received, because this could skew the review discussion. Focus on the strengths and weaknesses of the current application, as well as the responses to previous critiques, and score based on the current application and scoring guidelines.


Top 10 Things Reviewers Should Not Say in a Review !dapted from NIH

• “This I21 application does not have pilot data, which should be provided to ensure the success of the project.”

– I21s are pilot projects; preliminary data are not required.

• “I like this project but I’m giving it a poorer score because the applicant has too much money.”

– You should focus on the application s scientific and technical merit/ (However, you can note an excessive budget request in the budget section, though this should not affect your overall score.)

• “I didn’t read the application, but I scanned it and saw the applicant said XXX. He doesn’t know what he’s doing.”

– It s better for you to ask other reviewers who have read the application carefully what they think about XXX.


�efore Starting the Q and ! 0

For more information:

• See the guidelines posted under Meeting Materials on the IAR website for your meeting.

• Ask the SRO for your review meeting.

• Watch the video with advice from Dan Berlowitz, the Chair of the overall HSR&D review committee (http://www.hsrd.research.va.gov/for_researchers/merit _review/default.cfm).

• Send general questions to [email protected].

Please complete the CyberSeminar feedback survey that will be presented when you leave the session.


http://www.hsrd.research.va.gov/for_researchers/merit_review/default.cfm



mailto:[email protected]

Thank you! Next on to Questions and Answers

35

Appendix

The following slides are included simply as a reference.


Pilot projects

• Goals of a pilot (and next steps) should be clear; for example:

– Strengthen the evidence for larger study

– Preliminary data for full-scale proposal (e.g., effect size)

– Establish feasibility of objectives/methods (e.g., recruitment)

– Explore novel research approach through small scale study

• Methods should align with goals, be appropriate for pilot work

– e.g., sample size/population will depend on whether goal is to establish effect size vs. feasibility

• Preliminary data are not required.

• Scale is smaller than an IIR; scientific rigor still important

– Feasible to complete in one year, max budget of $100,000 VETERANS HEALTH ADMINISTRATION 37

RFAs

• IIR – Investigator Initiated Research – Maximum 4 years and $1.1M

• PPO – Pilot Project Opportunity – Maximum 1 year and $100,000


RFAs

• LHI-Learning Health Research Initiative – Provider Behavior

• Maximum 4 years and $1.1M

– Data and Measurement Sciences • Maximum 4 years and $1.1M

– Randomized Program Evaluation • Maximum 3 years and $750,000 ($250K/year)


RFAs

• CDA- Career Development Award – Maximum 5 years

– Clinicians: full salary and fringe benefits to support a 6/8ths appointment

– Non-clinicians: minimum 5/8ths appointment up to 8/8ths

– Supplementary project funds first three years of the award capped at $40,000/year for awardees at HSR&D Centers of Innovation (COINs), and $50,000/year for all other CDAs, and are subject to availability.


RFAs

• NRI – Nursing Research Initiative – NRI IIR

• Maximum 4 years and $1.1M

– NRI Pilots • Maximum 2 years and $100,000


Documents

VA Research & Development · 2017-01-31 · VA Research & Development Health Services Research and Development Scientific Merit Review Board CyberSeminar: Reviewer Update and Orientation