Peer Review Workshop: Three Part Series Awardee ... · •External peer reviewers provide comments •Associate editor sends you a synthesis letter of peer review comments. •Respond

Peer Review Workshop: Three Part SeriesAwardee Preparation for Peer Review

Session 3PCORI Annual Meeting

November 2, 2018 Welcome#PCORI2018

https://twitter.com/hashtag/PCORI2018?src=hash

2

Reminder

This session will be audiotaped and archived for future use. Please make comments only into a microphone.

3

Welcome

• Agenda• Why PCORI peer reviews final study reports• Preparation of the draft final research report• What to expect from the peer review process• What happens after peer review• Q&A

4

Purpose of PCORI’s peer review

• Meet our legislative requirements• Scientific integrity• Adherence to PCORI methodology standards

• Establish an archive of PCORI-funded research• Add value through complete reporting of all research projects,

even unsuccessful ones• Provide complete, openly accessible reports that can be useful to

systematic reviewers

5

Beneficiaries of peer review

• Researchers planning studies learn from detailed reports• Systematic reviewers have complete, unbiased reports of

methods and outcomes• Study authors have opportunity to record the full account of the

lessons learned from their study• Patients and others stakeholders gain information from the

summaries produced from the final reports that can help them make decisions

6

Preparing the Draft Final Research Report (DFRR)

• Complete account of the PCORI-funded study• All study aims & outcomes, including any developmental work

• Any changes made to the original protocol• Audience is general scientist reader, not PCORI staff

7

Sections of DFRR submission

Draft Final Research

Report

Structured abstract

Coverage of all study aims

Description of patient & stakeholder engagement

Detailed methods and

resultsStudy

limitations Subpopulation considerations

Checklist of adherence to PCORI’s Methodology

Standards

Copy of study protocol

8

Follow the DFRR Instructions!

8

9

DFRR structure

• Abstract (up to 750 words)• Background, ending with study aims• Patient & Stakeholder engagement (may be N/A for Methods

projects)• Methods & Results (organize by aim, follow CONSORT 2010 or

other similar guidelines, as appropriate)• Conclusion (high-level overview of main findings related to the

validity and implications of the study)

10

Multiple aims, qualitative or pilot

• Consider organizing the methods and results by aim, as in:• Aim 1: Methods, Results• Aim 2: Methods, Results

• Background and Discussion sections should bring together all of the aims to provide a single coherent picture of the work you undertook

• Do not leave out aims that are developmental• If one aim involves only engagement, you can present the methods and

results in the Engagement section

11

When you have already published

• Previously published material may be included, verbatim, in the DFRR

• Incorporate the material into the structure of the DFRR• Be sure to get permission from the publisher for the specific parts

you are using• Provide appropriate citations for each copied section

12

Common mistakes leading to revision• Description of analyses

• Primary vs. secondary outcomes; consideration of missing data; report of minimally clinically important difference

• Focus on secondary outcomes when primary outcomes are not significant• Secondary outcomes should still be considered exploratory

• Over-interpretation of results• Overly-positive interpretation of findings or reporting results without appropriate

caveats or limitations

• Not following DFRR format• There is some flexibility in the DFRR structure so that the DFRR can tell the full story

of the project.

13

Hear from PIs

• How long did you spend preparing the DFRR? • What was most challenging about the preparation?• What was most helpful in your preparation?• What would you tell someone else preparing their DFRR?

14

Peer Review TimelinePre-

Submission(OHSU)

• Begin writing your DFRR

• Verify and submit your abstract to Editorial Manager®

• Submit DFRR to Editorial Manager® for Peer Review

Internal Pre- Review

(PCORI)• Pre-Review editor

reviews for clarity• Program Officer

reviews for completeness

• You may need to revise before peer review

External Peer Review

(OHSU)• External peer

reviewers provide comments

• Associate editor sends you a synthesis letter of peer review comments.

• Respond to comments within 2 months

• Associate editor may request additional changes until they determine that the DFRR has met Peer Review expectations

Final Review(PCORI)

• Dr. Hal Sox, MD, PCORI Director of Peer Review and Scientific Publications provides final review for clarity

• This step may repeat until Dr. Sox approves your report on behalf of PCORI

Results Disseminated

• Lay and professional summaries (90 days)

• Peer review summary (90 days)

• FRR & study protocol (within 12 months)

15

Management of Peer Review Stages• Oregon Health & Science University (OHSU) contracted by

PCORI in 2016 to implement the scientific external peer review of PCORI-funded research and manage the peer review software system

• OHSU supports PCORI’s efforts to ensure DFRRs are suitable for review by external reviewers

• PCORI Editorial Office Address: [email protected]

• Final approval of DFRR determined by PCORI Director of Peer Review and Scientific Publications

mailto:[email protected]

16

Editorial Manager ®

Peer Review Management Software

• Only one person on the awardee team can interact with Editorial Manager for any given DFRR

• Contact [email protected] to have someone other than the PI designated as the corresponding author

• PI can still receive correspondence • Corresponding author will receive email with username and

directions for setting password


17

Editorial Manager ®

Peer Review Management Software

18

Pre-Submission Tasks• 3 months prior to the DFRR expected due date, the PCORI Peer

Review Team begins the reviewer recruitment process • To make the process as efficient as possible we request

information that helps us determine the number and types of reviewers and when they will be neededConfirm the date you are planning to submit the draft final research

reportList any articles you published or plan to publish, along with an indication

of the publication statusIndicate if any of the material included in DFRR has been or will have been

peer-reviewed at the time of the DFRR submission

19

Pre-Submission Tasks

An updated and accurate Abstract and Key Personnel List are critical for identifying and selecting individual reviewers with the right expertise and without any conflicts of interest

Include results and conclusions

whenever possible

Ensure Methods are clear and

complete

Change language from future tense

20

Pre-review

Developed to improve readability and ensure completeness of DFRR before it is sent to external peer reviewers

• Pre-Review Editor considers whether DFRR is adequately clear for external reviewers to assess the research for scientific integrity and adherence to standards

• Program Officer reviews content for completeness of aims, outcomes, and protocol changes

21

Pre-review

• Editor decision to ask for revisions or approve for external review takes 2 weeks

• Requested revisions are due from PI within 2 weeks

• PI will be notified when DFRR approved for external peer review

92% of DFRRs are approved for external peer review with original submission or one round of revisions

22

Hear from PIs

• What did you experience in pre-review?• What were the notes from the editor like?• Pros and cons of pre-review, what to expect

23

External Peer Review Process

Purpose of external

peer review is to assess:

Conduct of the study, presentation of results, and interpretation of findings

Adherence to PCORI’s Methodological Standards

Relevance and usefulness of the research to patients and stakeholders


(OHSU)

24

External Peer Review Process

88% of DFRRs are also reviewed by

patient/caregiver/patient advocate or stakeholder with personal familiarity

with the condition and/or health system

being studied

• DFRR is reviewed by the assigned Associate Editor and reviewers with expertise in the subject matter and methodology/statistics

• Reviewers have 14 days to submit a completed review


(OHSU)

25

What to Expect After Peer Review Closed

4-5 completed reviews are submitted at the end of the 2 week review periodAssociate Editors responsible for synthesizing the main

concerns identified in the DFRR based on their independent review of the DFRR and the key criticisms of other reviewers Completed 2-3 weeks after all reviews submitted

Unadulterated comments from each reviewer appended to synthesisSynthesis and comments may be 20+ pages long PI has 45 working days to submit revised DFRR (tracked-

changes and clean versions) and response letter

-42% of DFRRs meet requirements of peer review after 1 round of revisions

-Another 46% take 2 rounds of revisions


(OHSU)

26

Format of Response Letter

• Address all comments in body of letter, appended reviews, and attachments

• Restate each comment grouping comments by AE Synthesis, AE Review, Reviewer #1, Reviewer #2, et cetera

• Do not include laudatory or descriptive comments that do not require an action or explanation

• You may group similar comments from multiple reviewers identifying the source of each comment

• Or you may keep similar comments separated by reviewer and refer back to an earlier response


(OHSU)

27

Tips for Addressing Peer Review Comments

Give scientific rationale for your response

If you made a revision in response to the comment, give the page and section of the report where the

change was made

Address criticisms that cannot be resolved in the

limitations section

After making all revisions, read through the DFRR

again to ensure flow is not broken


(OHSU)

28

Hear from PIs

• What comments did you receive from reviewers?• Did the AE synthesis help you with your revisions?• What was the hardest thing about responding?• How did the comments change your report and your thinking

about your study?

29


Submission(OHSU)















Final Review(PCORI)







30

Final Review by PCORI Program Director of Peer Review and Scientific Publications

• Once the DFRR passes external peer review, it comes back to PCORI for final acceptance

• The DFRR is reviewed and areas that could be clearer are noted

• PI receives letter with main points and marked-up copy of the DFRR

• Once revisions are made, the program director accepts the report as Final on PCORI's behalf

• Preparation of final research report for posting of results begins

31

Conversation with PIs

• Purpose of the final review• Peer review overall – frustrations, expectations

32


Submission(OHSU)















Final Review(PCORI)







33

After peer review – What happens next

Materials posted on website• Lay summary and professional abstract• Peer review summary of reviewer comments & researcher

responses• Final report • Study protocol• COI disclosure form• DOI & PMC

34

Lay summary and Professional abstract

• PCORI translation center produces these based on the FRR.• Cognitive interviews• Reviews by PI and PO

• Not considered prior publication• 500 words, based on International Committee of Medical Journal

Editors’ guidelines• Posted within 90 days of FRR acceptance

• Legal requirement• COI/disclosure also posted at this time

35

Peer Review Summary

• Produced by PCORI staff, based on the FRR, reviewer comments & responses

• Focuses on major scientific issues brought up by reviewers and how researchers addressed them

• If PI does not agree with reviewers’ comments, they can add a response to the summary

• Template text describing peer review process, 2-5 specific bullet points

36

Posting the final research report

• FRR will be copyedited. You can opt to see the changes before they are finalized.

• We will contact you every 3 months to check in about your publications and whether you approve the posting.

• You will receive a DOI number for your report (linked to the project page at pcori.org) and instructions for posting the report to PubMed Central.

• Study protocol also posted, without copyediting.

https://www.pcori.org/

37

FRR vs. journal articles: which should come first?

• Investigators are free to include sections of published articles in the FRR with appropriate permissions and citations

• Most journals will not accept an article submission if the results have already been posted elsewhere (thus the 12 month window)

• Exceptions include BMC family of journals, Comparative Clinical Trials• What you should do to prepare

• What our PIs (Mike & Hanan) did

38

39

40

41

42

questions?

Thank you!

Ilya Ivlev, MD, PhDKelly Vander Ley, PhDMarina Broitman, PhDRebekah Webb, MPAWhitney Brower, MPH

Special thanks toAmy Price, PhDBarbara Sheehan, MEdHanan Aboumatar, MD, MPHJeffrey Oliver, MBAPaul McClean

43

Learn More

• Website: https://www.pcori.org/research-results/peer-review-our-studies

• Editorial Office: [email protected]

• Peer Review Office @ PCORI: [email protected]

www.pcori.org

[email protected]

#PCORI2018

https://www.pcori.org/research-results/peer-review-our-studies



http://www.pcori.org/


https://twitter.com/hashtag/PCORI2018?src=hash

Documents

Peer Review Workshop: Three Part Series Awardee ... · •External peer reviewers provide comments •Associate editor sends you a synthesis letter of peer review comments. •Respond