Peer review

  • View
    189

  • Download
    1

Embed Size (px)

DESCRIPTION

This is a copy of a talk i gave at the Vision Sciences Society meeting in 2012. The talk presentas a review of the history and evolution of peer review and scientific publishing, an evaluation of its current function based on survey data, and a proposal for improving the system.

Transcript

  • 1. The state of scientific publishing and a new modelDwight Kravitz Chris Baker

2. Motivation: Improving Cost and QualityNIH Budget / # Papers Budget per Paper ($1000)Published 3. Peer Review: Outline History and evolution Modern purpose Improving peer review Post-publication 4. History of Peer Review andPublishing Problems facingauthors Disseminating results Claiming ownership ofresults Publishing Publishers haddistribution channels Provides a record ofwhen results were 5. History of Peer Review and Publishing Problems facingpublishers Establishing veracity ofresults Deciding which results topublish given limited Peer reviewspace Stamp of scientificapproval Prioritizes papers forHenry Oldenburg, 1665inclusion by publishersMedical Essays and Observations, 1731 6. 55% Reject42% Revise3% Accept 7. 55% Reject42% Revise3% Accept 8. Given Revision: 98% Accept2% Reject 9. 64% of papers are rejected at least once 10. Direct Costs of the Current System Total Number of Reviews:6.3 (2-15) Total Days under Review:122 (21-321) Total Hours for Revisions:68 (5-300) Total Days to Publish:221 (21-533) Publication Fees: $2000 Thousands of dollars and hundreds of hours perpaper Long delays impair every form of assessment Variance hinders planning research programs andcareers 11. Peer Review: Outline History and evolution Invention Current form Modern purpose Improving peer review Post-publication 12. Original Purpose Reasons for publishing Disseminating results Claiming ownership ofresults Reasons for peer review Establishing veracity ofresults Deciding which results topublish given limitedspace Henry Oldenburg, 1665 Medical Essays and Observations, 1731 13. Modern Purpose Reasons for publishing Disseminating results Claiming ownership ofresults Reasons for peer review Establishing veracity ofresults Abstract:9495 Deciding which results to Full Text: 8564publish given limited PDF: 2549space 14. Pre-reception Peer review allows authors to test thepaper before sending it to the wholefield Catches fundamental errors Highlights important overlooked issues 15. Problems with Pre-reception Reviews are impure measures of quality 16. Problems with Pre-reception Reviews are impure measures of quality Review process is opaque to the wider field 17. Prioritization of the Literature Loop ranks papers based on which journalpublishes them Provides a way of deciding which papers to read Compensates for long publication lags 18. Problems with Prioritization Reviews not focused solely on quality Inefficient (avg. lag is 221 days) Creates detrimental short-term incentives 19. Problems with Prioritization Budget per Paper ($1000)http://pmretract.heroku.com 20. Problems with Prioritization Budget per Paper ($1000) Retractions per 100k Papershttp://pmretract.heroku.com 21. Problems with PrioritizationFang &Casadevall, 2011 22. Problems with Prioritization Reviews not focused on solely on quality Inefficient (avg. lag is 221 days) Creates detrimental short-term incentives Doesnt work 23. Problems with Prioritization 24. Problems with Prioritization 25. Problems with Prioritization 26. Problems with Prioritization 27. Peer Review: Outline History and evolution Invention Current form Modern purpose Pre-reception Prioritization Improving peer review Post-publication 28. Improving the current system Make publishing more efficient and lessvariable Days to Publish:221 (21- 533) Years to First Cite: ~1.5 + research Capture reviews in the prioritization Total Number of Reviews: 6.3 Total Hours of Reviewing: ~31.5Decouple peer review rather than Focus reviews on science from publishing 29. Proposed System 30. Decoupling peer review from publishing Allows reviewers to focus on scientific meritrather than publication Allows reviews to be captured for prioritizationand rewards for reviewers Makes publishing simpler and more efficient Eliminates the short-term incentive to produceleast publishable units 31. Peer Review: Outline History and evolution Invention Current form Modern purpose Pre-reception Prioritization Improving peer review Decoupling from publishing Post-publication 32. Format for papers 33. The need for post-publicationreview Proportion of field Why Shru UsefulField ? gDatapoint Altering 34. The need for post-publicationreview We need more reviews to Proportion of field approximate these complex distributions. Also enables personalized prioritization of the literature. Why Shru UsefulField ? gDatapoint Altering 35. Incentivizing post-publicationreview 36. Organizing the literature The utility ofmodern peerreview andpublishing isprioritizing andorganizing theliterature. 37. Organizing the literature The utility ofmodern peerreview andpublishing isprioritizing andorganizing theliterature. 38. Peer Review: Outline History and evolution Invention Current form Modern purpose Pre-reception Prioritization Improving peer review Decoupling from publishing Post-publication review