Peer Review

  • View

  • Download

Embed Size (px)


Peer Review, PUBLIC Understanding of Science



    Discussion paper from a Working Party on equippingthe public with an understanding of peer review

    SSeennssee AAbboouutt SScciieennccee60 Cambridge StreetLondon SW1V 4QQ

    Registered Charity No.1101114

    ISBN 0-9547974-0-X

    COVER 6/14/04 12:23 Page 1

  • Published by Sense About Science60 Cambridge StreetLondon SW1V 4QQTel: +44 (0)1795 591975

    Email: enquiries@senseaboutscience.orgWeb:

    Registered Charity No. 1101114

    ISBN 0-9547974-0-XMay 2004

    2004 Sense About ScienceReproduction of short parts of this text for the purposesof research and comment is permitted. No part of thepublication may be reproduced for any other purpose,nor stored in a retrieval system, nor transmitted in anyform or by any means, without the prior permission ofthe copyright owners.

    Online versionThe full text of this discussion paper is available todownload as a pdf document, free of charge, from

    Publication ordersPrinted copies of this report are available for 10 to cover production and postage (all countries). A limitednumber of copies are available at no charge. To order,and to request review copies, please contact

    A leaflet version of Section 2, A short guide to peer review, will be available for distribution in October 2004. Please contact Ellen Raphael at

    Printing and designPrinting costs kindly paid by the Medical ResearchCouncil as a contribution on behalf of all of the Research Councils. Printed in the UK by Short Run Press Ltd, Exeter. Cover design kindly contributed by Andrew Giaquinto,Institute of Physics Publishing.

    PAGE 2&3 6/14/04 11:57 Page 1


    Discussion paper from a Working Party on equipping the public with an understanding of peer review

    November 2002 May 2004

    Compiled and presented by Tracey Brown, DirectorSense About Science

    PAGE 2&3 6/14/04 11:57 Page 2

  • P e e r r e v i e w a n d t h e a c c e p t a n c e o f n e w s c i e n t i f i c i d e a s




    Foreword & Acknowledgements vii

    The Working Party viii

    Executive Summary ix

    Section 1: A cultural challenge for peer review? 1 Debates about scientific research 1 Science in a challenging environment 2 Can peer review help society navigate the constant stream of science news? 4 Why should the public be interested in peer review? 5

    Section 2: A guide to peer review and scientific publishing 7 What is scientific peer review? 7

    Why is peer review used? 7 The relationship between the referees and the journals 9 What happens to a paper once it has been reviewed? 9 Rejection 11 The problem of time delay 11 Referees motivation 13

    Do scientists use peer review to pursue their own personal agendas? 13

    Competition over publication, grants and commercial interests Abuse of privileged information Personal prejudices

    Do journal editors give priority to particular kinds of results or newsworthy stories? 14 Negative results

    Seeking to make news

    Should referees be anonymous? 15 Are some scientists results suppressed through rejection of their papers? 16

    Is maverick science rejected through peer review? 16

    Does peer review reinforce scientific resistance to change? 17

    Why cant there just be a technical checking system, rather than peer review,

    to make sure the researchers have abided by good practice? 18

    Should research be judged on the basis of who funds it? 18

    The limits: when is peer review not really peer review? 19

  • P e e r r e v i e w a n d t h e a c c e p t a n c e o f n e w s c i e n t i f i c i d e a s


    But is it true? Error and correction 19 Fraud and misconduct 20 Some challenges for scientific publishing 21

    The volume of research papers Electronic publishing and Open Access

    Asking questions about research claims 22

    Section 3: Raising the public profile of peer review 23 Introduction 23 Public promotion of research findings and the peer-review process 25

    Science information supply side

    Talks at conferences

    Following media reports

    Commercial publishing of research outside of the peer-reviewed literature 28 The legal situation

    Other commercial issues


    Peer review as part of science education 32 Equipping a wider public with an understanding of peer review 35

    The social implications of research claims

    Clarifiying the contribution of peer review to scientific publishing

    Concluding comments 37

    Appendices Appendix 1: References and sources 39 Appendix 2: Conflict of interest form 42 Appendix 3: Guidance on price sensitive information 44 Appendix 4: Other projects and sources of information about peer review 45 Appendix 5: Working Party members 46 Appendix 6: Sense About Science 49

    List of illustrations

    Box 1: Referees usually comment and make recommendations on some of the following 8 Box 2: Typical examples of referees comments 10

    Box 3: The scientific publishing scene 12

    Box 4: PPL Therapeutics and British Biotech 28

    Box 5: Scientific literacy 33

    Box 6: Flow chart indicating the typical process of peer review 34

  • P e e r r e v i e w a n d t h e a c c e p t a n c e o f n e w s c i e n t i f i c i d e a s


  • P e e r r e v i e w a n d t h e a c c e p t a n c e o f n e w s c i e n t i f i c i d e a s



    Few outside the scientific world can be expected to know about the immense effort, and even pain, that is experienced before research is ready to be published as a peer-reviewed paper in a scientific journal. The peer-review process is almost as complicated and unpredictable as the birth of a baby. Yet, as with delivery, it involves analysis, judgement and evaluation or at least it should.

    We believe it is important for the public to understand more about the critical engagement and the checks

    and balances that help to determine the plausibility of new knowledge (Section 1). This report therefore

    seeks to provide an outline of the complex way that science enters the public arena (Section 2) and why

    the time-honoured peer-review process is crucial, though it could be improved and better understood

    (Section 3).

    This report also endeavours to show how things can go wrong. Delivery of results may be premature and

    a claim can reach the airwaves before adequate testing has been performed. A consequence is that the

    unsuspecting are presented with untested opinions rather than peer-reviewed conclusions.

    Donald Kennedy in a recent editorial in Science1 pointed out that peer review of a scientific paper involves the addition of qualifications and limitations on conclusions. Other forms of communication, by contrast,

    such as intelligence, news reporting and campaigning, often delete qualifying language and caveats so that

    scientific conclusions are strengthened and simplified. This tension between the description of

    experimental findings and interpretation arises for all who aim for clarity and urgency. Yet it is the pursuit

    of truth that remains fundamental to the scientific endeavour.

    This report is for scientists and for all who grapple with the barriers and difficulties arising from new

    knowledge as societies come to terms with the latest scientific and technological news. It is a discussion

    paper, written in a cultural context, that makes a more determined case for peer review from a social

    standpoint than we envisaged at the outset. It is the outcome of a vigorous debate within the Working

    Party and is published with its full support.

    May I take this opportunity to thank all the members of the Working Party for their immensely valuable

    contributions, many other colleagues who also gave most generously of their time and energies, and most

    of all Tracey Brown, who has worked tirelessly over the past 18 months in bringing the report to a

    successful conclusion. She has directed and inspired our wide-ranging deliberations and has brought them

    together in a comprehensive and comprehensible document. We are further indebted to Dr Bridget Ogilvie

    and Professor Onora ONeill for their review of the document. Finally, we are deeply appreciative of the

    help of Dr Irene Hames for her thorough and thoughtful editing over several drafts.

    Professor Sir Brian Heap (Chair) May 2004

    1 Editorial (2004) Science, 303, 1945.

  • P e e r r e v i e w a n d t h e a c c e p t a n c e o f n e w s c i e n t i f i c i d e a s


    The Working Party

    This discussion paper has been endorsed by a sub-group of the Board of Trustees of Sense About Science. It has been prepared by a Working Party on equipping the public with an understanding of peer review. Members2 of the Working Party are:

    Professor Sir Brian Heap CBE FRS (Chair) Dr Derek Bell Professor Colin Blakemore FRS Ms Tracey Brown, Dr Peter Cotgreave Lord Drayson Ms Fion