Prc Summary 4 Ware Final

7/29/2019 Prc Summary 4 Ware Final

1/20

PRC Summary Papers 4

Peer review: benefts, perceptionsand alternatives

Mark Ware

Mark Ware Consulting


2/20

2008 Publishing Research Consortium

Published byPublishing Research Consortiumc/o The Publishers Association29B Montague StreetLondon WC1B 5BW

Printed in Great Britain

The Publishing Research Consortium (P RC) is a group representing publishers andsocieties supporting global research into scholarly communication, with the aim toprovide unbiased data and objective analysis. Our objective is to support work that isscientic and pro-scholarship. Overall, we aim to promote an understanding of the role

of publishing and its impact on research and teaching.

Outputs from work supported by the PRC are available from the website:www.publishingresearch.org.uk.Hard copies are available from The Publishers Association.

The founding partners are The Publishers Association, the Association of Learnedand Professional Society Publishers, and the International Association of Scientic,Technical Medical Publishers. Corresponding partners include The Association ofAmerican University Presses and the Professional/Scholarly Publishing Division of theAssociation of American Publishers.

This summary report was commissioned by the Publishing Research Consortium

(www.publishingresearch.org.uk) from Mark Ware Consulting, a publishing consultancy

(www.markwareconsulting.com). Mark Ware Consulting was responsible for designing

and managing the survey; the online hosting and data analysis was provided by Mike

Monkman Media (http://mikemonkman.com/).


3/20

Contents

4 Executiveoverview

5 Introduction

6 Whatispeerreview?

6 Typesofpeerreview

7 Peerreviewdurations

9 Thereviewersperspective

10 Roleoftheeditor

12 Thebeneftsopeerreview

12 Improvementsinquality

13 Peerreviewasalter

14 Viewsofsurveyrespondents

16 Critiquesopeerreview

17 Viewsofsurveyrespondents

18 Alternativeapproaches

18 Differenttypesofpeerreview

19 Post-publicationreview19 Openpeerreview

20 Reviewingauthorsdata

20 Conclusions

20 Acknowledgements

2008

Publishing Research Consortium


4/20

4

2008


Peerreviewistheprocess ofsubjectinganauthorsscholarlymanuscripttothe

scrutinyofotherswhoareexpertsinthesameeld,priortopublicationinajour-nal.Thissummaryreportpresentssomendingsfromanewinternationalsurvey

ofacademics,setwithinadescriptionofhowpeerreviewoperates,itsbenets,

somecritiquesofpeerreviewandthedevelopmentofalternativeapproaches.

Peerreviewiswidelysupportedbyacademics ,whooverwhelmingly(93%)

disagreedinoursurveythatpeerreviewisunnecessary.Thelargemajority( 85%)

agreedthatpeerreviewgreatlyhelpsscienticcommunicationandbelieved( 83%)

thatwithoutpeerreviewtherewouldbenocontrol.

Peerreviewimprovesthequalityothepublishedpaper. Researchersover-

whelmingly(90%)saidthatthemainareaofeffectivenessofpeerreviewwasin

improvingthequalityofthepublishedpaper,andasimilarpercentagesaidithad

improvedtheirownlastpublishedpaper,includingidentifyingscienticerrorsandmissingandinaccuratereferences.

Thereisadesireorimprovement.Whilethemajority(64%)ofacademics

declaredthemselvessatisedwiththecurrentsystemofpeerreviewusedbyjour-

nals(andjust12%dissatised),theyweredividedonwhetherthecurrentsystem

isthebestthatcanbeachieved.Therewasevidencethatpeerreviewistooslow

(38%weredissatisedwithpeerreviewtimes)andthatreviewersareoverloaded

(seeitem11below).

Double-blindreviewwaspreerred.Althoughthenormalexperienceof

researchersinmosteldswasofsingle-blindreview,whenaskedwhichwastheir

preferredoption,therewasapreferencefordouble-blindreview,with 56%select-

ingthis,followedby25%forsingle-blind,13%foropenand5%forpost-publica-

tionreview.Openpeerreviewwasanactivediscouragementformanyreviewers,

with49%sayingthatdisclosingtheirnametotheauthorwouldmakethemless

likelytoreview.

Double-blindreviewwasseenasthemosteffective .Double-blindreviewhadthe

mostrespondents(71%)whoperceivedittobeeffective,followed(indeclining

order)bysingle-blind(52%),post-publication(37%)andopenpeerreview(27%).

Double-blindreviewacessomeundamentalobjections.Double-blindreview

wasprimarilysupportedbecauseofitsperceivedobjectivityandfairness.Many

respondents,includingsomeofthosesupportingdouble-blindreview,didhow-

everpointoutthatthereweregreatdifcultiesinoperatingitinpracticebecauseitwasfrequentlytooeasytoidentifyauthorsfromtheirreferences,typeofworkor

otherinternalclues.

Post-publicationreviewwasseenasauseulsupplementtoormalpeerreview ,

ratherthanareplacementforit.Interestingly,thiswasdespiteaclearviewthatit

tendstoencourageinstantreactionsanddiscouragethoughtfulreview.

Limitedsupportorpaymentorreviewers.Respondentsweredividedon

whetherreviewersshouldbepaid,with35%infavourofand40%againstpay-

ment.Amajority,however,supportedthepropositionthatpaymentwouldmake

thecostofpublishingtooexpensive( 52%for,18%against)andthelargemajority

ofreviewers(91%)saidthattheyreviewedtoplaytheirpartasamemberofthe

academiccommunity.

Executive Overview

1

2

3

4

5

6

7

8

9


5/20

5

2008


Mixedsupportorreviewoauthorsdata.Amajorityofreviewers(63%)and

editors(68%)saidthatitisdesirableinprincipletoreviewauthorsdata.Perhapssurprisingly,giventhatmanyreviewersreportbeingoverloaded(seebelow),a

majorityofreviewers(albeitasmallone, 51%)saidthattheywouldbepreparedto

reviewauthorsdatathemselves,comparedtoonly 19%whodisagreed.

Some90%oauthorsinthesurveywerealsoreviewers .Theyreportedreview-

inganaverageof8papersinthelast12months.Thelargemajorityofreviews

(79%)wascarriedoutbyacoreofactivereviewers,whocompletedanaverage

of14reviewsperyear,nearlytwicetheoverallgure.Thisgroupreporteditwas

overloadeddoing14reviewsperyearcomparedtotheirpreferredmaximumof 13

suggestingthereisaproblemwithreviewerworkloads.

Introduction

Thisreporttakesalookatpeerreview:whatitis,andhowitworksinpractice;the

benetsofpeerreview;somecritiques;andsomealternativeapproaches.Itislargely

basedonanewinternationalsurveyof 3040academics,lookingattheirbehaviour

andattitudesandperceptionsofpeerreview.Thissummaryreportcontainsonlya

smallfractionofthedataavailableinthefullreport, 1whichinterestedreaderscannd

onthePublishingResearchConsortiumwebsite.

10

11

1Ware, M. and Monkman, M.(2008) Peer review in scholarly

journals: perspective of the scholarly

community an international study.Publishing Research Consortium.Available atwww.publishingresearch.org.uk


6/20

2008


What is peer review?

Peerreview,knownasrefereeinginsomeacademicelds,is(toquotetheun-peer-

reviewedWikipedia)aprocessofsubjectinganauthorsscholarlywork,researchorideastothescrutinyofotherswhoareexpertsinthesameeld.Inthisreportwewill

consideronlythepeerreviewofmanuscriptssubmittedtoacademicjournals(the

othermainuseofpeerreviewisfortheawardofresearchgrants).

Editorialpeerreviewissaidtohavebegunintheearly18thcentury;forexample,

theprefacetotherstvolumeoftheRoyalSocietyofEdinburghsMedical Essays and

Observations,publishedin1731,stated:Memoirssentbycorrespondencearedistrib-

utedaccordingtothesubjectmattertothosememberswhoaremostversedinthese

matters.Thereportoftheiridentityisnotknowntotheauthor.2Duringthe19th

andearly20thcentury,peerreviewdevelopedinafairlydisorganizedwayandmany

prominentjournaleditorsactedmorelikenewspapereditors,withlittleinterestin

formalpeerreview.Peerreviewinthesystematizedandinstitutionalizedformwe

knowtodayhasdevelopedlargelysincetheSecondWorldWar,atleastpartlyasaresponsetothelargeincreaseinscienticresearchinthisperiod.

Injournalspeerreview,theauthorsmanuscriptisusuallysubjectedtosomeinitial

checkstoassessitssuitabilityforreview(forinstance,incompletemanuscriptsor

workthatwaspatentlypseudosciencewouldbedeclinedwithoutreview),afterwhich

asmallnumberofreviewersareselected.Thetaskexpectedofthereviewersvaries

somewhatfromjournaltojournal,butinessenceitisusuallytoassistthejournals

editor(whomakesthenaldecision)ondecidingwhetherornottoacceptthe

manuscriptforpublication.Thereviewerwillcommentonthequalityofthework

done(forinstance,wastheexperimentaldesignappropriatetothequestionbeing

studied?)aswellasonitsoriginality(whatdoesitaddtowhatweknowalready?)and

itsimportance(doesitmatter?).

Typesopeerreview

Therearetwoapproachestopeerreviewincommonuseatpresent.Thenorminmost

academicdisciplines,knownassingle-blindreview,isfortheauthorsidentitytobeknown

tothereviewers,butforthereviewersidentitytobehiddenfromtheauthor.(Thisisthe

methoddescribedabovebytheRoyalSocietyofEdinburghin1731.)Themainargumentfor

blindingthereviewersidentityisthatitallowsthemtocommentfreelywithoutfearof

repercussions.Conversely,single-blindreviewhasbeencriticisedforallowingallkindsof

biasandotherkindsofirresponsibilityonthepartofreviewerstoourishbehindtheveilof

secrecy.(Weshalldiscussthecriticismsofpeerreviewinmoredetailbelow.)

Themainalternativeisknownasdouble-blindreview:inthisapproachtheidentitiesoftheauthorandreviewersarehiddenfromeachother.Becausethereviewerdoesnot

knowtheauthorortheirinstitution,itisargued,theywillfocusonthecontentofthe

manuscriptitself,unaffectedbyconsciousorunconsciousbias.

Anewerapproachtodealingwiththecriticismsofsingle-blindreviewisopenpeer

review:inthismodel,theauthorsandreviewersidentitiesareknowntoeachother,and

thereviewersnamesand(optionally)theirreportsarepublishedalongsidethepaper.

Advocatesofopenreviewseeitasmuchfairerbecause,theyargue,somebodymakingan

importantjudgementontheworkofothersshouldnotdosoinsecret.Itisalsoargued

thatreviewerswillproducebetterworkandavoidoffhand,carelessorrudecomments

whentheiridentityisknown.

Morerecently,electronicpublishingtechnologyhasallowedavariantofopenreview

tobedeveloped,inwhichallreaders,notjustthereviewersselectedbytheeditor,areable

2From Rennie, D. (2003) Editorial

peer review: its development and

rationale. In F. Godlee, T. Jefferson

(eds). Peer Review in Health Sciences.

Second Edition. pp. 1-13. BMJBooks, London.


7/20

2008


toreviewandcommentonthepaperandeventorateitonanumericalscalefollowing

publication.Thispost-publicationreviewcouldoccurwithorwithoutconventional

pre-publicationpeerreview.Thebenetsareseentobethatittakesaccountofcom-

mentsfromawiderrangeofpeople(thewisdomofcrowds)andmakesthereviewa

morelivingprocess.

Inoursurvey,wefoundthattheconventionalsingle-blindpeerreviewsystemwas

theonemostcommonlyexperiencedbyauthors,with85%sayingtheyhadexperienceofitcomparedto45%fordouble-blind,23%foropenandjust8%forpost-publication

peerreview(Figure1).Thisdoesvarybyacademicdiscipline:single-blindreviewwas

thenorminlifesciences,physicalscienceandengineering,whiledouble-blindreview

wasmuchmorecommonforauthorsinhumanitiesandsocialsciences,andclinical

medicalandnursingauthorshadexperienceofbothsystems.

Peerreviewdurations

Thepeerreviewprocessinevitablytakestime.Thesurveylookedatthisfromthe

perspectiveofauthors,reviewersandeditors.

Authorsreportedthatthepeerreviewprocesstookanaverageof 80days,withthe

longesttimesinhumanitiesandsocialsciences.Theywereevenlysplitonwhetheror

notthelengthoftimefromsubmissiontodecisionwassatisfactory,anditwasclear

thattheauthorsexperiencingthelongestdelaysweretheleastsatised.Forreview

timesof30daysorless,abouttwo-thirdsofrespondentsweresatisedwiththetime;thisdropssharplyat36monthsto19%,andto9%forreviewtimesinexcessof6

months.

Editorsreportedaveragesubmission-to-acceptancetimesofroughly130days,

splitroughlyequallybetweentheinitialpeerreviewstagetorstdecision,and

subsequentrevisionstages.Nearlythree-quarters(72%)reportedtimesof6months

orbelow.Timeswereshortestinmedicalandnursingjournals,andlongestin

humanitiesandsocialsciencesjournals.Mosteditorswerehappywithreviewing

timesontheirjournals,butasubstantialminority(aroundathird)wasunhappy.

Theoverwhelmingmajorityofeditors(98%)gavetheirreviewersadeadlinefor

responding,withtheaveragedeadlinebeingabout34days,andwith63%ofeditors

giving30daysorless.Deadlineswereshorterinmedicalandnursingresearch,and

longestinhumanitiesandsocialsciencesandphysicalsciencesandengineering.

Figure 1Types of peer reviewexperienced by authors andused by journal editors

Single-blind peer review

Double-blind peer review

Open peer review

Post-publication review

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Experienced by all respondents Used by editors journals

72%

85%

22%

45%

3%

23%

1%

8%


8/20

8

2008


Free subscription

Acknowledgement in journal

Payment in kind

CME/CPD points

Name published

Report published anon.

Signed report published

Name disclosed to author

-60% -45% -30% -15% 0% 15% 30% 45% 60%

-7% 56%

-10% 44%

-12% 43%

-11% 20%

-38% 19%

-35% 16%

-47% 11%

Less likely More likely

-49% 10%

Figure 3Factors affecting reviewers

likelihood to review for a journal

Figure 2

Reasons for reviewing

To play your part as a member of the academic community

To enjoy being able to improve the paper

To enjoy seeing new work ahead of publication

To reciprocate the benefit when others review your papers

To enhance your reputation or further your career

Personal recognition/build a relationship with the editor

To increase the chance of a role in the journals editorial team

-50% -25% 0% 25% 50% 75% 100%

Disagree Agree

-3% 91%

-6% 78%

-11% 69%

-14% 67%

-28% 44%

-37% 30%

-50% 20%


9/20

9

2008


Thereviewersperspective

Researchersreportedreviewingregularlyfor3.5journalsandforafurther4.2

journalsoccasionally.Onaverage,reviewerssaidthattheyreviewedabout 8papers

intheprevious12months.Thisaverageguredisguisesthedistributionofreviews

amongreviewers.Weidentiedthegroupofreviewerswhoreporteddoing 6ormore

reviewsinthelast12months(activereviewers),andthisgroupmanagednearlytwiceasmanypapersastheaverage.Thismeantthatalthoughactivereviewersmade

upjust44%ofreviewersinoursurvey,theywereresponsiblefor 79%ofreviews.

Reviewerssaythattheytookabout24daystocompletetheirlastreview,with

85%reportingthattheytook30daysorless.Theyspentamedian5hours(mean

9hours)perreview.ActivereviewersandthoseintheEnglish-speakingregions

reportedspendingconsiderablylesstimeperreviewthanlessfrequentreviewersand

thosefromAsiaandtheRestofworld.

Weaskedreviewerstostatethemaximumnumberofreviewstheywereprepared

toundertake.Theaveragegureforallrespondentswas9reviews.Thiscompares

totheaverageof8reviewscompletedinthelast12months.Overall,therefore,

therewouldappeartobeatleastsomeslackinthesystem.Thisapparentposition

ofcomfortablecapacitybreaksdown,however,whenthedistributionofreviewsistakenintoaccount.Activereviewers(responsiblefor79%ofallreviews)proposed

amaximumof13papers,comparedtotheiraverageof14reviewsdoneinthelast12

months,suggestingthereisaproblemofrevieweroverloading.

Why reviewers review

Wewereinterestedtoexplorethereasonswhyreviewersreview,andwhatincentives

wereofferedandwhichwereeffective.

Ingeneral,respondentspreferredtooffermorealtruisticexplanationsforwhythey

reviewed(seeFigure2),withsubstantiallythemostpopularreasonbeingplaying

yourpartasamemberoftheacademiccommunity.Self-interestedreasonssuchas

toenhanceyourreputationorfurtheryourcareerortoincreasethechanceofbeing

offeredaroleinthejournalseditorialteamweremuchlessfrequentlyadvanced.

Themostcommonrewardsforreviewingreportedbyeditorswerereviewerreceptions

atconferencesandwaiverofauthorcharges(e.g.publication,page,colour,offprint

charges)(both39%).Monetarypaymentwasrareatonly5%ofeditors,thoughmore

commonthancreditsforcontinuingprofessionaldevelopment(2%).Paymentwas

mostcommoninhumanitiesandsocialsciencesjournals(9%).

Fromthereviewersperspective,theincentivestheysaidweremostlikelytoencourage

themtoactforajournalwere(seeFigure3):

afreesubscriptiontothejournal(56%saidthiswouldmakethemmorelikelyto

reviewforthejournal)

acknowledgementinthejournal(44%)

paymentinkindbythejournal,forexamplewaiverofcolourorotherpublication charges,freeoffprints,etc.(43%).

Payment for reviewers

Reviewersweredividedonwhethertheyshouldbepaidforeachreviewtheycom-

pleted:35%agreedthattheyshould,while40%disagreed.ThosefromtheAnglo-

phoneregionswerethemostopposedtopayment,whereasresearchersfromAsiaand

fromEuropewereonbalancejustinfavour(44%for,32%against).

Therewaslesssupportfortheideathatpaymentwouldreducetheobjectivityof

peerreview(28%for,43%against)butamajorityforthepropositionthatpayment

wouldmakethecostofpublishingtooexpensive(52%for,18%against).

Forthemostpart,respondentsviewsonthesequestionsappeartobepersonal

matters,independentoftheireldofresearch.Asalreadynoted,respondents


10/20

10

2008


preferredtogivealtruisticreasonsforreviewing.Womenthroughoutthesurvey

tendedtoexpressmorealtruisticpositionsthanmen;theirresponseshereare

consistentwiththatposition,withwomenbeinglessinclinedthanmentothink

reviewersshouldbepaid,andmoreinclinedtoseethedownside.

Roleotheeditor

Thefunctionoftheeditoristoselectthemostappropriatemanuscriptsforpubli-

cationintheirjournalandtoensuretheyareeditedandpresentedinthebestway

forthejournalsreadership.Theirpreciserolevariesconsiderablyfromjournalto

journal;forsomelargerscienticjournalsthereviewersmaybeselectedandmanaged

byaneditorialteamatthepublishersofcewiththeeditoronlybecominginvolved

oncethereviewerreportsarereceived(orinsomecases,onlyifthereisadispute

betweenreviewerstobeadjudicated),whileothereditorsaremuchmorehands-on,

appointing,selectingandchasingupthereviewersthemselves.

Inthesurvey,wefoundthateditorssaidthatthenumberofpaperstheyhandled

(i.e.thenumberonwhichtheymadeaccept/rejectdecisions)wasabout50peryear.

Themajority(59%)handled25orfewerpapersbuttherewasasmallgroup(11%)of

muchbusiereditorshandlingmorethan150.Editorsassignedabout2.3reviewersperpaper.Selectionofthereviewersbytheeditorthemselveswasonlythethirdmost

popularoption(reportedby28%ofeditors),wellbehindselectionbyamemberof

theeditorsteam(73%)andbyamemberofthepublishersstaff(43%).

Onlinemanuscriptsubmissionandtrackingsystemswereusedbyaboutthree-

quartersofeditors.Theirusewasmorecommoninlifesciences(85%)andmarkedly

lesscommoninhumanitiesandsocialsciences(51%).

Editorsreportedthattheaverageacceptanceratefortheirjournalswasabout50%,

whichisconsistentwithotherstudies(Figure4).About20%ofsubmittedmanu-

scriptsarerejectedpriortoreview(eitherbecauseofpoorquality(13%)orbeingout

ofscope(8%))andanother30%arerejectedfollowingreview.Ofthe50%accepted,

41%areacceptedsubjecttorevision.Acceptancerateswerelowerinhumanitiesand

socialsciences,andhigherinphysicalsciences/engineeringjournals.

TrishGroves(deputyeditorofthebmj)haswritten3thatanobviouswaytoimprove

anyjournalspeerreviewsystemistoTellauthorsandreviewerswhatyouwantfrom

them... .Givereviewersclearbriefs,includingguidanceonwhattoincludeinthe

review.Inthelightofsuchcommonsenseadvice,itwassomewhatsurprisingtond

that30%ofeditorsdidnotprovidereviewerswithachecklist.Theuseofchecklistswas

somewhatlesscommoninhumanitiesandsocialsciencesjournals( 45%notusing).

Whereeditorsdidprovidechecklists,themostcommonquestionsinvolvedthestudy

methodology(87%ofchecklists),relevance,importanceandpaperlength(Figure5).

Grovesalsowrote4Reviewershavealsotoldustheywantfeedbackontheirperfor-

mancesothattheycanlearnandimprove.Thisseemsanothercommonsense

positionbutonly28%ofeditorsinoursurveyreportedthattheygavefeedbacktoreviewersonthequalityoftheirreports.Themostcommonfeedbackgivenwasjust

thepublicationoutcome.

3Groves, T. (200) Quality and

value: how can we get the best out

of peer review? Nature (Nature Peer

Review debate).

doi:10.1038/nature04995

4ibid.


11/20

11

2008


Figure 5Items used in reviewerchecklists provided by editors

Methodology employed

Relevance

Importance

Paper length

Tables

Statistics

Originality

Experimental data

Illustrations

References

Language

Quality

Ethical issues

Other

87%

85%

78%

75%

60%

53%

53%

47%

42%37%

35%

34%

26%

8%

0% 20% 40% 60% 80% 100%

Reject after review

30%

Reject prior(poor quality)13%

Reject prior

(out of scope)8%

Accept, no revision8%

Accept, with revision41%

Figure 4Ultimate fate of submittedmanuscripts


12/20

12

2008


The benets of peer review

Whatarethebenetsofpeerreview?Inoneview,therearebenetsforallplayersin

thesystem:editorsaresupportedintheirdecisionsbytheviewsofexperts;authorsbenetfromtheassistanceofferedbyreviewersandfromthestatusconferredon

thembypublicationinjournalswithhighpeerreviewstandards;readersbenet

becauseofthelteringthatpeerreviewprovidesandbythesealofapprovalthatpeer

reviewisthoughttoprovide;andevenreviewers(whodothebulkoftheworkforno

directrecompense)benettosomeextent(e.g.inseeingworkpriortopublication).

Lookingbeyondtheinterestsoftheparticularstakeholders,therearethreemain

benetsadvocatedforpeerreview:

improvementinthequalityofpublishedpapers;

lteringoftheoutputofpaperstothebenetofreaders;

asealofapprovalthatthepublishedworkmeetscertainstandards,inparticular

forlayreaders.Letslookinmoredetailattheseproposedbenets.

Improvementsinquality

Thereareanumberofwaysinwhichpeerreviewmightimprovethequalityof

publishedpapers,ofwhichthemostimportantare:

theveryfactofaqualityhurdleorthreshold,whichwillmotivateauthorsto

improvethequalityoftheirworkpriortosubmission;

thepeerreviewprocess,inwhichreviewerscommentsandcriticismsareaddressed

bytheauthorbyrevisingthemanuscript.Testingofworkthroughthecriticismof

peersisinabroadsenseattheheartofthescienticmethod.

Perhapssurprisingly,thereislittlescienticevidencetosupporttheuseofpeer

reviewasamechanismtoensurequality(seebelow,underCritiques o peer review).

Inoursurvey,however,thelargemajorityofauthors(around90%)wereclearthat

peerreviewhadimprovedtheirownlastpublishedpaperandasimilarproportion

agreedwiththemoregeneralstatementpeerreviewimprovesthequalityofthe

publishedpaper.

Respondentswhosaidthatpeerreviewhadimprovedtheirlastpaperwereasked

whichaspectsofthepaperhadbeenimproved,andineachcasebyhowmuch(using

a15scale).TheresultsareshowninFigure6.

Some64%ofrespondentsreportedthatpeerreviewoftheirlastpublishedpaper

hadidentiedscienticerrors,demonstratingrealvaluebeingaddedbypeerreview,and78%saidithadidentiedmissingorinaccuratereferences.

Madesuggestionsonpresentationwasthemosthighlyratedaspect;94%ofthose

whosaidtheirpaperhadbeenimprovedreportedimprovementinthisarea,and

55%ratedtheimprovementat4or5outof5.Thelanguageorreadabilitywasalso

frequentlycited(86%reportedsomeimprovementinthisarea).

Thosewithgoodaccesstothejournalsliteraturereportedlessimprovementin

identifyingmissingofinaccuratereferencesthanthosewithworseaccess.Thisis

whatwemightexpecttondandillustratesonewayinwhichrestrictedaccessto

literaturecanaffectresearchers.

Therewassomewhatlessimprovementreportedregardingtheidenticationof

statisticalerrorsthanforotherbenets,although51%stillreportedsomeimprove-

ment.Isthisbecauseauthorsarelesslikelytomakestatisticalerrorsthanother


13/20

13

2008


kinds,orbecausereviewersarelesslikelytospotthem?Giventhatsomestudieshave

shownthatpublishedpapersarerifewithstatisticalerrors(e.g.EmilGarcia-Berthou

andCharlesAlcarazfoundstatisticalinconsistenciesin38%ofpapersinNatureand

25%intheBMJ5)thelatterseemsamorelikelyexplanation.

Peerreviewasaflter

Therearetwosensesinwhichpeerreviewandthejournalsysteminwhichitis

embeddedcanlterresearchoutputsforthebenetofreaders.

First,peerreviewcouldbeseentolteroutbadworkfromtheliterature,byreject-

ingitforpublication.Badworkherecouldmeanpoorlyconceivedorexecuted,or

ofminimaloriginalityorinterest,orbadinthemoralsense,forinstanceinvolv-

ingacademicfraudorplagiarism.Workthatdoesgetpublishedinapeer-reviewed

journalisseentohavemetsomequalitythresholdorgainedasealofapproval.

Groupspromotingbetterpublicunderstandingofsciencewilloftenusepeerreview

inthisway;forinstance,theuk groupSenseAboutSciencepromotesunderstandingofpeerreview,whichitcallstheessentialarbiterofscienticquality.6

Thereare,however,atleasttwoproblemswiththisposition.Becausethepeer

reviewstandardsofdifferentjournalsvary,itiswidelybelievedthatalmostany

genuineacademicmanuscript,howeverweak,canndapeer-reviewedjournalto

publishitiftheauthorispersistentenough.Manuscriptsrejectedbyonejournalare

routinelysubmittedtoanother,probablyonewithalowerrejectionrate.Acceptance

byapeer-reviewedjournaldoesnotsayverymuchaboutthequalityororiginality

ofapaperbutitmaystilldistinguishitfrompseudoscienceoregregiouslybadwork,

andthisisthewayinwhichgroupslikeSenseAboutSciencebelieveitcanhelpthe

public.Theotherproblemisthatpeerreviewhasbeenshownnottobeparticularly

effectiveasaqualitycontroltool,oratdetectingerrorsoroutrightfraud.(These

problemsarediscussedinmoredetailbelow,seeCritiques o peer review.)

Dont know/Not applicable1 (no improvement) 2 3 4 5 (substantial improvement)

Made suggestions

on presentation

The language or readability

Identified missing orinnacurate references

Identified scientific errors

Identified statistical errors

5% 12% 27% 37% 18% 1%

12% 20% 25% 27% 14% 2%

24% 19% 21% 21% 10% 3%

30% 16% 17% 18% 11% 6%

35% 15% 15% 13% 6% 14%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Figure Improvements made by peer reviewto authors last published paper

5Garcia-Berthou, E. and Alcaraz, C.(2004) Incongruence between teststatistics and P values in medicalpapers. BMC Medical ResearchMethodology4: 13.

See http://www.senseabout-science.org.uk/index.php/site/project/29/


14/20

14

2008


Thesecondwayinwhichpeerreviewcanprovidealterforreadersismuchmore

importantforworkingacademics:itprovidesthebasisforthestraticationof

journalsbyperceivedquality(wherequalityisfrequentlytakentobeindicatedbythe

impactfactor,ameasureofhowoftenonaveragearticlesinthejournalinquestion

arecited).Peerreviewthussupportsthesystemthatroutesthebetterpaperstothe

betterjournalsandthisallowsacademicstofocustheirreadingonamanageable

numberofcorejournalsintheireld.Publishersinparticularseethiskindoflter-ingasoneofthemajorbenetsofpeerreviewandthejournalssystem.

Respondentstooursurveyhavealotofcondenceinthepeerreviewsystemto

supporttheselteringfunctions(seeFigure7).Aswellasverystronglysupporting

thenotionthatpeerreviewimprovesthequalityofpublishedpapers(asdiscussed

above),therewasalsostrongsupportfortheideathatitdeterminestheimportance

ofthendingsandtheoriginalityofthemanuscript.Therewassomewhatless

support(thoughstillanetmajority)forbelievingpeerreviewwaseffectiveatdetect-

ingplagiarismandacademicfraud.

Viewsosurveyrespondents

Overall satisaction with peer reviewThemajority(64%)ofacademicsdeclaredthemselvessatisedwiththecurrent

systemofpeerreviewusedbyjournals,withjust 12%sayingtheyweredissatised

(Figure8).Therewasverylittlevariationamongstthesampleinthesegures;for

instancetherewerenodifferencesbyage,genderorposition(seniority).

Respondentsattitudeswerealsotestedbyaskingfortheirdegreeofagreementor

disagreementtowardsanumberofstatementsaboutpeerreview,asshowninFigure9.

Onthepositiveside,thelargemajority(85%)agreedwiththepropositionthat

scienticcommunicationisgreatlyhelpedbypeerreview.Therewasasimilarlyhigh

levelofsupport(83%)fortheideathatpeerreviewprovidescontrolinscientic

communication.

Giventhegenerallylowlevelofoveralldissatisfactionwithpeerreview,though,

itisperhapssurprisingthatastrongstatementlikepeerreviewinjournalsneedsa

Figure Views on the effectiveness of peerreview in different areas

Disagree Agree

Improves quality

Determines importance

Determines originality

Picks best mss for journal

Detects plagiarism

Detects fraud

-30% -20% -10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

-3% 90%

-16% 60%

-17% 58%

-22% 49%

-24% 46%

-26% 43%


15/20

15

2008


completeoverhauldidnotreceivemoredisagreementinfactrespondentswere

divided,with35%disagreeingversus32%agreeing.Similarly,respondentswere

dividedonwhetherthecurrentpeerreviewsystemisthebestwecanachieve,with

32%agreeingversus36%disagreeing.

Therewas,however,virtuallynosupportfortheradicalpropositionthatpeer

reviewwascompletelyunnecessary.

Onlyaminorityoverall(19%)agreedthatpeerreviewwasholdingbackscientic

communication.Thosewithpoor/verypooraccesstothejournalsliteraturetended

toagreemore(23%)thanthosewithexcellentaccess(16%).

Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Very dissatisfied Dont know

6% 59% 22% 10% 2% 1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Greatly helps scientific commn

No control without peer review

Needs complete overhaul

Current system is best achievable

Holds back scientific commn

Completely unnecessary

Disagree Agree

-100% -75% -50% -25% 0% 25% 50% 75% 100%

-5% 85%

-10% 83%

-35% 32%

-36% 32%

-63% 19%

-93% 3%

Figure 8Overall satisfaction with the peerreview system used by scholarly

journals

Figure 9Views on peer review


16/20

1

2008


The dissatisfed group

Whilethelargemajorityofrespondentsexpressedthemselvessatisedwiththepeer

reviewsystemusedbyscholarlyjournals,aminority( 12%)saidtheyweredissatised

orverydissatised.Itisinterestingtoaskwhatwecansayaboutthisgroup.

Intermsofdemographics,therearerelativelyfewdifferencesfromtheaverage.

Therewerenosignicantdifferencesbyage,gender,typeoforganizationorposition

(seniority).Byregion,theyweremoresomewhatlikelytobeintheAnglophoneregions,andlesslikelytobeinAsiaortheRestofworld.Lookingateldofresearch,

theyweremostlikelytobeinhumanitiesandsocialsciences,andleastlikelyin

physicalsciences/engineering.

Intermsoftheirownexperienceofpeerreview,thisgroupreportedthatthepeer

reviewoftheirlastpublishedpapertooksignicantlylongerthanaverage(about110

comparedto80days),andtheyweremorelikelytobedissatisedwiththelength

oftimeinvolved.Thedissatisedgrouptendedtobesomewhatlesslikelytoreport

thatpeerreviewhadimprovedtheirlastpublishedpaper,andlikelytogivelower

scorestotheimprovementstheydidreport.Wecannotfromthedatasayifthereis

acausalrelationship;thatis,isthisgroupdissatisedwithpeerreviewbecausethey

haveexperiencedlongertimesandlesspersonalbenetontheirownpapers,ordoes

theirdissatisfactionarisefromothercausesandthenleadthemtogivelesspositivescores?

Intermsofalternativeapproachestopeerreview,thisdissatisedgroupwasmore

likelytoagreethatopenandpost-publicationreviewwereeffective.Asasmall

minority,however,theydidnotformthemainconstituencyforthesealternative

approaches.

Critiques of peer review

Peerreviewisnotwithoutitscritics.

Perhapsthestrongestcriticismisthatthereisalackofrealevidencethatpeer

reviewactuallyworks:forinstance,a2002studypublishedinthe Journal o the

American Medical Association7concludedthatEditorialpeerreview,althoughwidely

used,islargelyuntestedanditseffectsareuncertain.Similarly,theCochrane

Collaboration(auk-basedinternationalhealthcareanalysisgroup)rstpublished

itsownreviewin2003,whichconcludedthattherewaslittleempiricalevidence

tosupporttheuseofeditorialpeerreviewasamechanismtoensurequalityof

biomedicalresearch,despiteitswidespreaduseandcosts.Thelatestupdate(2007)oftheCochranereviewconrmsthisconclusion8,thoughitisimportanttounder-

standthatitissayingthattheevidencetosupportpeerreviewhasnotyetbeen

produced,notthatthereisevidencethatpeerreviewdoesnotwork.

Somehaveshownthatpeerreviewcanbeunreliable.Forinstanceonestudy9

showedthatthechancesoftworeviewersagreeingaboutaparticularpaperwere

onlyslightlybetterthanchance;inordertoproduceareliableresult,editorswould

needtousesixreviewersforeachpaper.(Inpractice,theytypicallyusetwoorthree

theaveragereportedinthissurveywas2.3.)

Otherstudieshaveshownthatpeerreviewcanbenotverygoodatdetecting

errors.Godleeandcolleaguesatthebmjtookapaperabouttobepublished,

insertedeightdeliberateerrors,andsentthepaperto420potentialreviewers:221

(53%)responded.Theaveragenumberoferrorsspottedwastwo,nobodyspotted

Jefferson, T., Alderson, P., Wager, E.and Davidoff, F. (2002) Effects ofEditorial Peer Review: A SystematicReview.Journal of the American Medical

Association28: 284-28.

8Jefferson, T., Rudin, M., Brodney Folse,S. and Davidoff, F. (200) Editorialpeer review for improving the qualityof reports of biomedical studies.

Cochrane Database of Sytematic Reviews200, Issue 2. Art. No.: MR00001.DOI: 10.1002/1451858.MR00001.pub3

9Rothwell, P.M. and Martyn, C.N.(2000) Reproducibility of peer reviewin clinical neuroscience: is agreementbetween reviewers any greater thanwould be expected by chance alone?Brain 123: 194-199.


17/20

1

2008


morethanve,and16%didntspotany10.

Itisalsosaidthatpeerreview,particularlyinitssingle-blindform,offerstoo

muchscopeforbiasonthepartoftherevieweroreditor.Forinstance,papers

publishedinanissueoftheJournal o the American Medical Association11devotedto

peerreviewpresentedevidencefornationalitybias,languagebias,specialtybias,and

perhapsevengenderbias,aswellastherecognisedbiastowardthepublicationof

positiveresults.Oneresponsetotheproblemsofreviewerbiashasbeentomovetodouble-blind

ratherthansingle-blindreview.However,thesecrecyinvolvedinblindingthe

reviewersidentityhasitselfbeencriticisedontwomaingrounds.Fromapragmatic

viewpoint,moststudiesthathaveinvestigatedreviewerblindinghavefailedto

measureimprovementsinthequalityofthereviewand,conversely,otherstudies

haveshownthatmakingthereviewersidentityknowntoauthorshadnoeffect

onquality.12Thereisalsoastrongethicalargumentagainstsecrecy,namelythatit

isseentobeunfairforsomebodymakinganimportantjudgementontheworkof

otherstodosoinsecret.

Anotherargumentagainstdouble-blindingisthatitisverydifcultinpractice

todisguisetheidentityoftheauthorofanacademicmanuscriptfromaskilled

reviewer;bydenitionthereviewerisanexpertintheeldwhowillfrequentlyknowthepreviousworkofauthorsintheeld.

Otherpragmaticcriticismsofpeerreviewincludethedelayitcausestopublica-

tionandtheviewthatitdoesnotscaleefcientlywiththegrowthofscience.The

surveyshowedsomebasisforeachofthese.Althoughtheaveragedelayreportedby

authorsforpeerreviewwasonlyabout80days,39%reportedtimesofgreaterthan

3months,and10%ofgreaterthan6months.Editorsreportedthattheaverage

timefromsubmissiontoacceptanceontheirjournalswasabout130days,with

22%reportingtimesofmorethan6months.Therewasacorrelationbetweenthose

reportinglongerreviewtimesandloweroverallsatisfactionwithpeerreview.The

surveyalsoshowedthatthelargemajorityofreviewswereundertakenbyacore

groupofactivereviewerswhoappeartobeoverloaded.

Viewsosurveyrespondents

Howdidsurveyrespondentsdealwiththesecriticisms?Inthemostpart,aswehave

alreadyseen,respondentshadpositiveviewsaboutpeerreviewanditseffectiveness

atimprovingthequalityofpublishedpapers.Theirviewsonalternativesystemsof

peerreview,whichhavebeenproposedatleastinpartasresponsestocriticismsof

conventionalpeerreview,areexploredinthenextsection.

11Journal of the American Medical

Association (1998) 280: issue 3.

10Godlee, F., Gale, C. R. and Martyn,C. N. (1998)Journal of the AmericanMedical Association 280: 23-240.

12E.g. Goldbeck-Wood, S. (1999)

Evidence on peer review: scienticquality control or smokescreen?British Medical Journal 318: 44-45.


18/20

18

2008


Alternative approaches

Dierenttypesopeerreview

Mostrespondentsinmosteldsexperiencesingle-blindreviewasthenorm.When

askedwhichoptionstheythoughtwereeffective,however,respondentsexpresseda

clearpreferencefordouble-blindreview,asshowninFigure 10.Thelevelofsupport

fortheeffectivenessofpost-publicationreviewissurprisinglyhigh.

Respondentsdidnothavepersonalexperienceofalltypesofreview.Thosewith

experienceofdouble-blindreviewweresubstantiallylesslikelytoratesingle-blind

reviewaseffectivecomparedtoothers.Similarly,thosewhohadexperienceofopen

peerreviewandpost-publicationreviewasanauthorwereconsiderablymorelikelytoratethemaseffective.Itisnotable,though,thatalthough37%ofrespondents

saidthatpost-publicationreviewwaseffective,only8%hadhadexperienceofitas

authorsthissupportisthereforesomewhathypothetical.

Askedwhichofthefourpeerreviewtypeswastheirpreferredoption,therewasa

clearpreferencefordouble-blindreview,with56%selectingthis,followedby25%

forsingle-blind,13%foropenand5%forpost-publicationreview.Post-publication

reviewgetsmuchlesssupportherecomparedtotheperceptionsofitseffectiveness:

thisisnotinconsistentbecauserespondentsclearlysawitasausefulsupplementto

currentpeerreviewmethodsratherthanareplacementforthem.

Itwasclearfromtheverbatimcommentsthatthepreferencefordouble-blind

reviewwaslargelyaresponsetothepotentialforbiasinsingle-blindreview:the

reasonsgivenforthispreferencewereprimarilyitsobjectivityandfairness.

Single-blind

Double-blind

Open

Post-publication

0% 10% 20% 30% 40% 50% 60% 70% 80%

25%

52%

56%

71%

13%

27%

5%

37%

Thought effective Preferred option

Figure 10Types of peer review thought to beeffective (multiple responsesallowed), and respondents

preferred choice (single response)


19/20

19

2008


Post-publicationreview

Lookinginmoredetailatpost-publicationreview(Figure 11),researcherssawit

asausefulsupplementtoformalpeerreviewinquitelargenumbers( 53%agreed

comparedto23%disagreeingwiththisstatement).Theyseethisusefulnessdespite

aclearperceptionthatittendstoencourageinstantreactionsanddiscourage

thoughtfulreview.Thereislesssupportfortheideathatitcouldbealessgoodbut

stillacceptablealternative(31%supportedversus43%opposed)andfairlystrong

oppositiontotheideathatitcouldbeanequallypowerfulalternativetoformalpeer

review(57%opposedversus19%supported).Therewasevenstrongeroppositionto

replacingpeerreviewwithpost-publicationratingsorusageorcitationstatisticsto

identifygoodpapers.

Openpeerreview

Supportforopenpeerreviewstartedtogrowduringthemid- 1990s.TheBMJwasone

oftherstmajorjournalstoadoptopenpeerreview,basingitsdecisionpartlyonthe

ethicalcaseagainstsecrecyandpartlyontheevidencementionedabovethatblinding

didnotimprovereviewoutcomes.Openreview,however,remainsfarfrombeingthe

norm.Themainargumentagainstitisthatreviewerswillbereluctanttocriticisetheworkofmoreseniorresearchersonwhomtheymaybedependentforcareeradvance-

mentorgrantawards.During2006,thejournalNatureconductedatrialofopen

peerreview13;itwasnotasuccessdespiteinterestinthetrial,onlyasmallpropor-

tionofauthorschosetoparticipate,andonlyafewcommentswerereceived,many

ofwhichwerenotsubstantive.Feedbacksuggestedthatthereisamarkedreluctance

amongresearcherstoofferopencomments.

Inthesurvey,thenumbersofrespondentspreferringopenpeerreviewwere

smallerthanforsingle-ordouble-blindpeerreview(about13%).Themainreasons

givenforpreferringitwere:revieweraccountability,leadingtobetterreportsand

lesslikelihoodofbias,andtheviewthatopenreviewmadereviewersmorecivil,

madetheprocessmoreofadialoguewiththeauthorandgenerallyimproved

author/reviewercommunication.

Disagree Agree

Encourages instant reactions

Useful supplement to formal review

Authors less likely to submit

Readers fear offending authors

Would relieve load on reviewers

Acceptable (but weaker) alternative

An equally powerful alternative

-60% -45% -30% -15% 0% 15% 30% 45% 60%

-16% 56%

-23% 53%

-29% 41%

-31% 39%

-32% 38%

-43% 31%

-57% 19%

Figure 11Views on post-publication review

13Nature editors/publishers.(200) Overview: Natures peerreview trial. Nature doi:10.1038/nature05535.


20/20

20

2008


Proponentsofopenpeerreviewwillalsohavetoovercomethefactthat47%of

reviewerssaidthatpublishingtheirsignedreportwouldmakethemlesslikelyto

reviewforajournalandthatasimilarproportion,49%,wouldseedisclosureof

theirnametotheauthorasadisincentive(seeFigure3above).

Reviewingauthorsdata

Asscienceutilizesmoreautomatedexperimentalequipmentandotherwisemoves

towardsamoredata-centrice-sciencemodel,theamountofdatathatsupports

(andcouldpotentiallybelinkedto)theaveragescienticpaperincreases.The

questionarisesastowhetherthisdatashoulditselfbesubjecttopeerreview.There

areclearlyanumberofpracticalissues:doreviewershavethetimetodothis?Is

thedatasufcientlystandardized,anddothesoftwaretoolsexisttohandleit?Are

authorsevenpreparedtosharetheirdatawithreviewers?

Amajorityofreviewers(63%)andeditors(68%)saidthatitwasdesirableinprin-

cipletoreviewauthorsdata.Perhapssurprisingly,amajorityofreviewers(albeita

smallone,51%)saidthattheywouldbepreparedtoreviewauthorsdatathemselves,

comparedtoonly19%whodisagreed.Thiswasdespite40%ofreviewers(and45%of

editors)sayingthatitwasunrealistictoexpectpeerreviewerstoreviewauthorsdata.

Conclusions

Thesurveythuspaintsapictureofacademicscommittedtopeerreview,withthe

vastmajoritybelievingthatithelpsscienticcommunicationandinparticular

thatitimprovesthequalityofpublishedpapers.Theyarewillingtoplaytheirpartin

carryingoutreview,thoughitisworryingthatthemostproductivereviewersappear

tobeoverloaded.Manyoftheminfactsaytheyarewillingtogofurtherthanat

presentandtakeonresponsibilityforreviewingauthorsdata.

Withinthispictureofoverallsatisfactionthereare,however,somesizeable

pocketsofdiscontent.Thisdiscontentdoesnotalwaystranslateintosupportfor

alternativemethodsofpeerreview;forexamplesomeofthosemostpositiveabout

thebenetsofpeerreviewwerealsothemostsupportiveofpost-publicationreview.

Acknowledgements

ThanksareduetoMikeMonkman(MikeMonkmanMedia,http://mikemonkman.

com/)forhelpindesigningthesurvey,forprovidingtheonlinehostingfacilitiesand

fortheanalysisofthesurveydata.ThanksarealsoduetoMayurAminandAdrian

Mulliganforhelpfuldiscussionsandsuggestionsregardingthequestionnairedesign

andinterpretation,andtoLouiseHallforhelpwithsomedataanalysis.

TheworkwasfundedbythePublishingResearchConsortium.

Documents

Prc Summary 4 Ware Final