Peer review

Peer Review: what is it?

• The central process of academia: your peers review, judge your work.

• The source of rigour in science.

• Hellish and unavoidable process of academia. The source of great pain, misery and crushing hammer blows to your confidence.

http://www.scieng.ed.ac.uk/transkil.htmScientific Paper Production

• My goal: introduce you to the hardship of peer review (with a positive coda)

Peer Review: how does it work?

• used to assess our two major currencies: journal publications and grant-getting

–But process similar: your work sent to a journal editor or funding council (e.g. NERC, MRC) or charity (e.g. Wellcome), who then mediate a (largely) anonymous peer-review process

–Your document sent to peers who pass judgement with a report

• specifically asked to highlight both positives and negatives

Who are your peers?

• Colleagues in the same field–Though definition of ‘same’ can vary–they are your competitors (= - bias)–Because we move in small circles, they are your friends (= + bias). –But don’t underestimate your friends

*small circles• Referees are always out there

–They go to your talks at conferences- take the opportunity to impress

• Give a good talk, be well read etc. IMPRESS! It could make your life much easier

–Thoroughly understand the work of potential referees. think about who might referee your paper–Criticise the work of others very carefully (e.g. in the intro of manuscripts)

Who are your peers? You can often choose

• Journals and funding bodies require you to nominate referees, though they will not use all of your suggestions

• Your friends may or may not treat you better- so choosing refs is not easy– deserves careful, careful thought

• But overall, it has been shown that choosing helps

Science, Vol 309, Issue 5743, 1974 , 23 September 2005

Can suggest referees. Can ask that certain refs be excluded

It pays to know your field very well

Who are your referees?

• Typically, you will not know- even if they are your suggested ref, they probably wont sign review. – Freedom to be cruel?

• But they will know who you are– Bias

• Helps famous people?

• But can also help you due to having met, socialised, impressed?

Referees are often wrong, often right and frequently nuts

Oikos 67: 577-581 (1993)

.. the explanation is interesting, and worth stating (but) I didn't need to read the paper to get the point

Annals of the Entomological Society of America 88: 100-103 (1995)

“This is still a most unexciting paper, but it is probably useful to confirm experimentally what everyone knows intuitively. And so I accept it for the Annals.”

Slight Insanity

Reviewer 1. “By necessity they [models] are so simplified and artificial that they possess little ability to represent even the most basic dynamics of the real world”

“Any manager that took this approach should be fired”

Reviewer 2.“a simple and elegant model”

“What makes this an exciting paper is that the conclusions can be readily acted upon by land managers”

Slight Insanity

• REFEREE 2: [this paper concerns] is an interesting topic, but this paper does not provide any substantial original contribution. Furthermore, the authors cite the literature very selectively, ….the failure to cite the literature in some instances, and the blatant misinterpretation of the literature in other cases, borders on unethical behavior.

More Severe Insanity

• I raised the issue of [topic] about a decade ago (XXXX, 1994, Journal). This is the basis of the introduction of the present paper, but the authors fail to acknowledge the 1994 paper. The … effect is discussed in XXXX et al, 2002 (Journal 2001) and then again in XXXX, 2002 (Journal). The introduction of the present paper adds nothing new.

Table 1 reviews studies of… the … experiment in XXXX et al. (2002), in which the decoy greatly increased the absolute preference for the target, there is an asterisk stating that "the authors report this result, however no supporting statistics are presented". XXXX et al. (2002) present these data on page 185 … . The decoy effect was so remarkably strong, that statistics were almost unnecessary. Standard errors are shown in the figure, showing clear differences between means, and the figure caption states that the differences were statistically significant. In the text, there are two paragraphs full of supporting statistics, with test statistic values and probability values.

… Even if English were the fourth language of the authors, I would find it difficult to understand how they could arrive at the conclusion that "no supporting statistics are presented". I can only interpret this as another blatant lie intended to justify their present study.

Some referees just don’t get out enough

…Seems hopeless,

Ms had 35 paragraphsNumber of negative comments from referee 70Number of positive comments by referee: 1 (though only allusion)

But for the same article……

• Reviewer #2(Remarks for the Author):I think this is a well written paper that details clearly a set of elegant laboratory experiments. The three key results (i.e. that virulence is temperature dependent, that census date matters and that there are strong GxE interactions that alter resistance rankings) are highly significant in the context of coevolution and have clear implications for many previous studies of parasite-mediated selection. As such, I think it makes a timely contribution to a topical research area and I fully support its publication in Evolution.

Another nice contrastReviewer 1>This is a very silly paper. It is not necessary to construct a model to see>that if two costs (not mating with the fittest male and not mating with a>compatible male) impinge on mating behaviour, the one with the larger effect>is likely to dominate. This is all that figure 1 shows. Previous verbal>arguments are sufficient. It would also be useful if the authors set out>what incompatibilities they are modelling. Is this an investigation of>interspecific incompatibilities?

Reviewer 2>>This is an interesting and well-written paper that represents the>>first quantitative attempt to incorporate both "good genes" sexual>>selection and genetic compatibility-based polyandry in a single>>theoretical framework. Although the authors present a very simple>>model that is not explicitly genetic, the paper nonetheless makes a>>very valuable contribution to the important issue of whether females>>should exercise choice at the precopulatory stage or engage in>>polyandry.

Overall

• Don’t put too much weight on the comments of any one referee.

Referees can be the wind beneath your wings

And even when they criticise, find silver linings

• Helpful: a source of good ideas (your next version of the paper will be improved), a source of collaboration (you will often meet them)

What are the potential outcomes?

What are the potential outcomes?

• Reject • Your paper is pretty shabby, but we’ll let

you spend a couple of months fixing it up before we review again. No guarantees, but reason for hope

• Accept provided you fix this and that (only editor re-reviews)

• Accept without revision (does’nt happen)

What can you do in response?

• First, there is what you have to do: respond– By extremely polite and tactful- this can be incredibly hard

• supervisor can provide you with examples of appropriate responses

– Referee has spent often considerable time on you

– Bear in mind that the comments you have received, may not be all the comments. Be easy on the editor (esp when they may not be professional)

– Bear in mind that every referee’s comment, no matter how harshly put or seemingly insane, will have a helpful kernel

– At the very least, their stupidity might reflect an less than perfect explanation on your part (this is anyways a good hook)

– address their comments even if you plan to submit elsewhere

Dear Editor,

Thank you for the offer of resubmission.

The referees gave the paper an impressively thorough reading, and I would like to thank them for that. The comments they made were on the whole fair- which meant the revision was a lot of work! But the paper is clearly much improved.

I believe have addressed the major concern of both referee's and the AE, namely, I have fixed up and justified the tests for selection……

Minor points:The referee was correct to point out that……

We sympathise with the referees confusion over table one, and have taken steps to rectify what was admittedly a muddled presentation of that aspect of the….

Responding to referees (always done via the editor)

You may have to work on the editor a bit as well…..

Additional things you can do

• For key criticisms, contact referee through editor- this can lead to something fruitful

• You cannot escape peer review, so learn from the process. It is an opportunity to improve your paper and your science communication generally

Then, if refereeing yourself:

• Always, always bear in mind how you felt when reading harsh reviews, and temper your own reviews– Just as all referees comments have merits, all

papers have good bits, so mention them.

– It is entirely possible to be rigorous and helpful