The Review Process: Responsibilities and Rewards

Patrick J. FlynnPatrick J. Flynn

Associate ProfessorAssociate Professor

flynn@nd.eduflynn@nd.edu

Outline

Introduction and DefinitionsIntroduction and Definitions Reviewing: what to do, what not to do, Reviewing: what to do, what not to do, etcetc.. Examples and anecdotesExamples and anecdotes Plagiarism: definitionsPlagiarism: definitions Avoiding plagiarism: rules for writersAvoiding plagiarism: rules for writers Examples and anecdotesExamples and anecdotes ConclusionsConclusions

Goals

Discuss (interactively) an infrequently-discussed Discuss (interactively) an infrequently-discussed professional duty and its detailsprofessional duty and its details

For your own benefit as an author, understand the For your own benefit as an author, understand the review processreview process

For the community’s benefit (from quality For the community’s benefit (from quality reviews), discuss expectations of reviewersreviews), discuss expectations of reviewers

For society’s benefit, reinforce the traditional For society’s benefit, reinforce the traditional negative attitude toward plagiarismnegative attitude toward plagiarism

Introduction

Peer Review: a “community assessment” of Peer Review: a “community assessment” of qualityquality of papers presented at meetingsof papers presented at meetings of papers submitted to the archival of papers submitted to the archival

literatureliterature of proposals submitted for fundingof proposals submitted for funding of the “body of work” of a professor of the “body of work” of a professor

seeking tenure and/or promotionseeking tenure and/or promotion

Introduction (ctd.)

Peer reviewPeer review ImportantImportant Fair (no “guaranteed publication” because of Fair (no “guaranteed publication” because of

who you are)who you are) ExpensiveExpensive

Time of reviewersTime of reviewersTime of editorial staff and volunteersTime of editorial staff and volunteers

Confidential (usually; depends on venue)Confidential (usually; depends on venue)

Editorial Flow in a typical journal

1.1. Author submits paper for reviewAuthor submits paper for review

2.2. Editorial staff logs the paper and sends it to Editorial staff logs the paper and sends it to the Editor (Editor-in-Chief, the Editor (Editor-in-Chief, etcetc.).)

3.3. Editor sends it to an Associate Editor (AE)Editor sends it to an Associate Editor (AE)

4.4. AE identifies reviewers and tells staff to AE identifies reviewers and tells staff to send paper to reviewerssend paper to reviewers

5.5. Several months go bySeveral months go by

Editorial Flow (ctd.)

7.7. AE assembles reviews and decides:AE assembles reviews and decides:• Accept (rare)Accept (rare)• Minor revision (examined by AE)Minor revision (examined by AE)• Major revision (re-reviewed by same Major revision (re-reviewed by same

reviewers)reviewers)• RejectReject

8.8. Inform author of decisionInform author of decision

Editorial Flow (ctd.)

9.9. Await revised manuscript (several Await revised manuscript (several months) and statement of changes made in months) and statement of changes made in response to reviewsresponse to reviews

10.10. AE examines revision and (if applicable) AE examines revision and (if applicable) obtains new reviews (several months)obtains new reviews (several months)

11.11. Decision is made; accepted paper Decision is made; accepted paper scheduled for publicationscheduled for publication

Editorial Flow Problems

Time delay (biggest problem)Time delay (biggest problem) Unresponsive AEs or EICsUnresponsive AEs or EICs Unresponsive reviewersUnresponsive reviewers

Poor quality reviews (short and Poor quality reviews (short and meaningless, biased, hypercritical, meaningless, biased, hypercritical, incompetent, …)incompetent, …)

Author complaints (often justified, Author complaints (often justified, sometimes not)sometimes not)

Referee Duties Handout: “The Task of the Referee”, A. Handout: “The Task of the Referee”, A.

Smith, Smith, IEEE ComputerIEEE Computer, April 1990, pp. 65-, April 1990, pp. 65-71.71.

Determine the paper’s ‘contribution’Determine the paper’s ‘contribution’ GoalGoal ScopeScope Potential impactPotential impact

Referee Duties (ctd.)

Check accuracy and validityCheck accuracy and validity Determine appropriateness to venueDetermine appropriateness to venue Assess quality of presentationAssess quality of presentation Spot instances of plagiarism or poor citation Spot instances of plagiarism or poor citation

qualityquality If paper is not publishable as is, but is If paper is not publishable as is, but is

salvageable, determine where and how salvageable, determine where and how improvements can be made.improvements can be made.

Referee duties

Write a Defensible ReviewWrite a Defensible Review Summary of paperSummary of paper Assessment of appropriateness, correctness, Assessment of appropriateness, correctness,

impact, and presentationimpact, and presentation Recommendation: accept, revise, rejectRecommendation: accept, revise, reject Suggestions for improvement (citations to add, Suggestions for improvement (citations to add,

details, details, etc.etc.)) Don’t use your position as reviewer to force Don’t use your position as reviewer to force

unnecessary citations to your own workunnecessary citations to your own work

Example Reviews

See handoutSee handout First: First: IEEE TransactionsIEEE Transactions review (negative) review (negative) Second: NSF (medium)Second: NSF (medium) Third: Third: IEEE TransactionsIEEE Transactions review review

(lukewarm)(lukewarm)

MostMost reviews are “medium” (few strongly reviews are “medium” (few strongly negative, few strongly positive)negative, few strongly positive)

Re-use and fair use

See Stone’s paper in the handoutsSee Stone’s paper in the handouts Citations: always required unless prior work was Citations: always required unless prior work was

not published not published anywhere in any formanywhere in any form Sometimes, even previously unpublished work is Sometimes, even previously unpublished work is

cited (it never hurts to cite)cited (it never hurts to cite) Generally, redundant citations (Generally, redundant citations (e.g.e.g., to conference , to conference

paper and a follow-on journal paper) aren’t paper and a follow-on journal paper) aren’t warrantedwarranted

Self-citations should be kept to a minimum (avoid Self-citations should be kept to a minimum (avoid gratuitous self-citations)gratuitous self-citations)

Re-use and fair use (ctd.)

QuotationsQuotations Short (fair use): cite and quoteShort (fair use): cite and quote Long: cite, inset or quote, obtain permissionLong: cite, inset or quote, obtain permission ““Almost verbatim”: cite, inset or quote, obtain Almost verbatim”: cite, inset or quote, obtain

permissionpermission Reuse of artwork: cite, inset or quote, obtain Reuse of artwork: cite, inset or quote, obtain

permissionpermission Permissions obtained from Permissions obtained from copyright holdercopyright holder

Copyrights

If your work is published, you assign If your work is published, you assign (transfer) copyright to the publisher(transfer) copyright to the publisher

You You cannotcannot then give permission for re-use. then give permission for re-use. Handout: copyright form (note: revised to Handout: copyright form (note: revised to

warrant compliance with the DMCA)warrant compliance with the DMCA) Handout: IEEE re-use permission form Handout: IEEE re-use permission form

(routinely granted)(routinely granted)

Plagiarism and other unprofessional acts Re-use without citationRe-use without citation Multiple submissionsMultiple submissions Theft and submission/publication of others’ Theft and submission/publication of others’

ideasideas

Re-use without citation

Handout: Lai/Sethi incidentHandout: Lai/Sethi incident Sethi’s complaintSethi’s complaint Ledley’s responseLedley’s response The paper in questionThe paper in question The relevant pages from Sethi’s The relevant pages from Sethi’s TPAMI TPAMI

paperpaper

Re-use ctd.

CommentsComments Plagiarized amount: 2-3 paragraphs (not Plagiarized amount: 2-3 paragraphs (not

much)much) It was still plagiarism, It was still plagiarism, even though the even though the

source was citedsource was cited.. More than fair useMore than fair use

Another example

Handout: Editorial from Bob LedleyHandout: Editorial from Bob Ledley Hsia and Huang, Hsia and Huang, Pattern RecognitionPattern Recognition

26:333-34926:333-349 Two pages taken from Kriegman and Ponce Two pages taken from Kriegman and Ponce

((IJCVIJCV 5:119-135) 5:119-135)

But here’s another problem

Insufficient number of reviewersInsufficient number of reviewers Pattern RecognitionPattern Recognition: only : only oneone review per review per

submissionsubmission Plagiarism hence easier to missPlagiarism hence easier to miss

But it’s less important

Authors should be professional in citing Authors should be professional in citing other workother work It’s expectedIt’s expected It’s required by law and conventionIt’s required by law and convention It speaks to reputation (very important to It speaks to reputation (very important to

young researchers)young researchers)

Another example

TPAMI submission, from a previously unknown TPAMI submission, from a previously unknown author with no academic or industrial affiliationauthor with no academic or industrial affiliation

Discovered to be a photocopy of a technical report Discovered to be a photocopy of a technical report from a leading researcher (with a new cover page)from a leading researcher (with a new cover page)

PAMI “death penalty” (no future submission will PAMI “death penalty” (no future submission will be accepted)be accepted)

Scary: the paper was out for review, but the Scary: the paper was out for review, but the cognizant AE spotted the problemcognizant AE spotted the problem

Multiple Submissions

Handout: Handout: IEEE TPAMI IEEE TPAMI editorial (Bowyer editorial (Bowyer and Flynn)and Flynn)

The The samesame paper was submitted paper was submitted administratively to TPAMI and IVCadministratively to TPAMI and IVC

Front 40% identicalFront 40% identical Remainder very similarRemainder very similar Rejected from TPAMI without reviewRejected from TPAMI without review

Multiple submissions (ctd.)

Another case: paper submitted to at least Another case: paper submitted to at least four journals (including PAMI) four journals (including PAMI) simultaneously (PJF got it twice)simultaneously (PJF got it twice)

Rejected without reviewRejected without review

Multiple Submissions: what’s the problem? ““Let’s allow it! Whom does it hurt?”Let’s allow it! Whom does it hurt?”

Reviewers - burden increasesReviewers - burden increases Prospective copyright holdersProspective copyright holders ReviewersReviewers ReviewersReviewers

Conclusions

Recall the goalsRecall the goals Sensitivity to the review process for your Sensitivity to the review process for your

and its benefitand its benefit Understanding citation expectations for Understanding citation expectations for

professional writingsprofessional writings Understanding legalities of copyrightUnderstanding legalities of copyright Make you more comfortable with the Make you more comfortable with the

“trivia” associated with writing“trivia” associated with writing

Conclusions

Writing is difficultWriting is difficult Reviewing is difficultReviewing is difficult Organizing journal issues and conferences Organizing journal issues and conferences

is difficultis difficult What makes it worthwhile is the knowledge What makes it worthwhile is the knowledge

gain they promotegain they promote You are expected to support thatYou are expected to support that