Embed Size (px)
Citation preview
Federico Bianchi1, Giangiacomo Bravo2, Lorena Cadavid3, Francisco Grimaldo4, and Flaminio Squazzoni1
1 GECS – Research Group on Experimental and Computational Sociology, University of Brescia (Italy)2 GECS – Research Group on Experimental and Computational Sociology, Linnaeus University (Sweden)3 Department of Computational and Decision Sciences, National University of Colombia4 Department of Computer Science, University of Valencia (Spain), LABSS – CNR (Italy)
Scientist strategies under the spotlight. A game theory-inspired simulation
model of peer review
PEERE Meeting, Valencia, Spain, March, 8-11
Scientists under constraints
PEERE Meeting, Valencia, Spain, March, 8-11
Research questions:
• Quality?• Bias?
The Peer Review Game
Repeated game with n agents.
Players are scientists who contribute to the general scientific effort by
– Submitting
– Reviewing
Each round scientists play both as “author” and “reviewer” (randomly paired)
Author’s aim is to obtaining a recommendation for publication (r)
At each game repetition both authors and reviewers can produce high (H) or low (L) quality.
PEERE Meeting, Valencia, Spain, March, 8-11
The Peer Review Game /2
Before each round, scientists allocate their resources (Ri) on submission and reviewing by setting level of submission effort (esi) and reviewing effort (eri).
This determines levels of submission quality (Qs
i) and reviewing quality (Qri).
Trade-off between submission and reviewing
PEERE Meeting, Valencia, Spain, March, 8-11
High vs. Low Quality
Ideal allocation of resources is
Ideal distribution of submission/reviewing quality
The Peer Review Game /3
PEERE Meeting, Valencia, Spain, March, 8-11
Resources are distributed at the beginning of each turn according to the ratio of each scientist’s contribution to the number of publications in the whole system
Publish or perish
PEERE Meeting, Valencia, Spain, March, 8-11
Scientists constantly increase their investment in submission effort
Random allocation of publications
No reviews
Blind cooperatorsScientists allocate resources in a fair way
Main aim is publishing investment in submission is constantly increased
When published, more resources are allocated on reviewing
Ideal situation: every scientist cooperates with the system by investing resources in the evaluation of others
PEERE Meeting, Valencia, Spain, March, 8-11
Blind punishersScientists strive for publication and constantly increase their submission effort as long as they are rewarded by the systemwith publication.
If they fail to publish, they relax effort in submission, thus increasing the investment in reviewing.
This resembles scientists who perceive failure as unfair and therefore punish peers by putting more effort in reviews.
PEERE Meeting, Valencia, Spain, March, 8-11
Non-blind cooperatorsScientists cooperate with the system as long as they perceive that the system works correctly.
Once a game round is over, they compare the quality of their submission with that of a sample of the published papers, and they estimate whether they actually deserved to be published or not.
Comparison with lowest quartile of published papers
vs.
comparison with top quartile of published papers
Impact of overestimation
PEERE Meeting, Valencia, Spain, March, 8-11
Non-blind punishers
Scientists strive for publication and constantly increase their submission effort as long as they are rewarded by the system with publication.
Once a game round is over, they compare the quality of their submission with that of a sample of the published papers, and they estimate whether they actually deserved to be published or not.
Comparison with lowest quartile of published papers
vs.
comparison with top quartile of published papers
Impact of overestimation
PEERE Meeting, Valencia, Spain, March, 8-11
Results
PEERE Meeting, Valencia, Spain, March, 8-11
Evaluation Bias = # good rejected papers / # papers ideally publishedReview Expenses = # reviewing resources / # submission resourcesTop Quality = Total submission quality of top 10 published papers
Preliminary conclusions
PEERE Meeting, Valencia, Spain, March, 8-11
• Too much reviewing by cooperative scientists might subtract resources for publication, thus generating overall poor performance, unless they strive for excellence.
• Scientists who strive for publication and invest in reviewing to outperform low quality science may secure the production of high quality at the top, by generating an efficient division of scientific labour.
