Upload
alex-holcombe
View
425
Download
5
Embed Size (px)
Citation preview
Macquarie University 4 Dec 2015
A new era of large-scale replication and collaboration?
@ceptional
http://www.slideshare.net/holcombea/
• Open access
• P-hacking
• Preregistration, Registered Reports
• Open data, open methods
• Registered Replication Reports
Macquarie University 4 Dec 2015 2
Macquarie University 4 Dec 2015 3
Open access - Research funders will mandate it. Will we scholars shape the system of the future, or passively let publishers do it in their own interest?
• Green or Gold
• Who pays?
• Institution, granting agency, library (from subscription savings)
• Everything technically adequate should be published
• “contribution to the field” is subjective
• Competition for Nature on down wasteful, distorting
• Frontiers? PeerJ? Ubiquity Press?
Macquarie University 4 Dec 2015 4
Our average base Article Processing Charge (APC) is only £300 (€375, $500) and this is the only thing we charge.
5Slide courtesy Chris Chambers (Cardiff)
Science publishing has an incentive problem
Macquarie University 4 Dec 2015
"This one's Psych Science. 1998. Put your money on ‘will replicate’?”
"What, you think I was born yesterday?”
"Fine," muttered the bookie.
#AcademicNoir
Macquarie University 4 Dec 2015
P-hackingSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-
positive psychology undisclosed flexibility in data collection and
analysis allows presenting anything as significant. Psychological
Science
Macquarie University 4 Dec 2015
False positive zones - researcher degrees of freedom high (p-hacking)
- victory declared whenever null hypothesis is rejected, where size of effect, plausibility considered unimportant.
Where science is not self-correcting
The newer the domain, the more cutting-edge the technology, the less-defined the expected analyses, and the greater the opportunities for p-hacking
Macquarie University 4 Dec 2015
courtesy Chris Chambers (Cardiff)
Macquarie University 4 Dec 2015
Well, Mr. Hacker, your luck is about to run out. Down at the journals, they’re running a new game. They call it ‘preregistration’.
CC-BY Alex Holcombehttps://twitter.com/ceptional/status/663850868396044288
Macquarie University 4 Dec 201511
courtesy Chris Chambers (Cardiff)
Macquarie University 4 Dec 201512
courtesy Chris Chambers (Cardiff)
Macquarie University 4 Dec 201513
https://osf.io/tvyxz/wiki/home/
Macquarie University 4 Dec 201514
https://osf.io/tvyxz/wiki/home/
Macquarie University 4 Dec 201515
https://osf.io/tvyxz/wiki/home/
100% P-hacking FreeHere, check our numbers.
Here’s how you can replicate our result.
The badges that are fixing science
Macquarie University 4 Dec 2015
"This one's Psych Science. 1998. Put your money on ‘will replicate’?”
"What, you think I was born yesterday?”
"Fine," muttered the bookie.
Macquarie University 4 Dec 2015 17
Failed to replicate my result?
Have you considered
the unknown moderators?
Macquarie University 4 Dec 2015 18
Any one of us alone, he’ll say we messed up. But if we all run the replication together, we
can show the big guy we’re right.
Macquarie University 4 Dec 2015
Meta-analysis forest plot
Macquarie University 4 Dec 2015
Meta-analysis forest plot
Failed to replicate my result?
Have you considered the unknown moderators?
You messed up!
Macquarie University 4 Dec 2015
Macquarie University 4 Dec 2015
Macquarie University 4 Dec 2015
Macquarie University 4 Dec 2015
Registered studiesLarge-scale replicationOpen dataOpen protocol and code
A new era of large-scale replication
and collaboration?
Macquarie University 4 Dec 2015