Laurie Goodman at #CSE2014: Reproducibility: It's going to cost you time and effort, but...

Reproducibility

Laurie Goodman, PhDEditor-in-Chief, GigaScience

Laurie@gigasciencejournal.com

Mostly Time & Effort

It’s Going to Cost You

But It’s Our Job

Growing Issue: increasing number of retractions>15X increase in last decade

Strong correlation of “retraction index” with higher impact factor

1. Science publishing: The trouble with retractions http://www.nature.com/news/2011/111005/full/478026a.html2. Retracted Science and the Retraction Index ▿ http://iai.asm.org/content/79/10/3855.abstract?

At current % increase by 2045 as many papers published as retracted!

Problem: Growing Replication Gap

1. Ioannidis et al., (2009). Repeatability of published microarray gene expression analyses. Nature Genetics 41: 142. Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8)

Out of 18 microarray papers, resultsfrom 10 could not be reproduced

The problems with publishing

• Scholarly articles are merely advertisement of scholarship . The actual scholarly artefacts, i.e. the data and computational methods, which support the scholarship, remain largely inaccessible --- Jon B. Buckheit and David L. Donoho, WaveLab and reproducible research, 1995

• Core scientific statements or assertions are intertwined and hidden in the conventional scholarly narratives

• Lack of transparency, lack of credit for anything other than “regular” dead tree publication

GigaSolution: deconstructing the paperNeed to credit and reward:

• Data/software availability

• Metadata/curation

• Interoperability

• Availability of workflows

• Transparent analyses

Data

Metadata

Methods

Analyses

Every ComponentHas a Citable DOI

How are we supporting reproducibility?

Data Sets inGigaDB

Analyses inGigaGalaxy

Paper inGigaScience

Linked to

Linked to

Open-access journal Data Publishing Platform

Data Analysis Platform

Example in Neuroscience

• Neuroscience Data are not typically shared

• Data AND Tools are not typically made available to the reviewers

• Journal Editors think Reviewers will not want to review data

GigaScience 2014, 3:3 doi:10.1186/2047-217X-3-3

Example in Neuroscience• Neuroscience Data are not typically shared• Author Dr. Stephen Eglen said: “One way of encouraging neuroscientists to

share their data is to provide some form of academic credit.”• We hosted with a DOI: 366 recordings from 12 electrophysiology datasets• GigaDB is included in Thompson Reuters Data Citation Index • Data AND Tools are not typically made available to the reviewers• We made manuscript, data and tools all available to the reviewers.• We make sure to include reviewers who are able to properly assess the data

itself and rerun the tools • To reduce burdens- we sometimes select a reviewer who ONLY looks at the

data.• Journal Editors think Reviewers will not want to review data• What Reviewer Dr. Thomas Wachtler said: “The paper by Eglen and

colleagues is a shining example of openness in that it enables replicating the results almost as easily as by pressing a button.”

• What Reviewer Dr. Christophe Pouzat said: “In addition to making the presented research trustworthy, the reproducible research paradigm definitely makes the reviewers job more fun!”

Reviewer Comments on the Process

http://www.biomedcentral.com/biome/christophe-pouzat-and-thomas-wachtler-on-reproducible-research-in-neuroscience/http://blogs.biomedcentral.com/gigablog/2014/04/16/qa-on-dynamic-documents/

Reviewer Take Home Message

http://www.biomedcentral.com/biome/christophe-pouzat-and-thomas-wachtler-on-reproducible-research-in-neuroscience/http://blogs.biomedcentral.com/gigablog/2014/04/16/qa-on-dynamic-documents/

But This Can’t Be Done For Wet Bench!!!

And Make All Data AvailableTake steps to put all the images and raw data in a repository,

and to use biobanking where possible

Address Insufficient-Method Syndrome:-Complete, Easily found, Searchable, and Updatable

Lessons learned:• It is possible to recreate a result from a paper

• Reproducibility is COSTLY. How much are you willing to spend?

• Learn a huge amount about the study, and provides lots of information not present in the paper

• Much easier to do this before rather than after publication

It’s Time to Move Beyond Dead Trees

18121665 1869

Thanks to:

editorial@gigasciencejournal.comdatabase@gigasciencejournal.com

@gigascience

facebook.com/GigaScience

blogs.openaccesscentral.com/blogs/gigablog/

Contact us:

Scott Edmunds, Executive EditorNicole Nogoy, Commissioning EditorPeter Li, Lead Data ManagerChris Hunter, Lead BioCuratorRob Davidson, Data ScientistXiao (Jesse) Si Zhe, Database DeveloperAmye Kenall, Journal Development Manager

Follow us:

www.gigasciencejournal.comwww.gigadb.org