Replication & Data Sharing: The Publication Perspective

Bradford W. Hesse, PhDChair-Elect, APA Publications & Communications Board

American Psychological Association

Friday, February 14, 14

APA Monitor: Feb 2013*

Complex Problem

• Bias toward positive findings

➡ “P-Hacking,” with false positives in literature

➡ The “file drawer” problem, with false negatives obscured

• Insufficient detail for method replication

• Lack of incentives to replicate

• Disincentives for data sharing

* Winerman, Lea. (2013). Interesting results: Can they be replicated? Monitor on Psychology, 44, 38-41.Friday, February 14, 14

Task Force on Data Sharing*Necessary for

verification

* Data sharing memo, April 4, 2012

Restriction in purpose: No reason for sharing other than “verification”

Burdensome, costs to be borne by replicating “professional”

Friday, February 14, 14

Trends Toward Open Science*

* e.g., Hesse, B. W., Croyle, R. T., & Buetow, K. H. (2011). Cyberinfrastructure and the biomedical sciences. Am J Prev Med, 40(5 Suppl 2), S97-102.

• Computer infrastructure• NSF, NIH requirements• Open Science mandates

• Emergence of “Big” problems• Emergence of “Big Data”• Team Science to solve problems

Friday, February 14, 14

2 Sides of Same Coin*Data Sharing

• Enables replication

• Promotes aggregation for knowledge synthesis, hypothesis generation, programmatic decisions, and generalizability testing

• Opens data for analysis with more powerful analytic techniques than possible originally

• Encourages a culture of openness in science

Replication

• Technical replication

➡ Method validation

➡ Verification

• Conceptual replication

➡ Theory building

➡ Generalizability

➡ Model testing

• Culture change

➡ Self-correcting

➡ Science as public trust

* Replication and Data Sharing (RADS) Task Force, October 2013

Friday, February 14, 14

* Replication and Data Sharing (RADS) Task Force, October 2013

Proposed Policy Revision*

Friday, February 14, 14

Consider: Article of the Future*

* http://www.articleofthefuture.com/about

Direct Access to Data

Interactive Content

Links to Author community (e.g., ORCID)

Ability to publish ePub only material

Friday, February 14, 14

APA’s Cyber-infrastructure*

• Credible and ubiquitous portal to psychological sciences

• Expanded scope promotes team science, with bidirectional links between data services (e.g., PubMed & PsycINFO)

• Direct access to bibliographic tools (e.g., *.pdf’s, cited references, reference software exportation) elevates scientific productivity, reduces friction to good scholarship

* Advised by Electronic Resources Advisory Committee (ERAC)

Friday, February 14, 14

Replication ePubs in all Journals*

• Subject to same, rigorous review as print pubs

• Not considered against impact factor

• May be highlighted by editor as important contribution to field

• May be open to online comments at some point in future

* Proposal by Council of Editors

online only

Friday, February 14, 14

APA’s Open Science

• Free and open to the public

• Standard, finely nuanced descriptions of study’s rationale, method, results, and interpretation

• Data collaboration

➡ data underlying analysis available

➡ data generators retain authorship rights

• Technical + Public abstracts

• Open comment discussions

Archives of Scientific Psychology*

* Courtesy Harris Cooper (Duke University) and Gary VandenBos (APA)

Friday, February 14, 14

Ten Prescriptions*Improve the transparency of research

1. Require research reports to include complete accounts of method, analyses and results

2. Require data sharing

3. Clarify rights and responsibilities of parties to data-sharing agreements

4. Construct a prospective database of IRB-approved research

*Cooper, Harris & VandenBos, Gary. (2013). Ten Prescriptions To Foster Better Social Science in the Internet Environment. Final Report on NIH Grant. Duke University. Durham, NC.

Friday, February 14, 14

Change the reward structure of research

5. Value replication

6. Train researchers in record keeping and data management

7. Use approaches to evaluations of research (and researchers) that promote transparent science

Ten Prescriptions*

*Cooper, Harris & VandenBos, Gary. (2013). Ten Prescriptions To Foster Better Social Science in the Internet Environment. Final Report on NIH Grant. Duke University. Durham, NC.

Friday, February 14, 14

Improve interaction with the global audience

8. Take responsibility for open access publication costs

9. Provide the means for interaction between researchers and an audience of non-researchers

10.Make special efforts to include members of the international community in training

Ten Prescriptions*

*Cooper, Harris & VandenBos, Gary. (2013). Ten Prescriptions To Foster Better Social Science in the Internet Environment. Final Report on NIH Grant. Duke University. Durham, NC.

Friday, February 14, 14

Task Force RecommendationsCoordinate with other stakeholders in psychological science

Work with APA’s Board of Scientific Affairs to engage Office of Human Subjects Protection & State IRB’s

Draft publishable document for Replication and Data Sharing Guidelines (similar to Journal Article Reporting Standards)

Friday, February 14, 14

Reinventing Discovery

The real question is: How do we use the methods and discoveries of the psychological sciences to solve problems of replicability,

generativity, & integration across all sciences?

Friday, February 14, 14