OPRE 2019 Methods Meeting Resource List

OPRE REPORT #2020-131

D. HANSEN AND R. HOLZWART

OCTOBER 2020

This document provides a list of resources for readers who wish to learn more about open

science methods. It was developed following the 2019 methods meeting, Methods for

Promoting Open Science in Social Policy Research, organized by the Office of Planning,

Research, and Evaluation (OPRE) in the U.S. Department of Health and Human Services’

Administration for Children and Families.1

1 See https:// opremethodsmeeting.org/ for additional information on OPRE’s methods meetings, including agendas and meeting products.

The resources were compiled from the meeting topic

memorandum, speakers’ slides, Q&A sessions at the meeting, and follow-up input from the

working group members who helped plan the event. They are divided into categories by topic

and can be accessed by clicking the blue hyperlink in each citation.

A. BACKGROUND INFORMATION

The resources in this section introduce the Federal initiatives and policies that support open

science, describe the motivation for promoting open science practices in social science

research, and provide an orientation to open science concepts.

Federal Initiatives and Policies Supporting Open Science

1. Evaluation Policy; Cooperative Research or Demonstration Projects, 79 F.R. 51574.

(2014).

2. Foundations of Evidence-based Policymaking Act of 2018, Pub. L. 115-435. (2019).

3. Holdren, J. P. (2013). Memorandum for the heads of executive departments and

agencies: Increasing access to the results of federally funded scientific research (p. 3).

Executive Office of the President, Office of Science and Technology Policy.

4. Vought, R. T. (2020). Memorandum for the heads of executive departments and

agencies: Phase 4 implementation of the Foundations of Evidence-based Policymaking

Act of 2018: Program evaluation standards and practices. Executive Office of the

President, Office of Management and Budget.

OPRE 2019 Methods Meeting Resource List 2

Motivation for Promoting Open Science in Social Science Research

5. Christensen, G. S., & Miguel, E. (2018). Transparency, reproducibility, and the credibility

of economics research. Journal of Economic Literature, 56(3), 920–980.

6. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries.

Scientometrics, 90(3), 891–904.

7. Funk, C., Hefferon, M., Kennedy, B., & Johnson, C. (2019). Trust and mistrust in

Americans’ views of scientific experts. Pew Research Center.

8. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of

questionable research practices with incentives for truth telling. Psychological Science,

23(5), 524–532.

9. Leek, J. T., & Peng, R. D. (2015). Statistics: P values are just the tip of the iceberg.

Nature, 520(7549), 612–612.

10. Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., & Wicherts, J. M.

(2016). The prevalence of statistical reporting errors in psychology (1985–2013).

Behavior Research Methods, 48(4), 1205–1226.

Orientation to Open Science Concepts

11. Bipartisan Policy Center. (2017). The promise of evidence-based policymaking.

12. Fecher, B., & Friesike, S. (2014). Open science: One term, five schools of thought. In S.

Bartling & S. Friesike (Eds.), Opening Science. Springer Nature.

13. National Academies of Sciences, Engineering, and Medicine. (2018). Open science by

design: Realizing a vision for 21st century research. National Academies Press.

14. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J.,

Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich,

E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M.,

Ishiyama, J., … Yarkoni, T. (2015). Promoting an open research culture: Author

guidelines for journals could help to promote transparency, openness, and reproducibility.

Science, 348(6242), 1422–1425.

15. Winerman, L. (2017). Trends report: Psychologists embrace open science. American

Psychological Association, 48(10), 90.

B. PRE-REGISTRATION

This section provides resources related to pre-registration, or the process of registering study

plans in a repository before beginning a research project. Pre-registering commits researchers

OPRE 2019 Methods Meeting Resource List 3

to a specific design, hypothesis, and/or data analysis plan. The table at the end of the section

contains links to registry websites and other practical tools.

1. Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and

possible improvements (NBER Working Paper 27250). National Bureau of Economic

Research.

2. de Groot, A. D. (2014). The meaning of “significance” for different types of research

[translated and annotated by E.-J. Wagenmakers, D. Borsboom, J. Verhagen, R. Kievit,

M. Bakker, A. Cramer, D. Matzke, D. Mellenbergh, & H. LJ. van der Maas]. Acta

Psychologica, 148, 188–194.

3. Duflo, E., Banerjee, A., Finkelstein, A., Katz, L. F., Olken, B. A., & Sautmann, A. (2020). In

praise of moderation: Suggestions for the scope and use of pre-analysis plans for RCTs in

economics (NBER Working Paper 26993). National Bureau of Economic Research.

4. Kupferschmidt, K. (2018). More and more scientists are preregistering their studies.

Should you? Science.

5. Lane, S. P., & Hennes, E. P. (2018). Power struggles: Estimating sample size for

multilevel relationships research. Journal of Social and Personal Relationships, 35(1), 7–

31.

6. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration

revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606.

7. van 't Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A

discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.

8. Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019). Recommendations

for increasing the transparency of analysis of preexisting data sets. Advances in Methods

and Practices in Psychological Science, 2(3), 214–227.

Pre-registration Tools, Websites, and Registries

Organization Name and Link Type Topical Area Overview and Purpose

American Economic

Association (AEA)

AEA randomized

controlled trials

(RCTs) registry

Registry

Economics and

other social

sciences

● Free registry for RCTs

Center for Open

Science Pre-registration

Tools/

Website

General/Cross-

disciplinary

● Website compiles pre-registration

resources and support materials

(e.g., literature, teaching materials,

instructions, FAQs) for researchers

Center for Open

Science

Registered

reports

Tools/

Website

General/Cross-

disciplinary

● Website compiles resources on

Registered Reports, a publishing

format used by over 250 journals

that emphasizes peer review prior

to data collection

OPRE 2019 Methods Meeting Resource List 4

Organization Name and Link Type Topical Area Overview and Purpose

Evidence in

Governance and

Politics (EGAP)

Registration Registry General/Cross-

disciplinary

● Registry for EGAP, a research,

evaluation, and learning network,

hosted in collaboration with the

Center for Open Science to

increase the sharing of accurate

and transparent information on

research projects

National Institutes of

Health, U.S. National

Library of Medicine

ClinicalTrials.gov Registry Medical

● Database of privately and publicly

funded clinical studies conducted

around the world on a wide range

of diseases and conditions

Office of Evaluation

Sciences (OES) Building evidence Registry

General/Cross-

disciplinary

● Searchable registry of more than 70

OES-supported evaluations on

various topics across the Federal

Government

OES OES evaluation

process

Tools/

Website

General/Cross-

disciplinary

● Website explaining OES evaluation

processes and policies and linking

to relevant resources and guidance

papers

Open Science

Framework (OSF)

Registration

forms and

templates

Tools/

Website

General/cross-

disciplinary

● Website providing sample

registration forms and templates for

general use

OSF Template: Pre-

registration-RR.rtf

Tools/

Website

General/cross-

disciplinary

● Guidance document explaining how

to build a registered report

U.S. Department of

Agriculture, Food

and Nutrition Service

(USDA FNS)

Study and

evaluation plans

Tools/

Website

Food and

nutrition

● Provides annual USDA FNS

research plans containing short

descriptions of projects expected to

be launched in each fiscal year

World Health

Organization (WHO)

International

clinical trial

registry platform

Registries Health ● Webpage links to various clinical

trial registries connected to WHO

C. REPRODUCIBILITY AND REPLICABILITY

This section provides resources related to reproducibility and replicability. Reproducibility is

defined as duplicating the results of a prior study using the same materials as were used by

the original investigator. By contrast, replicability is defined as duplicating the results of a prior

study following the same procedures but collecting new data.2

2 Bollen, K., Cacioppo, J. T., Kaplan, R. M., Krosnick, J. A., Olds, J. L., & Dean, H. (2015). Social, behavioral, and economic sciences

perspectives on robust and reliable science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National

Science Foundation Directorate for Social, Behavioral, and Economic Sciences.

The table at the end of the

section contains links to relevant websites.

OPRE 2019 Methods Meeting Resource List 5

1. Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M.,

Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen,

Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., Isaksson, S., … Hang, W. (2018). Evaluating the replicability of social science experiments in Nature and

Science between 2010 and 2015. Nature Human Behavior, 2, 637–644.

2. Errington, T. (2016). Reproducibility Project: Cancer biology - Barriers to replicability in

the process of research. OSF.

3. Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility

mean? Science Translational Medicine, 8(341), 341ps12-341ps12.

4. Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M.

C., & Lenne, R. L. (2018). Data availability, reusability, and analytic reproducibility:

Evaluating the impact of a mandatory open data policy at the journal Cognition. Royal

Society Open Science, 5(8), 180448.

5. LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A unified

framework to quantify the credibility of scientific findings. Advances in Methods and

Practices in Psychological Science, 1(3), 389–402.

6. National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and

replicability in science. National Academies Press.

7. Nosek, B. A., & Errington, T. M. (2020). What is replication? PLoS Biology, 18(3), 1–8.

8. Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., du Sert,

N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A

manifesto for reproducible science. Nature Human Behavior, 1, 1–9.

9. Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník,

S., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R.,

Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., ... Nosek, B. A.

(2018). Many analysts, one data set: Making transparent how variations in analytic

choices affect results. Advances in Methods and Practices in Psychological Science,

1(3), 337–356.

10. The Open Science Collaboration. (2015). Estimating the reproducibility of psychological

science. Science, 349(6251), aac4716-1-aac4716-8.

Reproducibility and Replicability Tools and Websites

Organization or Name and Link Type Topical Area Overview and Purpose

Author

Association for

Psychological

Science

Registered

Replication

Reports

Tools/

Website

Psychologica

l sciences

● Guidance and resources for

producing and reviewing Registered

Replication Reports, which are

multi-lab, high-quality replications of

psychological science experiments

OPRE 2019 Methods Meeting Resource List 6

Organization or Name and Link Type Topical Area Overview and Purpose

Author

Nuijten, M.

Checking

robustness in four

steps

Tools/

Website

Social

sciences

● Document provides guidance on

assessing and improving robustness

of psychological science (using

minimal resources)

Project Tier

The Documenting

Research in the

Empirical Social

Sciences (DRESS)

Protocol

Tools/

Website

Social

sciences

● Provides guidance and resources

for using the DRESS Protocol, a set

of standards for replication

documentation

Project Tier TIER Protocol

(version 3.0)

Tools/

Website

Social

sciences

● Provides guidance on implementing

the TIER Protocol, a research

project workflow that emphasizes

importance of replication

documentation throughout research

process

D. DATA MANAGEMENT AND SHARING

This section provides resources related to managing and sharing data. Data management

encompasses the organization, documentation, preservation, and accessibility of data. Good

data management is essential for high-quality research (Eynden, 2011), and data sharing is

crucial for reproducibility. The table at the end of the section contains links and tools specific to

codebooks.

1. Eynden, V. V. D., Corti, L., Woollard, L. B., & Horton, L. (2011). Managing and sharing

data: best practice for researchers. UK Data Archive, University of Essex.

2. Gentzkow, M., & Shapiro, J. M. (2014). Code and data for the social sciences: A

practitioner’s guide (Working paper).

3. Hardwicke, T. E., & Ioannidis, J. (2018). Populating the Data Ark: An attempt to retrieve,

preserve, and liberate data from the most highly-cited psychology and psychiatry articles.

PloS One, 13(8), e0201856.

4. Inter-university Consortium for Political and Social Research (ICPSR). (n.d.).

Recommended informed consent language for data sharing.

5. ICPSR. (2012). Guide to social science data preparation and archiving: Best practice

throughout the data life cycle (5th ed.).

6. ICPSR. (2012). Guidelines for effective data management plans.

7. Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., Kennett, C., Slowik, A., Sonnleitner,C., Hess-Holden, C., Errington, T.

M, Fiedler, S., & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple,

low-cost, effective method for increasing transparency. PLoS Biology, 14(5).

OPRE 2019 Methods Meeting Resource List 7

8. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of

psychological research data for reanalysis. American Psychologist, 61(7), 726–728.

Data Management and Sharing Tools, Websites, and Media

Organization Name and Link Type Topical Area Overview and Purpose

or Author

Amsterdam Public

Health Quality

Handbook

Codebook/Data

dictionary

Tools/

Website

Statistical

analyses for

public health

● Website and linked resources with

guidance on constructing high-

quality codebooks

Institute for Social

Science Research Codebook help

Tools/

Website

Social

sciences

● Website providing overview of

codebook components

Inter-university

Consortium for

Political and Social

Research (ICPSR)

Guide to

codebooks (1st

ed.)

Tools/

Website

Social

sciences

● Report providing guidance and links

to additional resources for

developing codebooks

ICPSR

Intro to data

management

plans

Video file Social

sciences

● Video of recorded presentation on

data management plans

Princeton University

Library

How to use a

codebook

Tools/

Website General

● Website with description and

examples of how to use a codebook,

with links to additional resources

E. SYSTEMATIC REVIEWS

This section provides background on systematic reviews and meta-analyses, including a

historical overview of these methods and information about current reporting standards.

Systematic review is a form of structured literature review that answers a question through the

analysis of all available evidence. It includes applying predetermined inclusion and exclusion

criteria to the literature, critically appraising the relevant literature, and extracting and

synthesizing data from the evidence base to formulate findings.3

3 National Information Center on Health Services Research and Health Care Technology. (2017). HTA 101: Glossary.

The table at the end of the

section contains links to tools and websites on systematic reviews; the text box contains

resources specific to the Institute of Education Sciences’ What Works Clearinghouse.

1. Agency for Healthcare Research and Quality. (2014). Methods guide for effectiveness

and comparative effectiveness reviews (AHRQ Publication 10(14)-EHC063-EF).

2. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The

APA publications and communications board task force report. American Psychologist,

73(1), 3–25.

3. Glass, G. (1976). Primary, secondary, and meta-analysis of research. Educational

Researcher, 5(10), 3–8.

OPRE 2019 Methods Meeting Resource List 8

4. Petrosino, A., Turpin‐Petrosino, C., Hollis‐Peel, M. E., & Lavenberg, J. G. (2013). 'Scared

Straight' and other juvenile awareness programs for preventing juvenile delinquency.

Cochrane Database of Systematic Reviews, 4.

5. Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter,

A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007). Development of AMSTAR: A

measurement tool to assess the methodological quality of systematic reviews. BMC

Medical Research Methodology, 7, 10.

6. Whiting, P., Savović, J., Higgins, J. P., Caldwell, D. M., Reeves, B. C., Shea, B., Davies,

P., Kleijnen, J., & Churchill, R. (2016). ROBIS: A new tool to assess risk of bias in

systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225–234.

Systematic Review Tools, Websites, and Registries

Organization or Name and Link Type Topical Area Overview and Purpose

Author

● Cochrane is a journal and

Higgins, J.,

Lasserson, T.,

Chandler, J., Tovey,

D., & Churchill, R.

Methodological

expectations of

Cochrane

Intervention

Reviews

Systematic

review tool Healthcare

database for systematic reviews in

healthcare. This report describes

methodological standards to

which all Cochrane protocols,

reviews, and updates are

expected to adhere

Joanna Briggs

Institute (JBI)

Checklist for

systematic

reviews and

research

syntheses

Systematic

review tool General

● JBI is an international,

membership-based research and

development organization; report

provides tools for adhering to its

systematic reviews

Preferred Reporting

Items for Systematic

Reviews and Meta-

Analyses (PRISMA)

PRISMA Website General

● Website with guidance and tools

for using PRISMA, an evidence-

based minimum set of items for

reporting in systematic reviews

and meta-analyses

Society for Research

on Educational

Effectiveness

Registry of

Efficacy and

Effectiveness

Studies

Registry Education

● Database of causal inference

studies in education and related

fields

National Institute for

Health Research

PROSPERO

international

prospective

register of

systematic

reviews

Registry Health

● International database of

prospectively registered

systematic reviews in range of

fields where there is a health-

related outcome

OPRE 2019 Methods Meeting Resource List 9

_ _

’ ’

-

Resources Specific to the What Works Clearinghouse

The What Works Clearinghouse (WWC) is an initiative of the Institute of Education Sciences

(IES). WWC reviews original research on education interventions. Links to helpful pages on

the WWC website and related resources follow:

Education Resources Information Center (ERIC) IES. (n.d.). Education resources information center. IES ERIC. (n.d.). FAQ - Peer review. IES WWC. (n.d.). Search publications. IES WWC. (n.d.). Find what works. IES WWC. (n.d.). Handbooks and other resources: Procedures and standards handbooks. IES WWC. (n.d.). Handbooks and other resources: Review protocols. IES WWC. (n.d.). Practice guides. IES WWC. (n.d.). WWC help desk. IES WWC. (n.d.). Using the WWC to find ESSA Tiers of evidence. IES. (2020). Standards for excellence in education research. IES. (2019). Request for applications.

Like OPRE

on Facebook

Facebook.com/OPRE.ACF

Follow OPRE

on Instagram

@OPRE ACF

Follow OPRE

on Twitter

@OPRE ACF

Sign up

for the OPRE

Newsletter

This brief was prepared by Insight Policy Research (1901 North Moore Street, Suite 1100, Arlington, VA 22209) under Contract Number HHSP233201500109I. The Administration for Children and Families Contracting Officer s Representatives are Emily Ball Jabbour and Kriti Jain. The Insight Project Director is Rachel Holzwart, and the Deputy Project Director is Hilary Wagner.

This brief is in the public domain. Permission to reproduce is not necessary. Suggested citation:

Hansen, D., & Holzwart, R. (2020). OPRE 2019 methods meeting resource list (OPRE Report 2020 131). Washington, DC: Office of Planning, Researchand Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

This brief and other reports sponsored by the Office of Planning, Research, and Evaluation are available at http://www.acf.hhs.gov/opre.

Disclaimer: The views expressed in this publication do not necessarily reflect the views or policies of the Office of Planning, Research, and Evaluation; the Administration for Children and Families; or the U.S. Department of Health and Human Services.