Big Data: Big Opportunities or Big Trouble?

BIG DATA BIG OPPORTUNITIES OR BIG TROUBLE?

Kathy Partin, Office of the VP for Research, Dept. of Biomedical Sciences

Shea Swauger, Libraries

What is Big Data?

• Volume• Variety• Velocity

• Too Big to Email

• Veracity• Variability• Visualization• Value

The Data Lifecycle

• Proposal• Infrastructure• Acquisition/Generation• Management• Dissemination• Preservation

Proposal

• Grant Funding Requirements

• Data Management Plan

http://lib.colostate.edu/repository/nsf

https://dmptool.org

Infrastructure

• Where do you store it?

• How do you move it?

• How do you analyze it? (HPC?)

+ Ultra High Speed Research LAN

+ College or Department Servers

+ Bioinformatics & other Clusters

http://istec.colostate.edu/activities/hpc/

Data Acquisition/Generation

Reuse Existing• Where to find it?

• How to understand/use it?

• Do you trust it?

• Create your own data

Metadata + README files

Data Provenance

Privacy, Security, Proprietary, Dual Use Research of Concern

Data Management

• Access/Permissions• File Naming• Metadata• Organization• Collaboration• Version Control• Fixity/Integrity

http://lib.colostate.edu/services/data-management

DisseminationWhere to share your data?

• Institutional Repository

• Discipline Specific Repository

How to cite your data?

• Permanent identifier (doi, handle, PURL, etc.)• Citation standards

http://lib.colostate.edu/services/data-management/citing-data

Data Preservation

• Media Obsolescence• Software Obsolescence• Bit Rot• Back-ups• Checksums

Public Outcry Regarding Data Integrity• “Why Most Published Research Findings are False”, Ioannidis, 2005• “Update of the Stroke Therapy Academic Industry Roundtable Preclinical

Recommendations,” Fisher et al., 2009• “Science Publishing: The Trouble with Retractions,” Van Noorden, 2011• “Believe it or not: how much can we rely on published data on potential drug

targets?” Prinz et al., 2011• “Misconduct Accounts for the Majority of Retracted Scientific Publications,”

Fang et al., 2012 • “Drug Development: Raise standards for Preclinical Cancer Research,”

Begley & Ellis, 2012

http://i97.photobucket.com/albums/l217/Shockwave_73/angry-mob-at-frankenstein-castle_zps364a2714.jpg

Integrity - Reliability - Translation• “Power Failure: why small sample size undermines the

reliability of neuroscience”, Button et al., 2013• “Challenges in Translating Academic Research into

Therapeutic Advancement,” Matos et al., 2013 (epilepsy)• “Reproducibility,” McNutt, 2014• “NIH plans to enhance reproducibility,” Collins & Tabak, 2014 • “Reproducibility: Fraud is not the big problem,” – Gunn, 2014• Taxpayers are wasting their investment because the

integrity of basic research is flawed, not due to intentional misconduct but to unintentional mismanagement.

Research Misconduct

1. Fabrication, falsification, plagiarism, or other practices that seriously deviate from those that are commonly accepted within the relevant scientific/academic community for proposing, conducting, reviewing or reporting research; that

2. Has been committed intentionally, knowingly or recklessly; and, that

3. Has been proven by a preponderance of the evidence (more likely than not)

Misconduct does not include honest error or honest differences in interpretations or judgments of data.

Reporting Concerns • All employees and individuals associated with CSU should report observed,

suspected or apparent Research Misconduct to their Department Head, Dean, the RIO and/or the Vice President for Research.

• If an individual is unsure whether a suspected incident falls within the definition of scientific misconduct, a call may be placed to one of these individuals to discuss the suspected misconduct informally.

http://reportinghotline.colostate.edu/

Research Integrity Officer› Primary contact for departments and deans with

questions about potential misconduct issues› Represents CSU with the PHS Office of Research

Integrity (ORI), NSF, USDA, etc› Manages the CSU MIS process to meet

institutional, state and federal standards› Kathy.Partin@colostate.edu

External Pressure to Fix or Be Fixed• Issues with data reliability have brought external pressure

on the scientific community• From Congress

• Presidential Council of Advisors on Science and Technology (PCAST) – “Improving Scientific Reproducibility in an Age of International Competition and Big Data” , 2014 http://www.tvworldwide.com/events/pcast/140131/

• From the popular press and “watch dog” websites/blogs• The Economist - “Unreliable research: Trouble at the Lab”, 2013• NYT– “New truths that only one can see”, 2014• RetractionWatch.com

The Gap Between Applied & Basic Research

InnovationReliability

The two opposite and contrary forces of data

Dynamic, agile, discovery, exploration, optimization, creative, outside-the-box, anti-dogmatic(pre pre-clinical study)

Reproducible, robust, translatable to bedside, rigid, immutable, non-optimized, boring(preclinical or clinical study)

What needs to change?• Funding agencies need to raise the bar for data

acquisition• Publishers need to raise the bar for data quality• Academic institutions need to reassess how success is

defined• Academic institutions need to provide their faculty with the

right tools and training to do it right• Faculty need to pass this down to their trainees

External Changes• NIH appears to be

• Developing a new training module on good experimental design to disseminate

• Developing a data checklist for grant proposals• DDI- Data Discovery Index• New biosketch format to reduce the focus on numbers of publications

and increase the focus on impact of publications• Considering blinded review of grant proposals

• Science Exchange Reproducibility Initiative

DDI“In summary, a Data Discovery Index (DDI) emphasizes development of an adaptable, scalable system through active community engagement that would serve as an index to large biomedical datasets.”

Rather than in a traditional “catalog” the DDI concept stresses discoverability, access, and citability.

This is a dataset of raw data, which rarely saw the light of day in academic research before.

Publishers• Preventing plagiarism with iThenticate• Preventing Fabrication/Falsification with new data checklists• Abolishing word limits on methods sections

Six Common Experimental Failings1. Poor experimental design2. Poor reagents3. Poor analysis 4. Failure to reject hypothesis after observing discordant,

valid experimental results5. Deliberate bias in selecting positive rather than

negative results to report, publish, cite, and fund6. Failure to follow through when wondering “Why is this

result NOT what I expected?”

Statistics & General Methods1. How was the sample size chosen to ensure adequate

power to detect a pre-specified effect size?2. Describe inclusion/exclusion criteria if samples, subjects

or animals were excluded from the analysis. Were the criteria pre-established?

3. If a method of randomization was used to determine how samples/subjects/animals were allocated to experimental groups and processed, describe it.

Statistics & General Methods4. If the investigator was blinded to the group allocation

during the experiment and/or when assessing the outcome, state the extent of blinding.

5. For every figure, are statistical tests justified as appropriate? Do the data meet the assumptions of the tests (e.g., normal distribution)?

a) Is there an estimate of variation within each group of data?b) Is the variance similar between the groups that are being statistically

compared?

Good Laboratory Practice for DataA Attributable (who made the entry)L LegibleC Contemporaneous/CompleteO OriginalA Accurate

http://www.paduiblog.com/pa-dui/why-forensic-science-testing-for-dui-bac-determination-is-the-silly-sister-of-analytical-science-good-laboratory-practices/

Data Notebooks – Another Vulnerability

• Binders• Electronic Notebooks• Software documentation• Field notes• Images• Algorithms

Data Corrections & Amendments• Errors, additions, and modifications should be identified

by crossing out the original data with a single line (do not obscure the initial data) and initialing, dating and providing a reason for the change.

• Missing or obscured data/pages are often interpreted as intentional obfuscation of data

•Absence is interpreted as guilt

Data Forensics• Numbers• Images• Hardware/Software

Numbers

Images

Hardware

Software

Questions?