The Importance of Good Clinical Data Management and Statistical

Programming Practices to Reproducible Research

Eileen C King, PhD Research Associate Professor, Biostatistics Acting Director, Data Management Center

Reproducible Research The term first proposed by Jon Claerbout at Stanford

University and refers to the idea that the ultimate

product of research is the paper along with the full

computational environment used to produce the

results in the paper such as the code, data, etc.

necessary for reproduction of the results and

building upon the research.

Why? • Scholarship can be recreated, better

understood and verified.

• Others can start from the current state of the art

• Simplifies task of comparing a new method to existing methods

• Create earlier results again in a later stage of the research

Reproducibility of Clinical Research is Cornerstone of Drug

Development Process • Phase III trials - Requirement for two

trials that produce similar results (reproducibility)

• Summary of Clinical Efficacy Document – Displays the reproducibility of the efficacy

results

Drug Development Submissions • Required to provide all data along with

complete documentation

• May be required to provide statistical analysis code along with complete documentation

Why? • Regulators (e.g. FDA) will recreate the

results in your submission

• Regulators will use the data to evaluate efficacy and safety issues across a drug class – cardiac effects from NSAIDS – effectiveness of antihistamines in OTC cold

products

Academic Research Centers should

steal shamelessly from the Drug Development Process in order to facilitate Reproducible

Research

Why? • NIH is requiring:

– Data Sharing Plan in large grants • Data and Documentation • Statistical Programming Code and Documentation

– Posting of clinical study results to

clinicaltrials.gov

Facilitating Reproducible Research

Develop Standard Operating Procedures (SOPs)

• SOPs should be written to cover all key elements of the conduct of a study

• Provide enough detail to ensure steps are consistently carried out

• Don’t provide so much detail as to end up with violations due to normal variations in working

• Written at a high level to outline: – Required tasks – Sign-offs – Checks performed

Develop Work Instructions • Work instructions can be more specific to

a particular division or study

• Can document in more specific detail how to do things

• Generally not formally audited by regulatory agencies on work instructions

Assemble Multi-Functional Teams

Assemble as soon as study planning is underway Team should provide input into the protocol regarding their functional area

Study Team

Data Management

Investigator

Clinical Operations

Statistical Programming

Statistics QA

Biomedical Informatics

Regulatory

Pharmaco-vigilance

Coordinator

Cannot Work in Functional Silos • Expertise is needed in each of these areas

from protocol development through manuscript submission

• Amount of effort varies dependent on stage of study

Documents Required for Reproducible Research

• Study Protocol

• Manual of Procedures (MOP)

• Data Management Plan (DMP)

• Statistical Analysis Plan (SAP)

Importance of Data Management for Reproducible

Research

“As its importance has grown, clinical data

management has changed from an

essentially clerical task in the late 1970s and

early 1980s to the highly computerized

specialty it is today”

Susanne Prokscha (2007): Practical Guide to Clinical Data Management

Society of Clinical Data Management (SCDM)

• Founded to advance the discipline of clinical data management (CDM)

• Organized exclusively for educational and scientific purposes

• Mission: Promoting clinical data management excellence including promotion of standards of good practice within clinical data management

• Provides certification: The CCDM® program establishes eligibility criteria and standards of knowledge as measured by a rigorous examination to qualified applicants.

Good Clinical Data Management Practices (GCDMP)

Charter: The review and approval of new pharmaceuticals by federal regulatory agencies is contingent upon a trust that the clinical trials data presented are of sufficient integrity to ensure confidence in the results and conclusions presented by the sponsor company. Important to obtaining that trust is adherence to quality standards and practices. To this same goal, companies must assure that all staff involved in the clinical development program are trained and qualified to perform those tasks for which they are responsible From SCDM GCDMP document

Certified Clinical Data Manager (CCDM)

• SCDM certification program was designed to meet the following goals: – Establish and promote professional practice standards

throughout CDM

– Identify qualified professionals within the profession

– Ensure recognition of expertise

– Enhance the credibility and image of the profession

Data Management Plan (DMP) • Details how data will be:

– Collected – Stored – Managed – Archived

• Describes and defines all data management

activities for a study – What – Who – When – How

Timing of DMP • Development begins after the protocol and

Case Report Forms are drafted

• Should be completed before the study begins

• Must be kept current – reflect important changes to the data

management process and computer systems that took place during the study

General DMP Contents

• Protocol Summary • Study

Personnel/Roles • CRF Design/Tracking • Database

Development • Data Entry and

Processing • Data Cleaning

• Reports • Audit Plans • External Data

Transfers • Managing Lab Data • SAE Handling • Coding • Training • Study Closeout

Benefits of DMP • Forces planning • Process and tasks become more visible to

the project team

• Expected documents are listed at the start of the study so they can be produced during the study

Benefits (Cont) • Provides continuity of process and a

history of a project

– Useful for long-term studies – Useful for personnel turnover and growing DM

groups

• Regulatory Requirement – Auditors will ask for it

DMP • Planning upfront saves time at end and

improves data quality

• First DMP is time-consuming to do

• Use templates and previous DMPs to ease upfront burden

Development of the Data Management Plan

is so important that the DMC is

recommending tracking of the percent of

studies for which a DMP was written prior to

first patient enrolled.

Selection of Database

21 CFR part 11 In March of 1997, FDA issued final part 11 regulations that provide criteria for acceptance by FDA, under certain circumstances, of electronic records, electronic signatures, and handwritten signatures executed to electronic records as equivalent to paper records and handwritten signatures executed on paper. These regulations, which apply to all FDA program areas, were intended to permit the widest possible use of electronic technology, compatible with FDA's responsibility to protect the public health.

Implications • If data are being captured in an electronic

system with no paper CRFs then should(must) use compliant system

• System can be compliant but user must follow standard procedures to assure total process is compliant

• At minimum, use system with audit trails

Current Options • REDCap

– Not yet

• Request for Proposals to purchase 21CFR part 11 compliant system – Oracle RDC – Medidata Rave – Study Manager – Omnicomm Trialmaster – TargetHealth

Estimated timing for purchase: Summer 2012 Estimated timing for installation: End of 2012

NOT Options • EXCEL spreadsheets

• Microsoft Access

Data Base Design • Investigators, Coordinators, Data

Managers, Biomedical Informatics, and Statisticians must work together

• Poor database design affects timings for: – Data Entry – Data cleaning – Extraction – Statistical Analysis

Things to do to quality of data in data base

• Enter data as soon as possible after it is received

• Run cleaning procedures throughout the study so that queries go out early

• Identify missing CRF pages and lab data by knowing what is expected. Use tracking systems

• Code AEs and meds frequently

Improving Quality (cont) • Reconcile SAEs periodically throughout the study

– Get listings from safety system early so you know what to expect

• Begin to audit data against the CRF early to detect systematic problems Continue to audit as the study proceeds to monitor quality.

• Open the study documentation at the start of the study and make an effort to keep it updated as the study progresses.

Data Standards Facilitate Reproducible Research

Value of standard modules in CRF Design

• Completion instructions are the same

• Database design is the same

• Edit checks are the same

• Associated statistical analysis programs are the same

• Easier to compare and combine data across studies

Developing Standards • Standards can include:

– Case Report Forms / Instructions – Variable names – Codelists

• Must be a commitment to following the

standards that are developed

Resource for Standards

• Clinical Data Interchange Standards Consortium (CDISC) – Global, open, multidisciplinary, non-profit

organization that has established standards to support the acquisition, exchange, submission and archive of clinical research data and metadata.

CDISC Mission

• To develop and support global, platform-independent data standards that enable information system interoperability to improve medical research and related areas of healthcare

• Standards are freely available via the CDISC website.

• Leading the development of standards for data collection is goal for Data Management Center

• Will need standards committee to develop and maintain which involves: – Clinical teams – Data Management – Statistics – Pharmacovigilance – Biomedical Informatics

Importance of Good Statistical Programming Practices to Reproducible Research

Statistical Programmers

• Work closely with Data Management to: – Specify format of data base that will be

received by statistics

– Assist with data cleaning by running analysis programs on live data base

• Benefit: – Statisticians don’t spend time cleaning data

Statistical Analysis Plan Includes: • Hypotheses to be tested

• Identification of primary and secondary endpoints

• Statistical analysis approach for each hypothesis

– Assumption testing – Alternative approaches – Data handling decisions

• Draft tables and figures for reports/manuscripts

Good Statistical Programming Practices

• Clear documentation contained within programming code

• Creation of Analysis Datasets – Ensures all programs are using a consistent set of

derived data.

• Validation of Statistical Programs – How: Second programming or code review – What: All analysis datasets and tables and figures

Tips to Facilitate Reproducibility

• Produce Report-Ready Output – RTF files cut and pasted into

manuscript/report – Authors need not edit the numerical output – Authors can change titles and footnotes

• Conduct quality assurance audits of

manuscripts/reports

Reproducibility (cont) • Programs for Tables and Graphs

– Archive SAS Programs, list and log files – All tables should contain a footnote with the

location and name of the SAS program that created it

• When report is complete, change permissions to read-only – All SAS programs, list and log files – Analysis datasets

Where Can I Get Help?

http://cctst.uc.edu/

Go to Research Central on Centerlink

CCTST: Center for Clinical and

Translational Science and Training

CCTST can identify resources for your study

Resources Available

• Clinical Trial Office

• Biomedical Informatics

• Data Management Center

• Biostatistical Consulting Units

• Drug Poison Information Center

Thank you!