Clinical Research Informatics: Challenges, Opportunities ...pmpathway.wustl.edu/files/2016/08/Embi_JAMIA_2009.pdf · OriginalJAMIA Investigations Research Paper Clinical Research

316 Embi and Payne, Clinical Research Informatics

JAMIAOriginal Investigations

Research Paper �

Clinical Research Informatics: Challenges, Opportunities andDefinition for an Emerging Domain

PETER J. EMBI, MD, MS, PHILIP R.O. PAYNE, PHD

A b s t r a c t Objectives: Clinical Research Informatics, an emerging sub-domain of Biomedical Informatics, iscurrently not well defined. A formal description of CRI including major challenges and opportunities is needed todirect progress in the field.

Design: Given the early stage of CRI knowledge and activity, we engaged in a series of qualitative studies withkey stakeholders and opinion leaders to determine the range of challenges and opportunities facing CRI. Thesephases employed complimentary methods to triangulate upon our findings.

Measurements: Study phases included: 1) a group interview with key stakeholders, 2) an email follow-up surveywith a larger group of self-identified CRI professionals, and 3) validation of our results via electronic peer-debriefing and member-checking with a group of CRI-related opinion leaders. Data were collected, transcribed,and organized for formal, independent content analyses by experienced qualitative investigators, followed by aniterative process to identify emergent categorizations and thematic descriptions of the data.

Results: We identified a range of challenges and opportunities facing the CRI domain. These included 13 distinctthemes spanning academic, practical, and organizational aspects of CRI. These findings also informed thedevelopment of a formal definition of CRI and supported further representations that illustrate areas of emphasiscritical to advancing the domain.

Conclusions: CRI has emerged as a distinct discipline that faces multiple challenges and opportunities. Thefindings presented summarize those challenges and opportunities and provide a framework that should helpinform next steps to advance this important new discipline.

� J Am Med Inform Assoc. 2009;16:316–327. DOI 10.1197/jamia.M3005.

IntroductionClinical research is critical to the advancement of medicalscience and public health. Conducting such research is acomplex, resource intensive endeavor comprised of a mul-titude of actors, workflows, processes, and informationresources. Ongoing large-scale efforts have explicitly fo-cused on increasing the clinical research capacity of thebiomedical sector and have served to increase attention on

Affiliations of the authors: Department of Medicine and Center forHealth Informatics, University of Cincinnati (PJE), Cincinnati, OH;Department of Biomedical Informatics and Center for Clinical andTranslational Science, The Ohio State University (PROP), Colum-bus, OH.Both authors contributed equally to the preparation of this manu-script. The authors acknowledge the contributions of those whoparticipated in our face-to-face session at the 2006 AMIA annualsymposium and of members of the AMIA CRI working group whoparticipated in phases of this research. In particular, the authorsthank the following individuals listed alphabetically for their addi-tional comments and other contributions to aspects of the prepara-tion of this manuscript: Barbara Alving, MD; Suzanne Bakken,DNSc, RN; Charles Barr, MD, MPH; Tara Borlawsky, MA; AmarChahal, MD, MBA; Christopher Chute, MD, Dr.P.H.; Milton Corn,MD; Don Detmer, MD; Bill Hersh, MD; Charles Jaffe, MD, PhD;
Stephen Johnson, PhD; Srini Kalluri; Stan Kaufman, MD; Rebecca
clinical research and related biomedical informatics activi-ties throughout the governmental, academic, and privatesectors.1–8 Such programs and initiatives have served assignificant catalysts for the emergence of a new sub-disci-pline of biomedical informatics focused on clinical researchreferred to as Clinical Research Informatics (CRI). The CRIspace is growing rapidly and has already enabled significantimprovements in the quality and efficiency of clinical re-

Kush, PhD; Judith Logan, MD, MS; Daniel R. Masys, MD; ShawnMurphy, MD, PhD; Ricardo Pietroban, MD, PhD, MBA; and IdaSim, MD, PhD; Justin Starren, MD, PhD.

Preliminary analysis and findings of phase one of this study werepresented at the AMIA 2007 Annual Symposium and published inthe AMIA 2007 Annual Symposium Proceedings.

Dr. Embi’s efforts in this research were supported in part by grants fromthe NIH/NLM (K22-LM008534, R01-LM009533). Dr. Payne’s efforts inthis research were supported in part by grants from the NIH/NCI(P01-CA081534, R01CA134232) and NIH/NCRR (U54-RR024384).

Correspondence: Peter J. Embi, MD, MS, Center for Health Infor-matics, University of Cincinnati Academic Health Center, 231Albert Sabin Way, PO Box 670840, Cincinnati, OH, 45267-0840;e-mail: �[email protected]�.

Received for review: 09/17/08; accepted for publication 02/12/09.

mailto:[email protected]

Journal of the American Medical Informatics Association Volume 16 Number 3 May / June 2009 317

search.5,9 As CRI emerges as a highly valued area of activity,it is imperative that those working in and concerned withadvancements in this space share an understanding of itsscope and the range of challenges and opportunities facingthe domain.

BackgroundClinical ResearchThe United States NIH defines clinical research as10:

“The range of studies and trials in human subjects that fallinto the three subcategories:

1. Patient-oriented research. Research conducted with hu-man subjects (or on material of human origin such astissues, specimens and cognitive phenomena) for whichan investigator (or colleague) directly interacts with hu-man subjects. Patient-oriented research includes: (a) mech-anisms of human disease, (b) therapeutic interventions, (c)clinical trials, or (d) development of new technologies.

2. Epidemiologic and behavioral studies.3. Outcomes research and health services research.”

Several studies have illustrated that a lack of sufficient ITinfrastructure and tools, as well as a reliance on workflowsdefined by historical precedence rather than optimal opera-tional strategies, account for significant impediments to theexpedient, effective, and resource-efficient conduct of clini-cal research activities.5 The rapid pace of biomedical scienceand the need for advances in medicine demand that theconduct of clinical research be timely, efficient, and yieldhigh quality results.4,11 As a result, the importance ofmaking clinical care data available for the secondary use insupport of clinical research has become a competitive re-quirement for clinical and research enterprises.11,12 More-over, the increasing complexity of clinical research and thechallenges of regulatory requirements associated with con-ducting clinical studies have led to further changes in theclinical research landscape, including a trend towards con-ducting clinical trials in community practice settings, asopposed to the historical norm of conducting such studies inlarge Academic Health Centers (AHCs).11,13 The rapidlyevolving and expansive clinical research landscape has ledstakeholders in the clinical research environment to ac-knowledge and call for system-level solutions.11,14

Consequently, clinical research is itself a domain in transition.Clinical researchers are faced with significant and increasinglycomplex workflow and information management challenges.The domain is also increasingly in the forefront of attention forthe governmental, academic, and private sectors, all of whomhave significant scientific and financial interests in the conductand outcomes of clinical research efforts. Due to the precedingcharacteristics of the clinical research environment and therecognition that effective and efficient information access iscritical to any solution to the many challenges faced by thedomain, there has been a corresponding and rapid evolution ofthe biomedical informatics methods and tools specifically de-signed to address clinical research information managementrequirements.

Clinical Research InformaticsThe evolution described above has led to efforts focused atthe intersection of biomedical informatics and clinical re-
search and the emergence of a domain that has become
known as CRI.15 In addition to the reasons already noted,part of this evolution of CRI can be attributed to theextraordinary increase in the scope and pace of clinical andtranslational science advancements that have been catalyzedof late by the emergence of major funding programs such asthose of the National Institute of Health’s (NIH) Road mapinitiative.16 A major goal of the NIH Road map involvesprograms to fundamentally re-engineer the way in whichorganizations translate basic science discoveries into practi-cable therapies.4,16,17 One such program is the Clinical andTranslational Science Award (CTSA) program which isintended to transform the manner in which AHCs conductand support clinical and translational science.3,4,18 As part ofthe national consortium of CTSA sites, efforts to coordinateand develop a variety of CRI-focused development effortsincluding data warehousing, clinical trials management andparticipant recruitment systems, collaborative team sciencetools, and integrative data “pipelining” and semantic har-monization platforms are underway.18

Beyond the CTSA program, there are several other impor-tant efforts that are representative of CRI domain activities.Examples include 1) the NCI’s Cancer Biomedical Informat-ics Grid (caBIG) initiative1,2,7,8; 2) Various CRI focusedstandards and harmonization bodies such as the ClinicalData Interchange Standards Consortium (CDISC), HealthLevel 7 (HL7), and the Biomedical Research IntegratedDomain Group (BRIDG)19–22; 3) The creation and growth ofclinical trial data registries;23–25 and 4) other NIH Road mapinitiatives such as the “re-engineering the clinical researchenterprise” program which preceded the CTSA initiativeand spawned several research projects that focused onevaluating the fundamental workflow and information man-agement needs of the clinical research domain.13,16,26,27

Additionally, the past five to ten years has seen the emer-gence of a growing body of literature describing centralchallenges to the national clinical research enterprise and thecorresponding benefits that could result from addressingthose challenges through, in part, effective integration ofbiomedical informatics and clinical/translational researchworkflows.5,6,9

Motivation for StudyAs is evident from the preceding overview of the clinicalresearch and CRI domains, the escalating level of activity inboth areas is leading to fundamental changes in the ways thatbiomedical investigators and research-related organizationsapproach the conduct of clinical research and the related role ofbiomedical informatics therein. At the same time, our review ofthe available literature has also demonstrated that there remainsignificant gaps in our collective definition and understandingof the emerging domain of CRI that impede its development asa cohesive and distinct discipline. Such gaps are manifest inpart by the lack of a comprehensive and community-acceptedset of fundamental and driving scientific questions concerningCRI itself and a corresponding inventory of the ways in whichmultiple, often overlapping, research and development pro-grams and efforts might contribute the answers to such ques-tions. These gaps in knowledge and the need to informadvancements in the highly critical CRI domain motivated this
study.


MethodsGiven the early stage of knowledge in this domain, weemployed a qualitative methodologic approach to identifythe range of challenges and opportunities that face the CRIdomain as well as defining its scope. Multiple phases wereemployed to approach the pertinent issues from variousperspectives and achieve triangulation of findings.28 Theparticular methods and output associated with each phaseare illustrated at a high-level in Fig 1, and described ingreater detail in the sections that follow. The University ofCincinnati’s Institutional Review Board (IRB) approved thisstudy.

Phase (1) Identify Community-perceivedChallenges and Opportunities in CRI

Data CollectionWe conducted a semi-structured, expert moderator facili-tated group discussion during the 2006 AMIA annual sym-posium. As described in greater detail in the manuscriptpresenting a preliminary analysis of this phase’s findings,15

the session took place in a hotel meeting room on Nov 14,2006, during the final 30 minutes of the Clinical ResearchInformatics Working Group Business meeting, following aseries of presentations on various ongoing national andinternational CRI initiatives by expert panelists who thenparticipated in the discussion along with other attendees.

To encourage broad participation in advance of the session,participant recruitment involved posting invitations aboutthe planned session on multiple AMIA mailing lists as wellas posting flyers announcing the session around the meetingarea of the annual session hotel. Prior to starting the session,all prospective participants were notified of the plannedstudy and provided with an IRB-approved informationsheet describing the study methods and how any data beingcollected would be used for research purposes. Consent to
have their comments recorded and analyzed in a confiden-
tial manner was implied if they remained and participated,though participation was not required for those who re-mained to listen. Further, to avoid any sense of pressure toparticipate, all were informed that they would have anotheropportunity to participate after the session via the workinggroup’s e-mail list. The facilitated discussion session wasconducted by a researcher with experience conducting suchgroup interviews (PE) using a semi-structured interviewtechnique, and the discussion was audio-taped. This ap-proach was used to allow for the emergence of unantici-pated relevant issues while still ensuring coverage of topicsof anticipated importance based upon prior discussionswithin our working group.

The discussion session began with a brief reminder of theCRI focus and a mention of the purpose of the discussion.This was followed by open-ended questioning of the groupabout their views on three major issues: (1) the challengesfacing CRI; (2) the opportunities facing CRI; and (3) the rolethat AMIA could play in this domain. Open-ended fol-low-up questions were asked as needed to catalyze deeperdiscussion of issues that the investigators considered of highimportance in satisfying the aims of this study. Only whenthese techniques yielded little or no response were closed-ended questions used to further explore issues of potentialimportance as anticipated by the investigators before thesession. Microphones were provided to all participants toclearly capture each utterance. The audio-taped recordingwas professionally transcribed verbatim and the audio filewas converted to an MP3 to facilitate audio-review duringthe analysis phase by the investigators as needed for verifi-cation of the transcript. Field notes were manually recordedduring the session by an experienced investigator (PP).

Data AnalysisField notes and verbatim transcripts of the approximately

F i g u r e 1. Overview of the four-phase meth-odology used to develop a systematic under-standing of the definition, challenges and oppor-tunities inherent to clinical research informatics(CRI).

25-minute recording were analyzed independently by two


reviewers (PE, PP). A sentence or phrase in the transcriptsserved as a unit of analysis for coding purposes. Contentanalysis was performed using a grounded theory approachto identify emergent themes.29 Descriptions for each unit ofanalysis were based upon participants’ utterances wheneverpossible. After each researcher conducted his independentreview of the transcripts and notes, the investigators enteredan iterative process beginning with discussion of theirdescriptions of the data. This led to agreement on broadcategorizations that could be directly linked to the raw data.As a final step, the categorizations were organized intocommon themes.

Phase (2) Follow-up E-mail Discussionwith CRI Professionals

Data CollectionFollowing phase 1, we proceeded to collect additional feed-back from the broader CRI community by initiating adiscussion on the AMIA CRI Working Group e-mail list. Themembers of this group had an opportunity to respond to ane-mail message that requested their feedback to questionsphrased just as they were during the in-person discussiondescribed in Phase 1 above. Beginning on Feb 1, 2007, themembers of this group received three separate e-mail re-quests sent by the authors. These requests also included anIRB-approved notification that responses posted to the e-mail list would be analyzed for research purposes. Directresponses to the requests or responses to the postings ofother respondents were collected for a period of 2 months atwhich point data collection was concluded.

Data AnalysisAs with the data collected during Phase 1, the data collectedduring this phase were collated, and the investigators (PE,PP) first independently and subsequently collaborativelyperformed content analysis using the same approach de-scribed above for the data collected during Phase 1.

Phase (3) Validation of Preceding Challenges andOpportunities by Domain Experts andDevelopment of a Formal Definition of CRITo verify and validate the challenges and opportunitiesidentified in the prior study phase, we used a combination ofqualitative approaches referred to as member-checking andpeer-debriefing,28,30 as described below:

Data CollectionFor member-checking, a selected sample of participantsfrom the session conducted during Phase 1 who are widelyrecognized as opinion leaders in the biomedical informaticsand/or clinical research domain and who were willing toprovide us with additional feedback (n � 8) were asked tocomment upon whether the findings identified in the previ-ous phases and presented below were representative of thekey CRI issues being targeted by this study. For peer-debriefing, we identified additional opinion leaders in thefields of biomedical informatics and/or clinical research(n � 12) to comment on those findings. In total, we engaged20 domain experts in this phase.

As in Phase 1, the subjects in this phase represented theacademic, government, and industry sectors. First, we senteach subject the summary of our findings from Phase 2 and
asked them to review the findings and commenting upon
them while answering the following questions: (1) “Do thecurrent categories/themes capture the range of major chal-lenges and opportunities facing the CRI domain today? Ifnot, what do you feel is missing?”; and (2) “Are there anyother additions/changes you would recommend?” Re-sponses from these subjects were returned via e-mail to thestudy investigators.

In a separate, second e-mailing, we sent this same group ofdomain experts a preliminary version of a definition of CRIand an accompanying list of exemplary CRI focus areas.These materials were initially developed by members of theAMIA CRI Working Group, including the authors, and theywere subsequently modified based upon our thematic find-ings and insights collected during the preceding phasesbefore delivery to our subjects. The domain experts wereasked to critique and comment upon the definition and listof CRI focus areas in an open-ended fashion and reply withtheir comments to the investigators.

Data AnalysisResponses to each of the two intervention steps conductedduring this phase were aggregated separately for qualitativeanalysis conducted using the same techniques employed inPhases 1 and 2. For the data collected during the firstfindings-validation step of this phase, the investigatorsfocused particularly on identifying any new or differentfindings that might justify modification of the results ofprior phases. For the data collected during the seconddefinition-critique step of this phase, the investigators eachfocused on identifying the range of comments made andmodifying the definition and focus-area list as appropriate.

Phase (4) Aggregation and Summary Analyses ofFindings from the Preceding PhasesFor the last phase of our study, we aggregated the primaryfindings from the preceding phases for a final, summaryreview and analysis stage. Specifically, we took note ofissues that were identified by more than one methodologyand particularly attempted to identify any new overarchingthemes or new organizational representations through whichthe findings’ meaning could be augmented. After independentreview of the findings and development of any new the-matic coding and reorganization, we discussed our findings,resolving any discrepancies, and consolidated our individ-ual findings from this phase into a common set of emergentitems, summary interpretations, and aggregated presenta-tion that arose from the cumulative findings.

ResultsIn the following subsections, we summarize the findings ofour multiphase study, including the issues, challenges andopportunities identified, and present them organized by thephases in which they were generated.

Phase (1) Identify Community-perceivedChallenges and Opportunities in CRIForty-six people were in attendance at the facilitated discus-sion and given the opportunity to contribute to the discussion,and 22 actively participated. To maintain the confidentiality ofthis relatively small group of participants, detailed demo-graphics cannot be presented. However, we can report thatparticipants represented a broad range of backgrounds and
perspectives including: academic informaticians, clinical and


translational investigators, governmental funding agency rep-resentatives, leaders of major health and research informaticsinitiatives and professional organizations, pharmaceutical andhealth IT industry professionals, and health IT investors. Allparticipants were directly involved in biomedical informatics-related and/or clinical research-related activities as a majorfocus of their jobs. Participants responded spontaneously to theopen-ended questions posed, with little need for the inter-viewer to prompt responses via the use of follow-up questions.As such, there was minimal need for closed-ended follow-up todiscussion points. In total, 3 pages of field notes and 7 single-spaced pages of verbatim transcript of the approximately25-minute recording were analyzed by two reviewers (PE, PP)using the methods described earlier.

Analysis of the data yielded numerous distinct categories ofcomments related to challenges, concerns, and opportunitiesfacing CRI. From these categories, twelve themes emergedthat together encompass all comments made during thediscussion. While some data elements could be assigned tounique categories, many represented more than one conceptand were therefore cross-categorized. For example, if acomment addressed the need for education about datastandards, it was coded as relating to both education andstandards. The 12 broad themes included: research planningand conduct; data access and integration; educational needs;fiscal and administrative issues; policy issues; leadershipand coordination; recruitment issues; scope of CRI; socio-technical issues; standards; workflow; lessons not learned.

Phase (2) Follow-up E-Mail Discussion withCRI ProfessionalsForty-three e-mail responses were received during thisstudy phase in response to investigator initiated messagesthat requested comments from AMIA CRI working groupmembers concerning their perceptions of prevailing chal-lenges and opportunities facing the clinical research infor-matics domain. Among these, 4 (10%) messages were directresponses to investigator prompts, while an additional 39(90%) messages were related to the initial prompts, but werenot direct responses to them. Thematic analyzes of thesemessages demonstrated that they spanned a complete spec-trum of topic areas, including all major themes described inthe prior study phases. Notably, no new categories orthemes arose during this phase that were not alreadycaptured during the Phase 1 study. The most commonthemes under which findings from these messages werecategorized were 1) the need for leadership and coordina-tion of CRI research and development efforts; 2) the need forincreased CRI education for informaticians and researchers;3) the need for more widespread adoption of data encodingor interchange standards; and 4) the pervasive nature ofsocio-organizational factors that prevent the ready adop-tion of CRI focused IT platforms.

Phase (3) Validation of Preceding Challenges andOpportunities by Domain Experts, andDevelopment of a Formal Definition of CRIAmong the 20 participants in this phase of the study, 10agreed that the initial list captured the range of issues, whileanother 10 participants commented how the current list ofthemes and categories could be expanded, clarified, or reorga-
nized. Analysis of these comments led us to augment and
modify our preliminary findings identified during Phase 1of our study and presented previously. While many of theitems expressed were broadly covered in existing categories,some felt that certain items were of sufficient importance torequire their own category, and there were a few items thatwere newly added.

Examples of items previously identified but deserving ofexpansion included: regulatory over-interpretation as animpediment to research conduct; challenges regarding notjust data access and integration, but also analysis of dataacross sites and across types of data (i.e., biological andclinical); the need to standardize workflow and reportingrequirements across sites and sponsors; and the need to haveCRI professionals involved in setting the CRI agenda atleadership levels within institutions and in regulatory/governmental agencies. In addition to these and other im-portant augmentations to our preliminary findings, newelements emerged during this validation process. Theselargely centered around the new theme we entitled “CRIinnovation and investigation”, a theme that included suchsubcategories as 1) “research in CRI is critical to advancingthe field”; 2) “research in CRI often is secondary to fulfillingcurrent needs”; 3) “there is a need for more funding toadvance CRI theory, not just practice”; and 4) the related“need to recognize CRI professionals’ efforts in promotionand tenure considerations.”

Therefore, this validation phase led to modification of someof our previously identified themes and categories as well asadding one new theme and related subcategories to the setof findings identified. The resultant 13 identified and vali-dated themes along with their constituent categories andexample quotations are summarized in Table 1. Narrativedescriptions of these final 13 emergent themes derived fromthese three phases are summarized here:

• Research planning and conduct: it became clear frommost of our participants’ responses throughout all phasesof our study that explicit and effective connections be-tween CRI platforms and capabilities, and the planningor execution of clinical research efforts, were lacking ifnot nonexistent. This lack of coordination between infor-maticians and research investigators or staff was seen asa primary impediment to the realization of the potentialbenefits in terms of research productivity or capacityafforded by CRI methods and tools.

• Data access, integration, and analysis: Study partici-pants felt that a multitude of organizational, policy-based, and practical factors collectively made the abilityto access comprehensive and/or integrative data setsthroughout the clinical Research Spectrum, difficult toachieve. This issue was particularly evident in discus-sions surrounding the secondary use of clinical data insupport of research activities.

• Educational needs: Study participants identified a majorneed to educate informaticians, clinical research investi-gators/staff, and senior leadership concerning the theoryand practice of CRI. Such education was thought to benecessary to ensure appropriate expectation manage-ment; adoption/use of CRI related methods or tools; andthe allocation of appropriate resources to accomplish
organizational aims.


Table 1 y Summary Table of Emergent Themes, Their Underlying Categories, and Representative QuotationsThemes Underlying Categories Representative Quotations

1. Research planningand conduct

Improved research planning toolsAdvanced clinical research designTools cumbersome/poorly integratedNeed for advanced analysis tools/

methods

“. . . the issue is that the tools are cumbersome and poorly integrated with theworkflow of patient care, and so you can’t recruit either the investigators or thepatients.”

“(Investigators) over-promise because the work is too hard to accomplish with thetools they are given”

“Our computing power is great, yet we are basically using the calculatingcapability that was available to Fisher and his contemporary 80 yrs ago,without real appeal to the “knowledge level.””

2. Data access,integration, andanalysis

Poor data accessLack of data integrationSecondary use of data issuesIncorporate research into clinical

systemsNeed improved analysis across sites

“I think one of the largest challenges institutionally is data access and integration,and that’s the barrier for the investigator.”

“. . . the interface between the electronic medical record and the clinical research iscritical.”

“In the era of integration between clinical systems and clinical trials systems,there remain some interesting unsolved problems.”

3. Educational needs Educate students, investigators,clinicians about CRI

Educate those working in CRIEducate informaticians about CRINeed for cross-discipline educationEducate senior leadership about CRI

“. . . it might be worth considering what is the training required, because it is atremendous leap for a DBA to understand the regulations for running a clinicaltrial. It is a tremendous leap for an electronic health records guy. CIOs andacademic centers do not get it. No offense. This is a big leap here.”

“. . . (We should) educate some of our informatics colleagues . . . about theobstacles and roles and needs that we have in this research environment.”

4. Fiscal andadministrativeissues

Research billing challengesCosts of research softwareImproved business processes neededLack of incentives for adopting

research tools/need todemonstrate ROI for CRI solutions

“We are struggling with the appropriateness of tracking research costs andresearch charges and making sure that we are compliant with that and what isresearch and what is not. So, there is a whole other set of business processes onthe investigator’s side that need to be somehow linked and coordinated withprocesses on the study side”

“So one of the big things driving this is the complete lack of money or incentive tomove beyond paper.”

5. Regulatory andpolicy issues

International CRI activitiesRegulatory frameworksPolitical obstaclesRegulation mis/over-interpretation

as impediments to progressSecurity and privacy issues

“A broad issue is the international nature of clinical research, especially clinicaltrials; for example, clinical trials in developing countries where the informaticsinfrastructure and the regulatory and ethical oversight are sometimes not aswell developed—trials having to respond to a patchwork of national, regional,and international regulations and responsible agencies”

“. . . overly conservative or incorrect interpretations of a regulation can become aninadvertent impediment to clinical research”

6. Leadership andcoordination

Need for coordinated CRI agendaNeed for setting practical goalsNeed for coordination between

initiatives/among stakeholdersDesire for leadership/guidance from

professional organizationNeed for coordination between

regulators and CRI communityNeed to have CRI professionals

represented in institutionalleadership

No clear channel for CRI decisionmaking within academicinstitutions

“. . . (the part) AMIA can play a role in is keeping up with all of this . . . socreating this portal that allows people from this group to be able to contributeand to go and understand what is going on in this space.”

“(We need to encourage) open comment from all the different perspectives so weare hearing from the investigator, the vendor, the institution, the NCI and otherNIH institutes; it would be fabulous to get that input (about CRI initiatives).”

“. . . absent leadership from CRI sitting at the table in the Regent’s meeting and atthe Hospital board meeting and in the corridors of the Capitol, there is littlehope of the software developer curing the problem.”

“. . . no group across academic medical institutions fully ‘own’ the problem ofsolving clinical research informatics.”

7. Recruitment issues Ineffective subject recruitmentCurrent tools make recruitment

difficultLost opportunities to recruitInvestigator recruitment challengesNot maximizing existing clinical

information systems forrecruitment

“Recruitment is the single biggest challenge. The investigators always over-promise. They do feasibility assessment. They try to target as best they can butthere is under-performance that causes them to have to do rescues mid-waythrough. They have to close down sites, start up new sites. They cannot get agood handle on how to predict or estimate, or enhance and augmentrecruitment.”

“So the issue again rests on what can we do to make it easier so that cliniciansare interested in becoming investigators.”

8. Scope of CRI Recognize CRI is about more thaninformatics for clinical trials

Include support for nursing researchInclude research partners in agenda

“(We should) expand our vision from clinical trials to encompass all of clinicalresearch, because I think this is going to be critical as we move forward. It is notjust the trials. It is the outcomes, the biomarkers. It is the epidemiology studies.”

“Nurses use a wide range of research techniques, and I think we need to be
(e.g., pharma, government) investigating those as well.”

esearch


• Fiscal and administrative issues: Study participantsvoiced three primary concerns relative to the ability ofthe CRI community to overcome fiscal and adminis-trative barriers to the implementation and use ofinformatics platforms in the clinical research setting,namely: 1) the inability to overcome billing complianceissues (e.g., disambiguation of standard-of-care v. re-search related billable activities in an automated man-ner)—which are perceived as a primary motivator forthe adoption of CRI platforms in many large organi-zations; 2) a lack of consistent funding mechanisms forpersistent CRI infrastructure (e.g., beyond episodicgrant funding mechanisms); and 3) the absence ofdemonstrable return on investment (ROI) or alterna-tive fiscal models that may serve to justify the supportof CRI platforms at the organizational level.

• Regulatory and policy issues: respondents voiced theopinion that the current regulatory environment is con-fusing, misunderstood, and often contradictory. They

Table 1 y continued.Themes Underlying Categories

9. Socio-organizationalissues

Research/clinical missions notintegrated/coordinated

Inadequate/inappropriate resourceallocation, use, adoption

Poor stakeholder collaborationInappropriate expectations of

informatics versus IT groups

“Thar(r

10. Standards Need for CRI data standards, modelsApply clinical standards to researchNeed ways to span biological to

clinical ontologiesNeed to standardize nontechnical

institutional and sponsorrequirements

“. .aa

“. .ar

11. Workflow Integrate tools into workflowInefficiency of research processes,

need for effective modelsIRB (Institutional Review Board)/

regulatory challengesNeed to consider users needs

“. .“(C

p“W

at

12. CRI innovationand investigation

Research in CRI is critical toadvancing field

Research in CRI often secondary tofulfilling current/practical needs

CRI needs to focus on transformativeadvances rather than justsupporting current practice

Academic promotion process doesnot reward CRI practice

“Thaat

“thnCs

“As

13. Lessons notlearned

Concern about repeating clinicalinformatics mistakes

Need for best practicesUnder-use of current tools/best

practices

“. .tfmamtr

specifically stated that there is an essential conflict be-

tween the regulatory frameworks faced by the clinicaland translational sciences and public or governmentaldemand for safer or more effective therapies. This conflictexists due to the desire to systematically protect patientprivacy and confidentiality that often constrains theability to apply informatics techniques to develop andevaluate novel therapies or otherwise conduct clinicalresearch.

• Leadership and coordination: the study participantswere concerned about the current state of coordinationof current national, regional, and local-scale CRI re-search programs which appear to be focusing on thesame essential aims. Many participants felt that suchprograms included policy and fiscal barriers that pre-vented coordination—thus exacerbating this particularproblem.

• Recruitment issues: participants raised various concernsaround challenges to human subject recruitment forclinical research studies as well as challenges related to

Representative Quotations

some obvious technical challenges (to providing data integration andbut . . . one of the biggest challenges coming is actually integrating themission into the academic health care environment and ensuring that

l of) these barriers that we have to data and systems and use of them foris able to happen.”

rly representing medical concepts and the right terminology in order tonything down stream which includes real world data from insurancetronic medical records, databases, as well as other knowledge bases.”ors/pharma don’t have standards and make each center create thingsg to each sponsor separately, like invoices, budgets, data entry, SAEg, and subject tracking”

l complain about the time it takes for IRB review.”systems suffer from) a complete lack of integration between patient cares and clinical research.”ot of talk has been made about standards, (so) we can move data backh, what is actually happening is there’s no work processes that makepen.”

ity of such thinking and research (about transformational approachesry) in CRI means that information and technology management

s have been subservient to the current process. There has been little ofption and leapfrogging that would change the environment.”ship in CRI gets bogged down in “pet projects” because that is thef their funding, and disruptive theory and models (that help advancestead of being the raison d’être for their departments, actually is ratherr them.”

promotion committees do not require or reward data sharing or dataization efforts”

roblems and the hopeful solutions (being proposed) are so analogous toes that have swirled around the electronic health record here at AMIAlast 15 or 20 years or so, and I would be a little more comfortable ifas said about what went wrong with that. Because to start all overith the same good hopes and the same stuff about standards and we willftware and all the rest of it, it didn’t cut it the last time, so I am hopinglessens from the past will be incorporated into the work on the clinical(agenda).”

ere areccess),esearchremovaesearch

. propeccess and elec. sponsccordineportin

. we alurrentrocessehile a lnd forthat hap

e paucnd theodvancehe disrue leaderature oRI), in

cary focademictandard

. the phe issuor theore w

gain wake so

hat the

recruitment of site investigators for multicenter studies.


In particular, participants noted that while well validatedapproaches and associated informatics platforms de-signed to improve subject recruitment to clinical trialshave been developed, the ability to readily adopt anddeploy such tools and approaches in “real world” settingis impeded by both a lack of applicable standards and aset of policy-based barriers surrounding the use of clini-cal data to enable such applications.

• Scope of CRI: most of our study participants felt that thenature of CRI and its definition was still very ambiguousand often misunderstood. As an outgrowth of this find-ing, the definition of CRI presented in our results wasdeveloped and validated. Of interest in this context wasa concern raised by several respondents that the areas ofclinical research and clinical trials were often conflatedwhen discussing such definitions and scope, while in factclinical trials are a subset of the broader clinical researchparadigm.

• Socio-organizational issues: the topic of socio-organiza-tional issues, as raised by the study participants, spanneda spectrum from local policy-based objections; to theneed for greater integration between institutional clinical,research, and educational mission areas, to the lack ofsufficient resources as needed to implement robust CRIresearch, development, and support efforts. The mostcritical theme identified surrounding the topic of sociotech-nical issues was the fact that technical barriers are not thesole issue that CRI researchers and practitioners must dealwith to be successful in their field, and that overcomingincompatible and often times misinformed sociotechnicalbarriers and objections to informatics interventions in theclinical research domain was as important if not moreimportant than technical hurdles.

• Standards: Study participants felt that the adoption ofappropriate data coding and representation standards inthe CRI domain was greatly impeded by both the largenumber of competing standards available to select from,as well as the absence of appropriate standards or ap-proaches capable of supporting information exchangebetween the basic science, clinical research, and clinicalpractice domains.

• Workflow: Study respondents indicated that they feltmany if not most CRI platforms suffered from a lack ofintegration with existing or optimized clinical researchworkflows, thus impeding their adoption. Respondentsrepeatedly called for an increased focus on the develop-ment and validation of workflow optimization and infor-matics intervention strategies for the clinical researchdomain. In particular, these types of workflow issuesarose in the context of the intersection of clinical researchactivities and clinical care.

• CRI innovation and investigation: many of our studyparticipants felt that a community-accepted understand-ing of the essential or high priority scientific problemsand research foci needed to advance the discipline of CRIwas lacking. There were clear and repeated calls for anemphasis in the CRI community on “research-on-re-search” to advance the discipline. Participants also feltthat greater coordination across national research effortsand funding agencies was needed to catalyze and sup-port the definition of key CRI research problems and
corresponding research agendas.
• Lessons not learned: The last thematic finding pertainsto multiple comments by study participants that therewere similarities between the preceding 12 thematic areasand challenges or opportunities that had been facedduring the evolution of the clinical, public health, andbio-informatics subdisciplines. The consensus opinion ofthe respondents was that we should ensure the CRIcommunity takes heed of the lessons generated by suchsimilar experiences in other biomedical informatics com-munities, rather than replicate their experiences.

Defining Clinical Research InformaticsAs mentioned to above, a key finding identified throughoutour study phases was that members of the biomedicalinformatics community felt that a formal definition of CRIwas both necessary and lacking. Among the 20 participantsin this phase of the study that we asked to review andcomment upon our definition of CRI, 4 agreed that thedefinition was adequate, while 16 had suggestions for im-proving it. Most of these centered around the insertion ofadditional key words to capture the activities engaged in byCRI professionals as well as helpful semantic clarifications.While many also grappled with the fact that CRI clearlyoverlaps considerably with the closely associated field oftranslational research informatics (TRI), the general consen-sus was that this definition need not necessarily capture andrepresent the entire clinical and translational research infor-matics spectrum by combining CRI and TRI into a singledefinition. It was also concluded that this definition shouldbuild upon rather than attempt to redefine the fields ofbiomedical informatics and clinical research that have beenpreviously defined by others.10,31 Finally, respondents agreedthat an accompanying list of high-level representative CRIfocus areas was helpful to augment this definition and hadsome suggestions for augmenting the preliminary list pre-sented. The result is following definition for CRI:

Clinical Research Informatics (CRI) is the subdomain ofbiomedical informatics concerned with the development,application, and evaluation of theories, methods, and sys-tems to optimize the design and conduct of clinical researchand the analysis, interpretation, and dissemination of theinformation generated.

Augmenting this definition is the following non-exhaustive,but illustrative list of CRI focus areas and activities:

• evaluation and modeling of clinical and translationalresearch workflow

• social and behavioral studies involving clinical researchprofessionals and participants

• designing optimal human-computer interaction modelsfor clinical research applications

• improving and evaluating information capture and dataflow in clinical research

• optimizing research site selection, investigator, and sub-ject recruitment

• knowledge engineering and standards development asapplied to clinical research

• facilitating and improving research reporting to regula-tory agencies

• enhancing clinical and research data mining, integration,
and analysis


• integrating research findings into individual and popu-lation level health care

• knowledge integration across clinical and research infor-mation systems

• defining and promoting ethical standards in CRI practice• educating researchers, informaticians, and organizational

leaders about CRI• driving public policy around clinical and translational

research informatics

Phase (4) Aggregation and Summary Analyses ofFindings from the Preceding PhasesBased upon our iterative assessment of the findings from thepreceding phases, it became clear that the 13 emergentthemes of CRI challenges and opportunities could be logi-cally organized and aggregated into higher-level themes. Itwas also clear that these overarching themes crossed stake-holder groups and appeared to fall into place along two-dimensional axes as follows.

First, the emergent higher-level themes involved three majorgroupings relevant to the advancement, practice, and lead-ership aspects of CRI. These included:

1. CRI academics and advancement, which encompassesthe themes of: educational needs; scope of CRI, CRIinnovation and Investigation.

2. Practice of CRI, which encompasses the themes of: re-search planning and conduct; data access, integration andanalysis; recruitment issues; workflow; standards.

3. Society and leadership aspects of CRI, which encom-passes the themes of: regulatory and policy issues; leader-

ship and coordination; socio-organizational issues; financialand administrative issues.

4. Lessons not learned arose as an overarching theme that wasrelevant to and spanned all the above higher-level themes.

Second, it was evident that these thematic groupings werelogically related to various stakeholder groups affected bythese findings that roughly spanned geographic scales from:

1. Individuals, including Researchers and IT/InformaticsProfessionals;

2. Organizational entities, including academic and privatesector institutions;

3. National and International entities, such as funders ofresearch and informatics, regulatory bodies, and govern-mental agencies.

A graphical representations of the themes and findingsreported above and organized by these higher-level the-matic and stakeholder dimensions is presented in Fig 2.

DiscussionThose involved and familiar with efforts at the intersectionof clinical research and biomedical informatics identify CRIas an important new subdiscipline that faces a range ofcritical challenges and opportunities. To our knowledge,these challenges and opportunities have not previously beenformally assessed or organized to illustrate the degree towhich they span the various stakeholder groups identifiedabove. Particularly at this relatively early stage in thedevelopment of a new domain, the availability of a system-atically and formally developed set of challenges, opportu-

F i g u r e 2. Overview of identi-fied themes organized into higher-level groupings by scope, and ap-plied across the groups of stake-holders to which they apply. Ofnote is the fundamental and cross-cutting nature of the frequently ar-ticulated theme labeled as Lessonsnot learned.


nities, and definition should help to facilitate further action,prioritize resourcing, and spur planning and agenda devel-opment to advance the field.

Defining CRIWhile presenting a formally vetted, community-deriveddefinition of the CRI domain is one part of this study’sresults that collectively describe the CRI domain, we feel thisoutcome is notable as it serves to delineate the domain’sscope and augment the list of major challenges and oppor-tunities identified. Of course, as with any effort to define acomplex domain, this definition will almost certainly notsuit everyone’s needs or desires. Indeed, even among ourvaried respondents, there were conflicting sentiments aboutthe right balance between generality and specificity for thedefinition. Another area of debate related to whether trans-lational research informatics (TRI), a closely related andsimilarly emergent discipline practiced by many who focuson the full spectrum of clinical and translational researchinformatics, should be captured under the definition of CRI.While it was our conclusion that, just as clinical and trans-lational research are distinct but closely related domains, sotoo are CRI and TRI, it is also clear that multiple intersec-tions exist between CRI, TRI, and other informatics andresearch-related subdisciplines that may support the argu-ment that they are somewhat inextricable from each other.

By examining the relationships between CRI and other areas ofbiomedical informatics domains of knowledge and practice asdepicted in Fig 3, we can better understand how informaticssubdomains concerned with aspects of basic and early trans-lational science (e.g., Bioinformatics), clinical practice (e.g.,clinical informatics), and population health (e.g., public healthinformatics) overlap with and complement CRI. For instance,an overlap exists between CRI and the related but distinct fieldof Translational Bioinformatics32 on the T1 end and withclinical and public health informatics that exists at the T2 end ofthe translational research spectrum.6 We feel that this illustra-tion also serves to complement the definition of CRI presentedabove by addressing some of the “fuzzy” boundaries that exist
between the complementary subdomains of biomedical infor-
matics, and which were noted by subjects in all phases of ourstudy.

Implications of FindingsBeyond definitional issues, an examination of the categori-zations and themes that emerged from this study is alsosignificant in that it enumerates major issues facing CRI,issues that have broad implications for the domain. Whenthese findings are considered within the current state ofbiomedical informatics knowledge and practice, severaladditional points and suggestions for action arise as partic-ularly noteworthy:

• While CRI is a unique, emergent subdiscipline ofbiomedical informatics, it is also exceptionally cross-cutting in nature. As alluded to in the discussion above,CRI, at least as much and perhaps more than otherinformatics subdomains, draws upon or contributes toother informatics domains by its very nature. Indeed, asa subdiscipline concerned with critical aspects of thebroad translational science spectrum, CRI necessarilyrelates to and advances through coordination and cross-fertilization of theories and methods with related subdo-mains spanning the basic and clinical aspects of biomedicalinformatics.

• CRI is not simply concerned with IT platforms for theclinical research setting and should not be thought ofthat way by those involved in clinical research orbiomedical informatics enterprises. Unfortunately, asvoiced by many of our concerned participants, this percep-tion is the norm in many settings. It is the perspective of theauthors that biomedical informatics and IT can be situatedin a translational spectrum. At one end of that spectrum, thescientific pursuit of biomedical informatics is concernedwith the creation and evaluation of novel theories, frame-works, models, and practices as needed to address essentialbiomedical information management needs, while at theother end of the same spectrum, IT is concerned with theoperationalization and support of such theories, frame-

F i g u r e 3. Illustration of types of researchacross which CRI is focused, and the relation-ships between CRI and the other sub-domainsof translational bioinformatics, clinical infor-matics, and public health informatics. Theserelationships also parallel the focus areas andmethodologies associated with the clinical andtranslational science paradigm, including thecommonly referred to T1 and T2 blocks intranslational capacity (where the T1 block isconcerned with impediments to the translationof basic science discoveries into clinical studies,and the T2 block with the translation of clinicalresearch findings into community practice).

works, models, and practices in “real world” or “produc-


tion” settings. It is in this continuum from theory to practicethat CRI can be placed firmly and justifiably in the biomed-ical informatics end of the spectrum.

• There is a need for greater community building in theCRI, and more generally, the clinical and translationalscience domains. Our study participants frequently raisedconcerns about lack of coordination and outreach betweeninformatics subdomains, as well as across potentially com-plementary but partitioned national programs like CTSAand caBIG, thus the need for community-wide coordina-tion and cohesion to drive advances in the domain.Furthermore, the limited membership of the CTSA con-sortium may in effect serve to create a large “silo” withinthe informatics and research communities that couldinadvertently impede advancement of the CRI domainthrough unintended but natural fragmentation of effortsif such a development is not guarded against. Therefore,it is the conclusion of the authors that significant effortsneed to be undertaken to ensure effective and productiveinteraction between those working to advance CRI, par-ticularly but not exclusively between CTSA and non-CTSA sites and investigators.

• Biomedical informaticians who focus on CRI should beaware of and apply the historical lessons learned fromthe clinical informatics community. As was voiced byseveral our study participants, there are strong corollar-ies to be drawn between the formative and evolutionarystate of CRI at this writing, and the similar experience ofthe clinical or medical informatics community in the past.Such a perspective is also reflected in the publication byAsh and colleagues concerning people and organiza-tional issues in the context of research information sys-tems.14 The study participants who raised this issueargued that, if we are to learn from the experiences of theclinical informatics community, the two most criticallessons to be applied in the CRI setting are that: 1) just asa single solution was often not the answer for clinicalinformatics needs, it is likely the case that a single,one-size-fits-all solution is not the right answer for clin-ical research; and 2) the continuous identification andinvolvement of key stakeholders from all organizationlevels is critical to the success of any informatics effort.

Next Steps for the CRI CommunityBuilding on the above implications and the primary findingsand analyses of our study, several next steps emerge as callsto action for the CRI community to address to achievefurther cohesion and advancement of the discipline:

• There is a critical need for the development of targetedpolicy and/or research agendas, driven by the CRICommunity, to inform the structure and function ofresearch programs and funding mechanisms at all lev-els. There are many stakeholders with interests in theadvancement of CRI, but, as our findings make evident,there is concern that the policies and research initiativesthat have helped to drive the CRI agenda have often beenparts of other primary initiatives. While helpful, there iswidespread concern that such efforts are often too nar-rowly focused (e.g., only on technical development andinfrastructure to support existing workflows), are fre-quently under-resourced, and often result in uneven
development of products and tools at the expense of
more fundamental CRI knowledge, best-practices, andexpertise. With a growing group of informaticians fo-cused on the CRI space, there is a critical need for thisCRI community to drive a strategic policy and researchagenda to help avoid haphazard and ad hoc solutionsthat while well-intentioned may result in further frag-mentation and worsening of existing challenges.

• Trusted and widely accessible mechanisms for thesharing of knowledge, best practices, and technologiestargeting the theory and application of CRI must beestablished. The CRI community is concerned that all toooften knowledge about CRI innovations, best practices,and solutions are maintained in inaccessible silos. Whilethere are many reasons including a lack of incentives forsharing resources and knowledge, one potential solutionis the creation and maintenance of trusted mechanismsfor sharing such knowledge, insights and resourcesamong the CRI Community. It is critical that such solu-tions not only be widely accessible, but that they extendbeyond the technical products of CRI, given the impor-tance of theory and methods to achieving success acrossCRI challenges.

• Greater advocacy for the rationalization of regulatoryframeworks with clinical research informatics as a keycomponent for consideration alongside clinical andpublic health concerns must be established. Few issueselicited more concern from our respondents than thoserelated to regulatory challenges. While there was a clearunderstanding of, appreciation for, and even championingof the need for strong regulations surrounding such issuesas information privacy and data reporting, there was also aconcern that such regulations have often been developedwith little or no consideration for the implications they mayhave for the clinical research enterprise. The result of theseregulations has at times been to impair the researchenterprise’s ability to advance discovery, thereby leadingto costly delays and failures that our healthcare sectorcannot afford to sustain.33 Here again, there is an urgentneed for involvement of the CRI community to informthe rationalization of regulations and policies, particu-larly related to personal and population level health data,so that research information needs and considerations aretaken into account. Such efforts must be undertaken withthe goal of assuring privacy and safety while not inadver-tently impairing our ability to advance the very scientificadvancements that will ultimately lead to improvements inhealth.

• Professional organizations, such as AMIA, should playa role to help overcome certain CRI challenges andcatalyze the development of broad CRI solutions bybringing together key stakeholders and serving as impar-tial, coordinating entities. While independent groups, or-ganizations and agencies have and certainly can continue toachieve significant advances through independent or par-tially coordinated efforts geared at addressing some of theCRI challenges identified previously, these efforts have andwill often continue to be limited by the nature of the groupssponsoring them. What is necessary to address many of thechallenges identified in the context of advancing CRI as aprofessional discipline is a venue for discussion, coordina-
tion and collaboration surrounding existing and new initi-


atives involving the full range of stakeholders interested inthe CRI agenda.

LimitationsThere are several limitations to note related to the workpresented in this manuscript in addition to those alreadymentioned above. While the researchers’ status as partici-pant-observers of the CRI-community provide them withinsights into the domain, we acknowledge that this can alsoinadvertently introduce some biases into our qualitativeanalyzes. In addition, the self-selected and limited conve-nience sample of participants, particularly in phases one andtwo of our studies, may not represent the composition of thebroader CRI community. It is also possible that even withour multiphase, multimethod approach, we did not captureall the challenges and opportunities facing the CRI domain.Despite such limitations, we feel that our use of qualitativemultiexpert validation techniques mitigate these shortcom-ings and enhance the validity of our findings.

ConclusionsClinical Research Informatics (CRI) has emerged as a distinctsubdiscipline of biomedical informatics and one that whilestill maturing is faced with many challenges and opportu-nities. We believe that the findings presented in this reportprovide empiric support for advancing the field of CRIbeyond its current, nascent state toward a more maturediscipline by providing the contextual underpinnings thatcan inform a cohesive and systematic cycle of research,development, and evaluation. By engaging in such a transi-tion at this juncture, the CRI community has the opportunityto avoid the roadblocks and impediments experienced byother informatics subdisciplines during their own, similar,formative stages. It is our hope that this grounded scoping ofthe CRI space as well as cataloguing of the challenges andopportunities facing the CRI domain will serve to catalyzethe pace and scale of CRI advancement, thereby enablingimprovements in human health through advances in clinicaland translational science.

References y

1. The cancer biomedical informatics grid (caBIG): Infrastructureand applications for a worldwide research community. StudHealth Technol Inform 2007;129(1):330–4.

2. Kakazu KK, Cheung LW, Lynne W. The cancer biomedicalinformatics grid (caBIG): Pioneering an expansive network ofinformation and tools for collaborative cancer research. HawaiiMed J 2004 September;63(9):273–5.

3. Zerhouni EA. Translational and clinical science—Time for anew vision. N Engl J Med 2005 Oct 13;353(15):1621–3.

4. Zerhouni EA. Clinical research at a crossroads: The NIH road-map. J Investig Med 2006 May;54(4):171–3.

5. Payne PR, Johnson SB, Starren JB, Tilson HH, Dowdy D.Breaking the translational barriers: The value of integratingbiomedical informatics and translational research. J InvestigMed 2005 May;53(4):192–200.

6. Sung NS, Crowley WF, Jr, Genel M, et al. Central challengesfacing the national clinical research enterprise. J Am Med Assoc2003 Mar 12;289(10):1278–87.

7. Oster S, Langella S, Hastings S, et al. caGrid 1.0: an enterpriseGrid infrastructure for biomedical research. J Am Med InformAssoc 2008 Mar–Apr;15(2):138–49.

8. Saltz J, Oster S, Hastings S, et al. CaGrid: Design and imple-mentation of the Core Architecture of the Cancer Biomedical
Informatics Grid, BioInformatics, 2006 Aug 1;22(15):1910–16.
9. Chung TK, Kukafka R, Johnson SB. Reengineering clinicalresearch with informatics. J Investig Med 2006 September;54(6):327–33.

10. NIH Director’s Panel on Clinical Research Report. Bethesda,MD: National Institutes of Health; 1997.

11. Research Rewired: Merging Care and Research Information toImprove Knowledge Discovery, Price Waterhouse Coopers, 2008.

12. Westfall JM, Mold J, Fagnan L. Practice-based research—“BlueHighways” on the NIH roadmap. J Am Med Assoc 2007 Jan24;297(4):403–6.

13. Khan SA, Kukafka R, Payne PR, Bigger JT, Johnson SB. A day inthe life of a clinical research coordinator: Observations fromcommunity practice settings. Stud Health Technol Inform 2007;129(1):247–51.

14. Ash JS, Anderson NR, Tarczy-Hornoch P. People and Organi-zational issues in research systems implementation. J Am MedInform Assoc. 2008 May–Jun;15(3):283–9. Epub 2008 Feb 28.

15. Embi PJ, Payne PR, Kaufman SE, Logan JR, Barr CE. IdentifyingChallenges and Opportunities in Clinical Research Informatics:Analysis of a Facilitated Discussion at the 2006, Annual: AMIASymposium. AMIA Annu Symp Proc 2007:221–5.

16. Zerhouni E. Medicine. The NIH Roadmap, Science, 2003 Oct3;302(5642):63–72.

17. Zerhouni EA. US biomedical research: Basic, translational, andclinical sciences. J Am Med Assoc 2005 September 21;294(11):1352–8.

18. CTSA. Clinical and translational science awards, 2008. Availableat: http://www.ctsaweb.org/. Accessed 9/1/08.

19. Fridsma DB, Evans J, Hastak S, Mead CN. The BRIDG project: Atechnical report. J Am Med Inform Assoc 2008 Mar–Apr;15(2):130–7.

20. Kush RD, Helton E, Rockhold FW, Hardison CD. Electronichealth records, medical research, and the tower of Babel. N EnglJ Med 2008 Apr 17;358(16):1738–40.

21. Richesson RL, Krischer J. Data standards in clinical research:Gaps, overlaps, challenges and future directions. J Am MedInform Assoc 2007 Nov–Dec;14(6):687–96.

22. Souza T, Kush R, Evans JP. Global clinical data interchangestandards are here! Drug Discov Today 2007 Febr;12(3–4):174–81.

23. Ghersi D, Clarke M, Berlin J, et al. Reporting the findings of clinicaltrials: A discussion paper. Bull WHO 2008 Jun;86(6):492–3.

24. Sim I. Trial registration for public trust: Making the case formedical devices. J Gen Intern Med 2008 Jan;23 Suppl 1:64–8.

25. Sim I, Chan AW, Gulmezoglu AM, Evans T, Pang T. Clinicaltrial registration: Transparency is the watchword. Lancet 2006May 20;367(9523):1631–3.

26. Boyd AD, Hunscher DA, Kramer AJ, et al. The “Honest Broker”method of integrating interdisciplinary research data. AMIAAnnu Symp Proc 2005;902.

27. Khan SA, Payne PR, Johnson SB, Bigger JT, Kukafka R. Model-ing clinical trials workflow in community practice settings.AMIA Annu Symp Proc 2006:419–23.

28. Patton MQ. Qualitative Research and Evaluation Methods, 3rd

ed, Thousand Oaks, CA: Sage, 2002.29. Crabtree BF, Miller WL. Doing Qualitative Research, Newbury

Park: Sage, 1992.30. Morse JM, Barrett M, Mayan M, Olson K, Spiers J. Verification

strategies for establishing reliability and validity in qualitativeresearch. Int J Qualit Meth 2002;1(2).

31. Shorliffe EH, Cimino JJ. Biomedical Informatics: ComputerApplications in Health Care and Biomedicine, 3rd ed, New York,NY: Springer, 2005.

32. Butte AJ. Translational BioInformatics: Coming of age. J AmMed Inform Assoc 2008 Nov–Dec;15(6):709–14.

33. Ness RB. Influence of the HIPAA privacy rule on health
research. J Am Med Assoc 2007 Nov 14;298(18):2164–70.
http://www.ctsaweb.org/

Documents

Clinical Research Informatics: Challenges, Opportunities ...pmpathway.wustl.edu/files/2016/08/Embi_JAMIA_2009.pdf · OriginalJAMIA Investigations Research Paper Clinical Research