International Journal of Medical Informatics (2005) 74, 908916
Evaluation frameworks for nurs
a School ob New Yor
of inysteloyetivel ev
2005 Elsevier Ireland Ltd. All rights reserved.
Rigorousis increacare envhealthcamakingimplemeevaluatiothe potenhuman anof identievaluatioseveral fr[15,81slightly dform info
PresentYork, NY 10
1386-5056/doi:10.1016matics research methods
evaluation of informatics applicationssingly important in the current health-ironment. Effective evaluation can guidere Information Technology (IT) decision-related to system development andntation. In addition, effective iterativen during the development process hastial to avert system failure and thus saved nancial investments. Toward the goalfying appropriate selection and use ofn methods, researchers have put forwardameworks to guide the evaluation process4]. Each of these frameworks offers aifferent perspective from which to per-rmatics evaluation. Research frameworks
address: Mail Code 6, 630 West 168th Street, New032, USA.ddress: email@example.com.
published in the early 1990s describe the timing forquantitative and qualitative methods to be appliedwithin the system development life cycle (SDLC),with a primary focus on quantitative methods.More recently, however, evaluation frameworksaccentuate the need for qualitative research to bea critical portion of, and the only method used inthe evaluation process.
A recent review of informatics studies byAmmenwerth and Keizer  describes the mat-uration of healthcare informatics research fromsmall individual studies, to a more mature statemanifested by an increase in the number of sys-tematic reviews and publication in informaticsdomain-specic journals. The increased numberof systematic reviews demonstrates that there issufcient research in certain areas to conduct suchreviews. The increase in publications in informaticsdomain-specic journals indicates that the eld issufciently large to warrant several internationalinformatics publications. The Ammenwerth and
$ see front matter 2005 Elsevier Ireland Ltd. All rights reserved./j.ijmedinf.2005.07.007M. Curriea,b,
f Nursing, Columbia University, New York, NY, USAk Presbyterian Hospital, New York, NY, USA
Summary Rigorous evaluationtant so that the impact of such stitative methods have been empgrowing trend to utilized qualitamative phases of research. Severahighlight the need for both qualidevelopment and post-implementRecommendations regarding thefor each of these frameworks. Tof the published evaluation frammethods in use in each of these fing informatics
formatics applications in healthcare is impor-ms can be understood. Although many quan-d to evaluate informatics systems, there is amethods during both the formative and sum-aluation frameworks that have been proposede and quantitative evaluation methods in then phases of informatics systems development.g, type and use of qualitative methods differaper examines the strengths and weaknessesrks and enumerates the qualitative researchworks.
Evaluation frameworks for nursing informatics 909
Keizer review also noted an increase in the numbersof qualitative studies, particularly those examiningorganizational or socio-cultural factors , anindication of a shift in focus of informatics research.
Other work of note by Ammenwerth et al. reportson a Eurotion of stsystem evthe Declaprovidesof information. Thof compoand relatiaccompliscomponenmeasuringsystem. . .,be madetext lists twelvfor researthe availapractice abe part oable recomin advancdemonstramatics.
Friedmanof the ieffects onevaluationologies thcations, tfunction.tive or sumout duringwhereas scompletioevaluationmanner wbefore thmative reciated wi. Thesof programcomputerevaluationtem devel
The racurrent go
ical evaluation. The RCT process demonstrates ahigh level of scientic rigor because it maintainsstringent objectivity and controls for extraneouseffects. The RCT process involves the explicitdenition of quantitative variables and random
us cal gCTsionatic
opmm cprocthatle inoesargutureerna. Anr infRC
ativnicaalitarlyysted unssesunspahouiteraowssesis [isciemeuseermptiocultas e
ton inttionatiopean workshop geared toward identica-rategies to improve health informationaluation. This workshop culminated inration of Innsbruck in 2003 , whichdenitions of the two key componentsatics research, the system and evalu-e authors dene the system as a setnents. . ., together with their attributesonships, which as a whole is needed toh an objective.  The second keyt, evaluation, is dened as the act of. . .the properties of a health informationthe result of which informs a decision to
concerning that system in a specic con-. In addition, the Declaration of Innsbrucke recommendations including the needch to be grounded in scientic theory, forbility of guidelines for good evaluationnd the need for evaluation methods tof informatics curricula . These valu-mendations reect the general interest
ing methods in informatics research andte the maturation of the eld of infor-
and Wyatt dene evaluation as the studympact or effects [of software] or [its]users and the wider world . Thus,frameworks need to describe method-
at capture the processes integral to appli-he users and the world in which the usersEvaluation can be characterized as forma-mative. Formative evaluation is carriedthe development phase of the research,
ummative evaluation is carried out uponn of a project . During formative, processes are evaluated in an iterativehich provides feedback for improvementse nal product is put forth. During sum-search, the impact or the outcomes asso-th the use of the system are examinede terms tend to be used in the contextevaluation and are thus applicable to
system evaluation because informaticstakes place both during and after sys-
opment.ndomized controlled trial (RCT) is theld standard for informatics and biomed-
assignto therigoroimentthe Rconcluinformdevelperformentclaima singdrug dTheyby naan alt[6,13]ods foof theeffectdevelshould
Qualitfor clito quto eathat slimiteproceis notsincethrougfact,are nproceanalysmultidinvolving ofto Gdescriof subsuchabilityhumacollecinformt of groups. An intervention is deliveredperimental group(s), which provides foromparison between control and exper-roups. As with summative evaluation,process is typically carried out at theof the study . Critics of the RCT fors applications claim that the iterativeent process precludes evaluators toontinuous evaluation during the develop-ess rather than post hoc. Grant et al. the RCT is suitable for the evaluation of
tervention, such as a drug, only because anot change during the evaluation process.e that software system development isan evolutionary process which demandstive to the RCT evaluation methodologyincrease in the use of qualitative meth-ormatics evaluation reects the inabilityT to capture information required forsystem development during the formativeent process. Instead, an RCT can andused for summative evaluation.
tative evaluation methods
e methods have been effectively usedl informatics evaluation [2,3]. The movetive methods was largely in responseinformatics work which demonstratedm failures were related to developersderstanding of human factors and system. The phenomenon of system failureique to healthcare informatics and haswned the use of qualitative methodst the software development industry. Intive processes and user-centered designindustry standards, with ethnographicbeing one of the primary tools for needs19]. User-centered design refers to aplinary design approach based on activent of the user to improve the understand-r and task requirements . Accordingain, ethnography is the systematicn, analysis and interpretation of culturesural groups . Qualitative methods,thnography and focus groups, have thecapture experiences, emotions, and
eraction processes through the inductiveof and subsequent rigorous analysis ofn from the individuals perspective .
910 L.M. Currie
As such, these methods are appropriate for useduring the formative stage of system development.
Friedman and Wyatt, in their seminal text Eval-uation Methods in Medical Informatics, distinguishbetween objectivist and subjectivist approachesto evaluation by using an archetypal framework rstput forth by House in 1980 [3,23]. In their workthey use the term subjectivist to mean qualita-tive. However, subjectivism is also a philosophicaltheory that states that knowledge is generated asa subjective experience without any connection toreality or objectivity . As such, the term sub-jectivism cannot be equated with the term qual-itative research which, though emanating from asubjective perspective, is not subjectivism per se.Moehr criticizes Friedman and Wyatts apologeticintroduction of subjectivist research methodolo-gies in their text, as they describe subjectivistmethods as a different set of premises. . .that maybe. . . discomforting to some readers . Thesecautionary words emphasize the perspective thatqualitative research methods are inferior researchmethods. This perspective has been perpetuatedbecause of the value placed on the RCT, and as suchthe perception that qualitative research is inferiorbecause it lacks rigor. However, it is increasinglyevident th
Researchand reliabrefers tosurement
out that the quest for reliability can hinder qual-itative validity. She posits that by assuming thatreality is. . .singular and tangible, rather thanmultiple and constructed, the researcher mightobscure validity . In other words, if one adheresto rigid measurements, the inherent limitations ofpredened concepts might obscure the truth.
Qualitative researchers, aware of the perceivedinferiority of their methods, have developed strictprocesses to ensure rigor. Lincoln and Guba putforth qualitative concepts to reect the rigorof the qualitative process. These terms includecredibility, transferability, dependability andconrmability. Table 1 lists the concepts andthe methodological processes associated withmaintaining qualitative rigor. These processesinclude maintaining accurate audit trails, frequentrevisits with the subjects to ensure validity and,methodological triangulation . Although theseprocesses are neither objective nor quantiable,they are arguably equally rigorous . Despitethe potential distinction between the words sub-jectivist and qualitative, and the continueduse of the term subjectivist by Wyatt and Wyatt this paper will refer to all qualitative andpsychological methods as qualitative methods.
ordistut redevto mproativd ineing
Transfera r co
Dependab low d
Conrmaat this is not the case.
iability and validity in informaticsn
is considered rigorous if it is both validle. To quantitative researchers reliabilitythe extent to which the results of mea-are consistent . Sandelowski points
Acccan exwithonot aportsis theQualitsiderenow b
Terms used to describe rigor in qualitative paradigms
e concept Process to maintain rigor
y Vivid and faithful description of theProlonged engagementPersistent observationPeer debriengTriangulationMember checks
bility How well the ndings t into anotheTheoretical purposive samplingThick, descriptive data
ility Ability of another researcher to folat every stage of analysisAudit trailExternal audit
bility Data, interpretational conrmabilityTriangulationPracticing reexivitying to Waltz and associates, reliabilitywithout validity, but validity cannot existliability. Validity refers to whether orice or method. . . measures what it pur-easure . In other words, validity
ximity to the truth of a measurement.e methodologies, though historically con-ferior to quantitative methodologies, areseen as methods that might generate
nomenon Internal validity
ntext External validity
ecision trail Reliability
Evaluation frameworks for nursing informatics 911
a closer approximation of the truth. Qualitativeprocesses bring researchers closer to the truth of adomain via the subsequent rich and detailed anal-ysis of the human experience .
The evaluwere deveevaluationunique neuation bot
The aim oevaluationmatics rerecommening the ein researcreects thcal informAs such, tif they dseries ofas a framematics apidenticatsocial sciincludingVillage 2 afor the teevaluationmatics evfrom relevrelevantsearch. Inexaminedchapters osolely tohuman didods did nframewor
The evthe needclinical inin the pla. Thepaper tarhuman prsuch as minuences
uses of technology. Recommendations regardingthe timing and use of qualitative methods differfor each of these frameworks. The qualitativeresearch methods described in the followingevaluation frameworks include both the traditional
opeday bt thworia: (ncti
ss mo noretithan. The incomsbruce owor
. Gefram, thation frameworks fortics
ation frameworks reviewed in this paperloped in an attempt to provide rigorousprocesses that can be tailored to the
eds of informatics which requires eval-h during and after system development.
tication of evaluation frameworks
f this review was to identify and critiqueframeworks relevant to clinical infor-
search and to explore the types of andded timing for qualitative methods dur-
valuation process. The relative increaseh that uses qualitative research methodse need to understand the impact of clini-ation systems from a human perspective.he frameworks were deemed applicableescribed an evaluation framework or aevaluation methods that might be usedwork for the evaluation of clinical infor-
plications for use by humans. Towards theion of published frameworks, healthcare,ence and computer science databasesPubMed, CINAHL, PsychInfo, Engineeringnd the ACM digital library were searchedrms informatics evaluation, computer, evaluation frameworks, and infor-
aluation frameworks. The reference listsant articles were then searched for otherarticles not identied via the databaseaddition, informatics text books werefor evaluation frameworks. Articles,r books were excluded if they pertainedinformatics applications with which anot interact, or if the evaluation meth-
ot describe a formal framework. Twelveks that t these criteria were identied.aluation frameworks identied highlightfor the use of qualitative methods informatics either in conjunction with orce of traditional quantitative methodsqualitative methods described in this
get the identication and evaluation ofocesses integral to clinical informatics,anmachine interaction, organizationalon users and domain and context specic
Qualitcontethat ativelydevelods mreecframecriter(3) fusystemor abUser-cevaluaspectito whthe siried odegrefunctiworksprocethat da...