ORIGINAL PAPER

Methods to Evaluate Health information Systemsin Healthcare Settings: A Literature Review

Bahlol Rahimi & Vivian Vimarlund

Received: 1 February 2007 /Accepted: 6 July 2007 / Published online: 27 July 2007# Springer Science + Business Media, LLC 2007

Abstract Although information technology (IT)-basedapplications in healthcare have existed for more than threedecades, methods to evaluate outputs and outcomes of theuse of IT-based systems in medical informatics is still achallenge for decision makers, as well as to those who wantto measure the effects of ICT in healthcare settings. Theaim of this paper is to review published articles in the areaevaluations of IT-based systems in order to gain knowledgeabout methodologies used and findings obtained from theevaluation of IT-based systems applied in healthcaresettings. The literature review includes studies of IT-basedsystems between 2003 and 2005. The findings show thateconomic and organizational aspects dominate evaluationstudies in this area. However, the results focus mostly onpositive outputs such as user satisfaction, financial benefitsand improved organizational work. This review shows thatthere is no standard framework for evaluation effects andoutputs of implementation and use of IT in the healthcaresetting and that until today no studies explore the impact ofIT on the healthcare system’ productivity and effectiveness.

Keywords Evaluation studies .Medical informatics .

Literature review

Introduction

Information technology (IT)-based applications in health-care have existed for more than three decades and havegained widespread use [1, 2]. Nowadays, it is hard toimagine healthcare without IT-based applications for boththe accumulation and interchange of clinical information[3]. This is in part because IT has been recognized as an“enabler,” that is, as a tool that offers solutions to theproblem of the increasing accumulation of patient data [4, 5].Because of their central role enabling the diverse use of in-formation, IT systems ensure the timely and accurate collec-tion and exchange of information, and thus a more efficientuse of the scarce resources of healthcare organizations.

With an increased need for the implementation of IT inall healthcare domains—such as primary healthcare andclinical settings or home healthcare environments—for thepurpose of providing an optimal use of resource invest-ment, its use is expected to rise. Evaluating such ICTapplications to help decision makers acquire knowledgeabout the impact of IT-based systems therefore becomes akey issue to all organizations that aim to implement anynew application [6].

However, despite the fact that evaluation studies haveincreased in importance, they usually only provide answersto questions such as why the system should be studied, whya specific IT application should be chosen among manyother systems, or why they only present a general pictureabout costs and benefits to both users and the organization[7]. Case studies that examine the costs and benefits ofspecific IT applications are the most common examples ofempirical analysis, but the published work is neitherextensive nor comprehensive. Studies discussing evaluationmethodologies in the area of medical informatics haveusually been performed descriptively, and often use

J Med Syst (2007) 31:397–432DOI 10.1007/s10916-007-9082-z

B. Rahimi (*) :V. VimarlundDepartment of Computer and Information Science,Linköping University,Campus valla,Linköping 58183, Swedene-mail: bahra@ida.liu.se

B. RahimiDepartment of Social Medicine,Urmia University of Medical Sciences,Urmia, Iran

approaches from domains such as computer science,cognitive science, economics, and organization, and usuallydo not use in a multi-actor focus that includes the conse-quences of the implementation and use of IT-based systemsfor all participants, including care recipients and stake-holders, involved in the healthcare process.

The aim of this review is to outline the methodologicalapproaches and results of studies that measure the impact ofIT on healthcare organizations during 3 years (between2003 and 2005). We limit the study to review evaluations ofhealth information systems, including computer-basedpatient records, electronic health records, electronic medicalrecords, telemedicine, and decision support systems.

Previous literature reviews

There are several previous literature reviews in the field ofevaluation in medical informatics before 2003. They have,however, been conducted with several different aims, andtheir results are therefore difficult to compare. In thissection we present three reviews relevant to this currentpaper; all related to the evaluation of effect of IT-basedapplications for either clinical decision support systems orcomputer-based patient records systems.

Kaplan [72] reviewed studies focusing on the evaluationof clinical decision support systems (CDSS), with the mainemphasis upon changes in clinical performance andsystems that could improve patient care. Kaplan’s studyincludes many evaluations of CDSSs using designs basedon laboratory experiments or Randomized ControlledClinical Trials (RCTs). The author used Medline for aliterature search from 1997 to 1998. In addition to this, amanual search identified 27 papers referenced frequently byother experts in medical informatics, and these are includedin Kaplan’s review. Kaplan concluded that conductingRCTs is the standard method of evaluation approach forCDSS. She argued that RCT-type studies could not beuseful to investigate what influences whether systems areused. She also concluded that RCT-type studies do notanswer questions such as why some systems tend to beused while others are not, why the same system may beuseful in one setting but not in another, why a CDSS mayor may not be effective, or why different results areobtained in different studies. For this reason, more researchin this area is needed.

Ammenwerth and Keizer’s [71] review identified thetrend of evaluation research in the area of medical infor-matics from 1982 to 2002. The aim of the review was toanswer questions such as which countries or journalsdominate the evaluation research publications, which typesof information systems have been evaluated, which aimshave been studied in selected papers, which methods wereapplied for evaluation, how many papers have been

published, in which locations and settings did the studiestake place, and which evaluation strategy has been applied.Ammenwerth and Keizer’s review was based on a sys-tematic literature search in PubMed from 1982 to 2002,during which time 1,035 articles were selected. The authorsconcluded that the number of evaluation studies in the areaof medical informatics is rising significantly. However,evaluation studies cover only around 1% of all medicalinformatics publications indexed in PubMed. Ammenwerthand Keizer noted that explanatory research and quantitativemethods dominated evaluation studies during the periodchosen for analysis. The most common, recurring aspectsof the evaluation in the studies reviewed were appropriate-ness of patient care, efficiency of patient care, usersatisfaction, and software quality. They showed thatcomparable development in medical informatics since1982 was that the number of lab studies focusing ontechnical aspects was found to have declined. They alsoshowed that focus upon the quality of care processes andpatient outcomes was found to have increased. Theyinterpreted this shift as a sign of the maturation of evaluationresearch in medical informatics.

Delpierre et al. [8] reviewed studies of computer-basedpatient record systems (CBPRS). The objective of theirreview was to carry out a systematic survey of studiesanalyzing the impact of CBPRS on medical practice,quality of care, and user and patient satisfaction. Theyreviewed selected papers published from January 2000 toMarch 2003, identified by their search through Medline,Cochrane, and Embase databases. They selected a total of26 articles using this method. The authors concluded thatthey could show a better understanding of the relationshipbetween the use of CBPRS and medical practice. Anincreased satisfaction of users and patients was noted,which could lead to significant changes in medical practice.They also noted that most of the studies did not includequalitative factors such as characteristics of the disease andthe tool, the ward in which it was developed, and therelationship between various healthcare professionals,which could have an impact upon the use of CBPRS.Delpierre et al. [8] suggested in his study, a broad reviewincluding all the factors that may influence the success orfailure of the use of CBPRS in medical practice.

Some notable differences between the included reviews

The following table categorizes the differences and simi-larities among the three reviews discussed above.

As can be seen in Table 1, evaluation studies have beenperformed with different aims and objectives, and are con-cerned with different domains and areas of focus. However,all of them are focused on identifying trends in evaluationstudies, analyzing the impact of CPR, or to identify how

398 J Med Syst (2007) 31:397–432

CDSS changes clinical performance. Despite the largenumber of studies included in the reviews, there are nostudies that look specifically at the impact of IT onhealthcare efficiency or productivity. Almost all evaluationsare therefore partials, and the results are difficult to gen-eralize from each particular study to a broader context.Most of the studies that explored the impact of IT con-sidered decision support systems or electronic healthrecords. The main benefits identified to quality areincreased adherence to guidelines, better surveillance andmonitoring, and decreased medical error. However none ofthe reviews found studies that have a clear impact on laborinput or on the efficacy of the provision of care.

Materials and methods

A literature review was performed for published evaluationstudies of IT-based systems in healthcare, including CPR(including EMR and EHR), telemedicine, and differentkinds of DSS related to information systems like CPOE,between January 2003 and March 2006.

The search for the related material started in January2006 and finished in March 2006. Linköping University’sdatabase was used to gain access to papers on this subject.The keywords used to search for the articles were: patientrecords, medical Records, health records, informationtechnology, medical informatics, healthcare information,health informatics, hospital information system, patient careinformation system, CPOE, evaluation methods/theory,assessment, appraisal, information system/technology, eco-nomic evaluation, and evaluation study. In addition, themost important database, MEDLINE, was used to searchfor related papers. We conducted our search with the help

of search rules, for example MeSH (major topic), language,publication type, and publication date.

The following strategy was used to select the articlesfrom MEDLINE:

(1) We searched using the MeSH database, using theterms computerized patient records (CPR), decisionsupport systems, clinical (DSS), and telemedicine.

(2) In the next step we searched MeSH for evaluationstudies and found two categories called “evaluationstudies” and “Evaluation Studies [Publication Type].”

(3) We combined step 1 and 2 according to the followingdetails: (“Evaluation Studies”[MeSH] OR “EvaluationStudies”[Publication Type]) AND (“Medical RecordsSystems, Computerized”[MeSH] OR “Decision Sup-port Systems, Clinical”[MeSH] OR “Telemedicine”[MeSH])

(4) We limited these reviews to articles published between2003 and 2005 in English and with the aim of toevaluate some IT applications. We hit 674 articles.

We excluded papers concerning software and technicalevaluation, as well as ethical and education-related evalua-tions. We also excluded some other evaluation articles thathad no direct relationship with information systems, such asdocumentation and ordering through computers. Sixty-onearticles were left at the end of this review process. Thirty-eight studies had been performed in hospital settings, 15studies in primary care, and nine studies in homecare, withsome of the studies performed in two domains at the sametime. Two studies were performed in all domains (hospital,primary care and home care). Some projects, like IDEATelor Kaiser Permanente telehomecare, had a larger samplesize and publication. However, we limited our review tothis range, and excluded studies published before 2003 orafter March 2006.

Table 1 Differences and similarities among Kaplan, Ammenwerth, and Delpierre’s reviews

Review category Kaplan review Ammenwerth et al. review Delpierre et al. review

Aim of study How CDSS changes clinicalperformance

To identify trends of evaluation studies Analyse the impact of CPRon medical performance

Domains in whichsystems are evaluated

Clinical area National, Hospital, primary care, homecare

Hospital, primary care, homecare

Type of systemsevaluated

Published articles which included CDSSfrom 1997 to 1998

All kinds of systems—included publishedarticles from 1982 to 2002—onlyabstracts were reviewed

Published articles from 2000to 2002 based on CPR,CDSS, and CPOE

Study design Mostly RCT All kinds of study designs Mostly RCTThe approaches ofpapers which havebeen reviewed

Mostly quantitative approach Both quantitative and qualitative, butquantitative methods dominated

Mostly quantitative approach

Focus of evaluationstudies

Usefulness of system improved carewith RCT and show RCT as a goodmethod for evaluation

Appropriateness of patient care, usersatisfaction, and software quality

Clinical practice, patientoutcome, user and patientsatisfaction

J Med Syst (2007) 31:397–432 399

Results

In this section we will present a summary of the 61published articles in the medical informatics area from 2003to 2006 previously classified as evaluation studies. Thefollowing three tables present a brief description of thenames of the authors, the domains the study was performedin, the design of each study, the time of evaluation, thesources of evidence, the aim of the study, and the findings.The studies are divided into the following main categories:CPR evaluation studies, telemedicine evaluation studies,and DSS evaluation studies. In Tables 2, 3 and 4 wesummarize the evaluation studies classified as evaluation ofCPR, telemedicine, and DSS:

Comments

Evaluations of CPR tend to be concerned with systemusefulness regarding the quality of care, and user-relatedissues such as user acceptance and satisfaction and attitudestowards new systems, as well as the financial effects,usually limited to the identification of the costs of systemimplementation [9–11, 12–22].The inclusion of indirectcosts is not a common procedure, at least, in the studiesincluded in this literature review Usually the evaluationsclassified as “economic” include only the direct costs of theimplementation of the new system [11, 14, 23].

Another important remark is that some of the studiesincluded in this review tended to evaluate the effects of thenew system’s implementation on the quality of workperformance, such as user job performance and computerknowledge, and investigation of skill among other users[24–28].

From Table 2 it can be seen that introducing a newcomputerized patient record system had found to havepositive effects such as economic benefits [9, 11, 14], highacceptance score and satisfaction among the users in theimplemented sites [10, 13, 16, 21, 24, 29] and alsoImprovements in management and work process [11, 16,18, 22, 24, 28].

From Table 3 we can see that evaluations of tele-medicine tend to be concerned with the economic effects ofsystem implementation [30–34]. Usually the outputs areexpressed as costs and benefits, or effectiveness in thetelemedicine area. There are also, however, studies regard-ing user attitudes and perspectives, user satisfaction [35–38],and the usefulness of the system [30, 39–43], such as timeof service delivery, usability, feasibility, and changes in thenumber of visits after implementation of the new system.

From Table 3 it can be seen that introducing atelemedicine system had positive effects such as reducingspent time per patient during the visiting by clinical staff

[41, 42], economic benefits [30, 31, 39], and also quality ofcare [30, 39, 42, 44].

Table 4 provides insight into some issues regardingevaluations of DSS. It can be seen from Table 4 that mostof the systems that have been evaluated are CPOE, with theaim being to evaluate the usability of systems [45–49] orthe effectiveness of the system for patients [32, 50–53].Some of the studies reviewed tend to also evaluate thefinancial impacts of introducing the new system [54, 55]thus without systematic identification of all costs. There areeven some in this area that included measures of usersatisfaction and attitudes towards the system [31, 56].

From Table 4 it can be seen that introducing a clinicaldecision support systems in healthcare organizations hadpositive effects such as improved quality of care [46–49],satisfaction among users in the implemented sites [56], andalso improvements in management and work process [45,50, 52, 57].

However, most of the studies did not discuss a specifictheory to be applied when evaluating IT-based applicationsin healthcare. Few studies presented discussion of someeconomic theories such as cost-benefit/effectiveness analy-sis [9, 31–33], and none generated new theories or extendedolds ones.

It is also interesting that many studies used formativetests as an evaluation method, and a high proportion ofstudies used summative tests. Moreover, more than half ofthe papers used summative tests simultaneously. Focusingon most of the articles included in our study, we find thatthey have used objective and subjective perspectivessimultaneously.

Most of the studies based on the financial model, likecost-benefit/effectiveness, showed that there were improve-ments with the introduction of the new systems, especiallyin the telemedicine area. In contrast, some studies showedthat the implementation of new DSS or Telemedicine hadno economic benefits [32, 33, 54], and few showed that theintroduction of the new CPR or DSS were problematic[58–61].

Discussion

The purpose of this paper was to conduct a literature reviewfocusing upon the evaluation of IT-based systems inhealthcare. The results show that several similar studieshave been performed that focus upon clinical supportsystems, CPR for medical practice, telemedicine, decisionsupport systems, etc. Previous review studies concludedthat experimental designs are excellent for studying systemperformance or changes in clinical practice behaviour, butnoticed that evaluation approaches such as usability testing,cognitive studies, ethnography studies or socio-technical

400 J Med Syst (2007) 31:397–432

Tab

le2

CPRevaluatio

nstud

ies

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

ies

Finding

Sam

uelJ.

Wanget

al.

[200

3][9]

Primary

care

Survey

After

implem

entatio

nEcono

micdataon

costsand

benefitscamefrom

patient

medical

records

–Exp

ertop

inion

Toestim

atethenetfinancialbenefitor

costof

implem

entin

gelectron

icmedical

record

system

sin

prim

arycare

during

aperiod

of5years.

Thisstud

yshow

sthat

apo

sitiv

ebenefitof

abou

t$86,400was

obtained

bytheorganisatio

nover

the

period

of5years.The

implem

entatio

ncost

invo

lved

inthewho

leprocesswas

$46

,400

and

catego

rizedinto

twotypesof

costs:system

cost

(system

implem

entatio

n,supp

ortandmaintenance,

andhardware);andindu

cedcost(tho

seinvo

lved

intransm

ission

from

paperto

electron

icsystem

).The

benefitsob

tained,werecatego

rizedas;A)payer-

independ

entbenefits(obtainedfrom

thefactors

such

aschartpu

llandtranscription);B)capitated

patient

benefits(e.g.from

theadvancedrug

events;

from

theutilizatio

nof

drug

labo

ratories

and

radiology);andC)fee-for-servicepatients(w

hich

includ

edbettercharge

captureandredu

cedbilling

costsdu

eto

less

errors.)Significant

benefitfactors

werefoun

dto

be:saving

sin

drug

expend

iture

by33

%;theredu

ctionin

theutilizatio

nof

radiolog

yby

17%;im

prov

ementin

thecaptureof

charge

and

decrease

inbilling

costwhich

equalledto

15%

each

ofthetotalbenefits.

Hallvard

Laerum

etal.[200

3][10]

Hospital

Survey

After

implem

entatio

n–Q

uestionn

aire

(openended

questio

ns)completed

by70

physicians

–Interview

with

8ph

ysicians

Evaluatingtheeffectsof

scanning

and

elim

inationof

paperbasedrecordsby

stud

ying

physicians

andtheirattitud

etowardthesystem

Thisstud

yshow

sthat

scanning

andelim

inating

paperbasedmedical

recordswerefoun

dto

beacceptable,asitprov

ided

access

totheoldmedical

recordsin

theform

ofan

imageforthedo

cument.

The

stud

yalso

show

sthat

scanning

papers

are

beingused

with

ahigh

frequency,

butitis

considered

only

aninterm

ediate

stagetowards

having

fullaccess

totherecords.Thisstud

yshow

spo

sitiv

eresults

regardingtheperformance

ofthe

medical

tasksanduser

satisfaction.

The

useof

the

system

asawho

lewas

foun

dto

offersatisfactory

functio

nality,

i.e.forinform

ationretrieval,for

exam

ple,

9ou

tof

11tasksbecomeeasier

toperform.Moreover,ph

ysicians

also

foun

diteasier

toworkwith

thesystem

whereby

partof

the

system

hand

lingwas

combinedwith

theregu

lar

electron

icdata.How

ever,no

tallph

ysicians

were

foun

dto

besatisfied

with

theuseof

electron

icmedical

recordsdu

eto

basicdraw

backssuch

as,

J Med Syst (2007) 31:397–432 401

morenavigatio

ntim

eisrequ

ired

whenaccessing

thedata

ascomparedto

thetim

eused

indirect

interaction,

andthereislim

itedfunctio

nalityof

the

scannedim

ages

dueto

therigidstructureof

the

system

.Cornelia

M.

Ruland

etal.[200

3][11]

Hospital

Coh

ort

After

implem

entatio

n–E

cono

mic

data

extracted

from

nursingrecords

–Sem

istructured

Interview

(7nu

rsemanagers)

–Focus

grou

pwith

nurses

–Questionn

aire

to7nu

rse

managers

Todescribe

theeffectsof

thenew

system

for

evaluatio

non

:costredu

ctions;financial

managem

ent,anddecision

making

Thisstud

yshow

sthat

during

theevaluatio

nperiod

of4mon

ths,theim

plem

entatio

nof

CLASSICA

hadan

overallpo

sitiv

eim

pact

oneffectivenu

rsing

resource

managem

ent.Duringtheevaluatio

nperiod

test,un

itshada41

%redu

ctionin

expend

iture

comparedwith

theaverage

expend

iture

before

theintrod

uctio

nof

thenew

system

.Thismeans

that

theim

plem

entatio

nof

the

system

resultedin

better-costmanagem

ent.A

sign

ificantim

prov

ementin

inform

ation

managem

entw

asalso

noticed.Itw

asalso

observed

that

itbecameeasier

toanalysetherelatio

nship

betweenpatient

activ

ities,staffing

,as

wellas

the

costof

care.Highuser

satisfactionwith

thesystem

andtheinform

ationanddecision

supp

ortit

prov

ided

was

observed.The

system

was

foun

dto

beeasy

touse.

The

interviewsregardingtheease

ofuseanduser

satisfactionrevealed,accordingto

theauthors,po

sitiv

eresults.Itwas

also

observed

that

ithadbecomeeasier

torecogn

izethecauses

andexplanations

forbadandgo

odresource

managem

ent.

AdamsW.G.

etal.[200

3][64]

Hospital

and

Primary

care

Pre–p

ost

interventio

nanalysis

Beforeandafter

implem

entatio

nReview

ofmedical

records

(235

paperbasedvisits

and98

6compu

terbased

visitsdo

cuments)

Toevaluate

thequ

ality

ofdo

cumentatio

nand

deliv

eryof

paediatric

prim

arycare

before

and

aftertheim

plem

entatio

nof

anEMR

The

find

ings

ofstud

yshow

that

userswere

prom

pted

toaskabou

triskfactorsandcouldseeby

thetemplatedesign

that,theiransw

erswou

ldbe

availableat

thenext

visit,theseitemsare

frequently

lost.The

quality

ofanticipatory

guidance

also

improv

edconsiderably.The

outputs

ofthesystem

werewellaccepted

byfamilies.

Nearlyallusersalso

agreed

that

thesystem

improv

edthequ

ality

ofgu

idance

givento

families.

Clin

icians

areoftenov

erwhelm

edby

time

limitatio

nsin

theprim

arycare

setting

.The

electron

icmedical

records(EMR)im

prov

edthe

completenessof

deliv

eryof

ageapprop

riate

Tab

le2

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

ies

Finding

402 J Med Syst (2007) 31:397–432

guidance.Also6of

the7clinicians,who

use

compu

ter-basedrecords,repo

rted

usingthesystem

for95

%to

100%

ofno

nurgentcare

visits.Nearly

allusersagreed

that

thesystem

was

easy

touse,

increasedcompleteness,andremindedthem

todo

things

that

they

might

otherw

iseforget.4of

7usersfeltthat

thesystem

extend

edthedu

ratio

nof

visits.Mostusersalso

agreed

with

thestatem

ent

that

useof

thesystem

redu

cedeye-to-eye

contact.

Allusersagreed

thatthesystem

shou

ldcontinue

tobe

used.

Hou

ston

T.K

etal.[200

3][12]

Hospital

Survey

After

implem

entatio

nInterview

with

93patients

and82

physicians

Tosurvey

patients’

perceptio

nsof

hand

held

compu

teruseby

physicians,and

compare

those

with

theirprov

iders’

perceptio

ns

The

find

ings

ofthisstud

yshow

that

thegeneral

attitud

etowards

compu

ters

intheexam

ination

room

was

positiv

eandgenerally

patientsat

that

clinic

hadpo

sitiv

eattitud

estowardhand

held

compu

ters.33

patientswho

neverused

compu

ters

less

frequently

repo

rted

that

they

liked

theidea

ofresidentsusinghand

held

compu

terscomparedwith

thosewho

didusecompu

ters.Patientswho

seph

ysicianhadused

ahand

held

compu

terhada

sign

ificantly

morepo

sitiv

eattitud

etoward

hand

held

compu

ters.The

patientsexpressedthat

theuseof

hand

held

compu

tercouldbe

very

useful

inhelpingthem

toim

prov

ethecare

that

they

received.66

%ph

ysicians

agreed

that

therewas

enou

ghtim

eto

usethehand

held

compu

terin

the

clinic

practices.Som

eph

ysicians

(23%

)repo

rted

reservations

abou

tusingthehand

held

with

patients.

Elske

Ammenwerth

etal.[200

3][13]

Hospital

Coh

ort

Beforeand

during

and

after

implem

entatio

n

–Questionn

aire

completed

by39

nurses

–Interview

with

12nu

rses

Toevaluate

theprecon

ditio

nsandconsequences

ofcompu

ter-basednu

rsingprocess

documentatio

nwith

aspecialem

phasison

acceptance

issues

amon

gnu

rses.

Thisstud

yshow

sthat

sign

ificantly

high

initial

acceptance

scores

wereob

served

inallwards

before

theintrod

uctio

nof

thecompu

terbased

documentatio

nsystem

.A

difference

betweenthe

psychiatricwards

andthesomatic

wards

was

noticed

with

ahigh

erinitial

acceptance

scorefor

thesomatic

wards,as

thepsychiatricwards

were

nearly

completelyelectron

ically

documented.

No

overallchangesin

theacceptance

scores

were

noticed

during

thestud

y,ho

wever

rather

high

ermeanscores

wereob

served

onallwards.The

introd

uctio

nof

thecompu

ter-baseddo

cumentatio

n,ho

wever,didno

tseem

tohave

anymeasurable

influenceon

thegeneralacceptance

ofcompu

ters.

Acceptancescores

from

medium-to-high

were

J Med Syst (2007) 31:397–432 403

observed

aftertheim

plem

entatio

n.In

thecase

ofwardC(one

ofthewards)theinitial

acceptance

scaledeclined

andthen

rose

again.

Itwas

noted

that

meanscores

typically

decreasedafterthe

introd

uctio

nof

thesystem

,asseen

inotherstud

ies,

thus

show

ingadecrease

intheattitud

etowards

itsuseov

ertim

eat

theinterm

ediate

stage.

The

level

ofacceptance

bytheuserswas

observed

tobe

rang

ingfrom

medium

tohigh

.The

majority

ofthe

interviewsshow

edtheirlik

enesstowards

the

continuatio

nof

workwith

thePIK

inthefuture.

Thu

s,theov

erallacceptancescores

werequ

itehigh

andwerefoun

dto

rise,except

inthecase

ofward

C.The

high

eracceptance

scores

werefoun

dto

becorrelated

with

thenu

rse’syearsof

compu

ter

experience,aswellaswith

thehigh

erinitialscores.

Noinitial

know

ledg

eof

compu

ter,lack

ofconfidence

inuseandoldageof

staffwerethou

ght

tobe

thereason

sbehind

thelower

scores

inthe

case

ofwardC.Thu

s,itwas

concludedthat

compu

terkn

owledg

eandprevious

acceptance

ofnu

rsingprocessesweretheprerequisitesforthe

high

results

regardinguseandacceptance

ofthe

system

.W.P.

Zhang

etal.[200

4][14]

Hospital

Survey

After

implem

entatio

nQuestionn

aire

open-end

edqu

estio

nsto

81ho

spitals

Toclarifytheim

plem

entatio

nandmaintenance

costsof

acompu

terizedpatient

record

(CPR)

system

inho

spitals

The

find

ings

ofthisstud

yindicatethatCPRsystem

sim

prov

edtheho

spitaladministrationandmedical

exam

inations

practices.Itwas

foun

dthat

the

design

erof

thesystem

hadthegreatestim

pact

ontheperbedim

plem

entatio

nam

ongthevariou

sfactorssurveyed.Num

berof

servers,institu

tion

type,andim

plem

entatio

ndate

also

played

anim

portantrole

intheperbedim

plem

entatio

ncost.

The

exam

inationof

perbedim

plem

entatio

ncost

byho

spitalsize

show

edthat

themeancosttend

edto

bethelowestformid-sizeho

spitalsbu

thigh

erforsm

allas

wellas

largehospitals.E

xaminationof

annu

almaintenance

costsagainsttotal

implem

entatio

n,ho

wever,show

edno

difference

amon

gthethreesizescatego

ries.Morethan

60%

oftheusers(hospitaldirectors,system

managers,

Tab

le2

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

ies

Finding

404 J Med Syst (2007) 31:397–432

nurses

andgeneralstaffmem

bers)respon

dedwith

“veryhigh”or

“high”

satisfaction.

Anov

erall

improv

ementin

themedical

care

was

observed

bytheintrod

uctio

nof

CPRsystem

.From

the

improv

edho

spitaladministration(w

ith80

%po

sitiv

erespon

se),user

satisfaction(w

ith54

%po

sitiv

erespon

se)andbetterpresentatio

nof

inform

ation(w

ith75

%po

sitiv

erespon

ses);the

overallop

timistic

attitud

ewas

foun

dto

beevident.

How

ever,abigpercentage

of82

%respon

ded

negativ

elyregardingthedisadv

antagesof

CPRi.e.

confusionon

installatio

n,po

oruser

friend

liness,

andincreasedstaffworkloaddu

ring

learning

and

training

.Laerum

Het

al.[200

4][15]

Hospital

Survey

After

implem

entatio

n–Q

uestionn

aire

completed

by79

medical

secretaries,

172nu

rses,70

physicians

–Interview

with

8–12

representativ

emedical

secretaries,nu

rses,

physicians

Toevaluate

useof

andattitud

esto

aho

spital

inform

ationsystem

bymedical

secretaries,

nurses,andph

ysicians

The

find

ings

show

thatthenu

rses

andtheph

ysicians

usetheHIS

muchless

than

themedicalsecretaries.

The

medical

secretariesrepo

rted,ho

wever,that

all

definedtasksregardingcomputer-based

applications

wereperformed

moreeasily

than

paper-basedapplication.

The

nurses

andph

ysicians

notedthat

theelectron

icho

spitalinform

ation

system

hasless

simplifiedtheirworkthan

medical

secretaries.The

medical

secretariesweremuch

moresatisfies

with

theuseof

theHIS

than

the

nurses

andph

ysicians,particularly

with

theuseof

scanneddo

cumentim

ages.The

secretariesgave

sign

ificantly

morepo

sitiv

erespon

sesthan

the

nurses

andtheph

ysicians

toHIS.Nursesand

physicians

wereless

satisfied

with

usingthe

system

.The

mostfrequently

repo

rted

prob

lems

amon

gtheph

ysicians

consistedof

variou

ssoftware

andhardware-relatedprob

lems,thesystem

working

tooslow

ly,andlack

ofcompu

ters

where

theclinical

workwas

beingdo

ne.Bothnu

rses

and

physicians

inthemedical

wardfoun

dthat

patient

data

weremoreaccessible

whenstored

electron

ically

than

whenstored

onpaper,in

particular

regardinglabtestdata.

DanielB.Hier

etal.[200

4][16]

Hospital

Survey

After

implem

entatio

nQuestionn

aire

distribu

tedto

330faculty

andho

use

staffph

ysicians

(closed

andop

enended

questio

ns)

Todeterm

inewhether

anydifferencesexist

betweenho

usestaffandfaculty

physician’s

acceptance

ofan

electron

ichealth

record

system

.

The

stud

yshow

sthat

theuser

acceptance

ofthe

EHRwas

high

(86%

)forbo

thfaculty

physicians

andho

usestaffph

ysicians.How

ever,ho

usestaff

physicians

show

edsign

ificantly

high

erfrequency;

88%

preferredtheEHRto

apaperrecord

and75

%said

they

enjoyedusingtheEHR.64

.7%

ofthe

faculty

mem

bersindicatedthey

prefer

itand60

.6%

J Med Syst (2007) 31:397–432 405

show

edtheirinterestin

itsuse.

88%

oftheho

use

staffand64

.7%

ofthefaculty

believedthat

the

EHR

increasedavailabilityof

medical

records,

easedthedeterm

inationof

thetreatin

gph

ysician,

increasedtheability

tocommun

icatewith

other

physicians,increasedthelegibilityof

documentatio

n,andeasedaccess

tolabo

ratory

results.A

majority

ofho

usestaffph

ysicians

believedthat

theEHRmadeworkflow

more

efficient,redu

ceddu

plicateor

unnecessarytesting,

savedtim

edo

cumentin

gcare,andredu

cedpatient

care

errors.Majority

ofthefaculty

shared

their

view

onthesequ

estio

nnaire

items.System

speed

andlack

ofcompu

ters

wereconsidered

tobe

the

greatestbarrierby

both

grou

ps.

Joseph

K.

Rotichet

al.

[200

4][17]

Primary

care

Tim

eseries

Beforeandafter

implem

entatio

nTim

emeasurement

Evaluationof

theim

pact

ofintrod

ucingCPRon

timeof

healthcare

deliv

erywith

intherural

health

centre

The

find

ings

ofthisstud

yshow

that

theMosoriot

Medical

RecordSystem

(MMRC)clearlychanged

theflow

ofhealthcare

intheMosoriotR

uralHealth

Center(M

RHC).Asubstantialdecrease

inthetim

eof

patient

interactionwith

healthcare

prov

ider

was

observed.The

usingof

thesystem

show

edaclear

redu

ctionof

10minutes

(from

41to

31minutes)in

each

visitforthepatients,i.e.patientswho

were

noticed

tospendless

timewaitin

g.For

the

healthcare

prov

iders(nursesandclinical

officers)

thetim

e–motionredu

ctionwas

also

observed.

Theyspentless

timewith

patientsandotherstaff

andhadas

aresult,

moretim

eforperson

alactiv

ities.For

thenu

rses

andclerks,ho

wever,it

show

edmoretim

espentin

registratio

nbu

tless

timeforwritin

grepo

rts(3%

vs.18

%).Sim

ilarly

therewas

anincrease

inthetim

eforsearchingthe

inform

ationthroug

hthesystem

(3%

vs.0,

5%).

Overall,

inthisstud

y,itwas

agreed

that

asimple,

inexpensive,

andeffectiveelectron

icmedical

record

system

couldbe

establishedin

adeveloping

coun

trywith

poor

resource;thebenefitsof

which

will

berealised

inthelong

runin

term

sof

finances

andtherewill

bebetterhealth

services

served

tothepeop

le.

Tab

le2

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

ies

Finding

406 J Med Syst (2007) 31:397–432

DrikSteng

elet

al.[200

4][18]

Hospital

Rando

mized

controlled

trial

After

implem

entatio

nDataob

tained

from

patient

medical

records

–Archivedecon

omic

recordsassessed

byexpert

observers

Todeterm

inewhether

theintrod

uctio

nhand

held

compu

ter-baseddo

cumentatio

ncouldim

prov

ebo

ththequ

antitativeandqu

alitativ

easpectsof

medical

records

From

thisstud

y,itwas

foun

dthattheintrod

uctio

nof

ahand

held

compu

terinto

daily

routines

increased

thenu

mberof

diagno

sisrecorded

andalso

itim

prov

edtheov

erallqu

ality

ofthepatient

records.

The

stud

ydemon

stratedthattheratin

gson

allthree

investigated

aspectsof

documentatio

nqu

ality

were

betterwith

theelectron

icdevice

than

with

conv

entio

nalhand

writtendo

cumentatio

n.The

timerequ

ired

forhand

held

compu

ter

documentatio

nwas

also

foun

dto

besign

ificantly

decreased.

Itwas

also

noticed,ho

wever,that

the

difference

inthenu

mberof

diagno

seswas

slightly

balanceby

theaccompany

ingload

ofredu

ndant

itemswhich

was

said

tobe

sign

ificantly

infavo

urof

theexperimentalhand

held

documentatio

nsystem

.The

conclusion

basedon

thestud

yclearly

affirm

sthepracticability,accurateness,and

accessibility

ofthedo

cumentatio

nas

comparedto

thetradition

almetho

ds.

Likou

rezosA

etal.[200

4][19]

Hospital

Survey

After

implem

entatio

n–Q

uestionn

aire

completed

by23

physicians,and21

nurses

Tomeasure

clinicians’compu

terbackgrou

ndand

experience,satisfactionwith

,perceptionof,and

concerns

aboutan

EMR

inem

ergency

department(ED)

The

find

ings

ofstud

yshow

that

allclinicians

were

know

nwith

compu

terandalso

hadused

compu

ter

before

than

thenewsystem

isinstalled.

Mosto

fthe

physicians

werefamiliar

with

theInternet

asthey

used

everyday.Allclinicians

weretrainedanduse

theem

ergencydepartment(ED)EMR.Eighty-

sevenpercentof

physicians

and85

.7%

ofnu

rses

useEMReveryday.

The

similarpercentage

has

access

totheEMR

from

anyplacein

theclinical

setting

.Easyto

enterdata,access

data,read

data

onthescreen

werefoun

dto

befacilitated

bythe

EMRrepo

rted

byclinicians.Mostnu

rses

butno

tph

ysicians

perceive

that

patient

inform

ationis

moreconfidentialwith

EMR

than

with

paper

record.A

high

erpercentage

ofnu

rses

than

physicians

perceive

that

anEMR

will

elim

inate

paperw

orkandwill

redu

cetherisk

oferrors.A

majorities

ofph

ysicians

andnu

rses

repo

rtthat

the

EMRwill:no

tim

prov

ethequ

ality

ofmedical

care

received

bythepatients,no

tmakepatient

care

less

expensive,

notdecrease

patient

waitin

gtim

e,no

tdecrease

thenu

mberof

ED

visits,no

tdecrease

the

crow

ding

intheED.

Rainu

Kaushal

etal.[200

4][23]

All

domains

Survey

Before

implem

entatio

n–Interview

with

expert

panel(10peop

le)

–Econo

mic

data

obtained

Toestim

atethecostsof

anNHIN

(National

health

inform

ationnetwork)

The

find

ings

show

that

ITisan

impo

rtanttool

used

toim

prov

ethesafety

andefficiency

oftheU.S.

healthcare

system

,bu

tits

adaptatio

nwas

J Med Syst (2007) 31:397–432 407

from

publishedcost

estim

ates

considered

toremainlim

itedlargelybecauseof

alack

ofalignedfinancialincentives

andnatio

nal

standards.The

estim

atefortheim

plem

entatio

nof

theNationalHealth

Inform

ationNetworkwas

regarded

tobe

expensive.

Itwas

estim

ated

that

NHIN

will

costabou

t$15

6billion

capitalcostsand

approx

imately2yearstim

eisrequ

ired

forreal

grow

thin

theU.S

healthcare

costs.Itwas

recogn

ized

that

benefitsfrom

investmentsgained

substantially

both

interm

sof

mon

eyandqu

ality.

The

dataob

tained

from

thestud

ysugg

estedthatIT

coulddecrease

costsof

healthcare.Atthesame

time,

ITcouldim

prov

equ

ality

ofcare

andwork

performance.The

stud

yalso

show

edthat

byinvestingin

differenttypesof

clinical

inform

ation

system

sandtheprov

idersof

healthcare,itwill

likelyto

maxim

izetheinvestment.

Claud

iaPagliariet

al.

[200

4][20]

Primary

hospital

Survey

After

implem

entatio

nInterview

with

stakeholders,

questio

nnaire

toproject

team

sandclinical

users

Toassess

thedevelopm

entof

theScottish

Nationalelectron

icclinical

commun

ications

system

afterim

plem

entatio

n

The

stud

yindicatesdifficultieswhile

attemptingto

implem

entatechnicalandbehaviou

ralchange

interventio

n.Moregeneralim

prov

ementsin

human

andtechno

logy

capabilitywerefoun

dto

befacilitated

bytheprog

ram.The

prog

ram

was

perceivedto

have

facilitated

aspectslik

ehu

man

infrastructure

andstim

ulated

jointworking

amon

gstakeholdergrou

ps.The

stud

yalso

show

sthat

the

implem

entatio

nof

thesystem

was

themost

successful,andelectron

icou

tpatient

book

ingthe

leastsuccessful

during

thestud

y.Analia

Baum

etal.[200

4][21]

Hospital

Survey

After

implem

entatio

nQuestionn

aire

to15

0administrativestaff,

doctors,andnu

rses

Toevaluate

theim

pact

oftheim

plem

entatio

nof

anew

mod

elof

helpdesk

andtechnicalsupp

ort

inHIS

usersat

theho

spital

The

stud

yshow

sapo

sitiv

eattitud

eof

theusers

towards

theim

plem

entatio

nof

HIS.A

greater

frequencyof

positiv

erespon

sesby

thedo

ctorsthan

anyothergrou

pwas

observed.The

administrative

staffandnu

rses

weresatisfied

concerning

the

accessibility

throug

htheinternet.Doctors

show

edmoresatisfactionhigh

erthan

thenu

rses.R

egarding

theanalysisof

theun

derstand

ingof

thehelpdesk,

allthreegrou

psof

usersshow

edtheirsatisfaction,

althou

ghtherespon

sefrom

thenu

rses

was

foun

dto

beless

satisfactory.

Asaresultof

thestud

yof

therespon

setim

egivento

theusers’

requ

ests,the

threegrou

psagainaffirm

edthesuccessof

the

Tab

le2

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

ies

Finding

408 J Med Syst (2007) 31:397–432

system

.There

was

with

agreaterrespon

sefrom

the

doctorswho

foun

dHIS

tobe

atool

with

fast

accessibility

andavailability24

hours.The

useof

software,

tokn

owtherequ

irem

entandto

measure

therespon

setim

e,contribu

tedto

improv

ingthe

system

sim

ageon

thepartof

theusers.Itwas

foun

dthat

respon

setim

eaffectsbo

thnu

rses

and

doctorssatisfactionandisacond

ition

that

isto

beim

prov

ed.

Laura

Zurita

etal.[200

4][29]

Hospital

Survey

After

implem

entatio

n–Interview

with

12patients

–Directob

servationof

daily

routines

Toprov

ideinform

ationabou

tpatient

values

and

commun

icationthatwill

beuseful

inthedesign

ofamorepatient

friend

lyhealth

record

system

.

The

results

ofthisresearch

underlie

thegeneral

societal

tend

encies

oftheusersacceptance

ofthe

useof

new

techno

logy.How

ever,itwas

also

observed

that

theuserswerevery

enthusiastic

onprotectin

gtheirprivacy.

The

useof

EHR

isperceivedto

protecterrors,prov

idemoreeffective

health

system

sandgive

betterserviceas

wellas

allow

thehealth

prov

ider

tohave

easier

access

totheinform

ation.

The

ease

ofaccessibility

ofinform

ationwas

also

expected

andthestud

yshow

sthat

theexpectations

weremet,as

94%

gained

access

tothesystem

throug

htheintranet

andthe

rest(6%)accessed

itviateleph

one.

The

intranet

(helpd

esk)

was

foun

dto

beuseful

forthemedical

grou

ps.Allusersalso

show

edsatisfactionabou

ttherespon

setim

e.Users’attitud

ewas

foun

dto

bepo

sitiv

etowards

thechange.

Emans

Evang

elJoel

Rajaet

al.

[200

4][24]

Hospital

Survey

After

implem

entatio

n–S

tructuredQuestionn

aire

distribu

tedto

120nu

rses

ToAssessCom

puterkn

owledg

e,attitud

estoward

CPR,andrelatedskillsam

ongnu

rses

The

find

ings

show

anov

erallpo

sitiv

eattitud

etowards

compu

terutilizatio

nby

thepatients.The

results

ofthisstud

yshow

that

themajority

ofthe

nurses

(75%

)hadgo

odcompu

terkn

owledg

eand

20.8%

ofthem

hadgo

odcompu

terkn

owledg

ein

aclinicalcare

setting

.Halfof

thenu

rses

hadaverage

compu

terskills.The

stud

y,ho

wever,show

sno

relatio

nshipbetweencompu

terkn

owledg

eand

compu

terattitud

eandskillsam

ongnu

rses.

Com

putertraining

typesdidno

tseem

toaffect

nurses’compu

terkn

owledg

e.Attitude

towards

compu

terusewas

foun

dto

beindepend

entof

sex,

age,

nurse’sexperiences,profession

alqu

alifications,andmon

thly

family

incomeon

compu

terusage.

Jonathan

C.

Hob

sonet

al.

[200

5][22]

Hospital

Coh

ort

Beforeandafter

implem

entatio

nDataextractedfrom

medical

hand

writtenno

tes

andcompu

terdata

Toevaluate

whether

orno

ttheuseof

acompu

terizedMicrosoftWordtemplatewou

ldincrease

theaccuracy

ofclinical

notes

The

find

ings

ofthisstud

yreveal

that

compu

terized

notesweremoreaccurate

andeasier

topu

tin.The

results

ofthestud

yshow

that

before

implem

entin

g

J Med Syst (2007) 31:397–432 409

anew

system

,theaccuracy

ofhand

writtenno

tes

didno

tmeetthe

predefined

standard

set.The

initial

stud

yfoun

dthat

impo

rtantdemog

raph

icdata

were

missing

from

ordinary

hand

writtenno

tes.After

implem

entin

gtheno

teswerefoun

dto

beaccurate

inmorethan

90%

ofthecases.The

form

sused

toenterdata

wereadop

tedeasily.Calendardata

entrieswereinserted

automatically

toenable

easier

andmoreaccurate

details.Itwas

concludedthat

compu

terizedno

tesshou

ldbe

introd

uced

inclinics

dealingwith

urgent

referrals.

GeorgeW.

Odh

iambo

-Otieno

etal.

[200

5][58]

Primary

care

Survey

After

implem

entatio

n–Interview

with

system

design

er’sop

erators,users

–Day-to-dayob

servationof

theop

erations

ofthe

system

–Review

ofrelevant

literature

Toevaluate

theextent

ofwhich

existin

ginform

ationsystem

shave

supp

ortedthe

operationalmanagem

entof

health

services

atthedistrict

levelin

Kenya.

The

authorsconclude

that

DistrictHealth

Managem

entInform

ationSystem

(DHMIS)were

notsup

portiveof

theDistrictH

ealth

System

(DHS)

Manager’sstrategy

andop

erationalmanagem

ent

functio

n.There

was

amarkeddifference

inthe

focusof

healthcare

workers

andDHMIS

staff.

The

health

workers

werefoun

dto

complainabou

tthequ

ality

ofinform

ationprod

uced.Ontheother

hand

,theDHMIS

staffs

wererepo

rted

tocomplain

abou

ttheinadequate

supp

liesof

basicresources

necessaryforeffectiveop

erationof

thesystem

.A

lack

ofintegrationwas

noticed

intheinform

ation

system

stud

ied;

they

weredisjointed

with

noeffectivecentralcontrolor

coordinatio

nto

ensure

that

theinform

ationcouldbe

madeavailable

wheneverandwherever.The

variationin

collected

data

also

madeitim

possible

toallow

comparison

interm

sof

performance

amon

gDHSs.The

results

also

revealed

that

thereweresomeinadequacies

ofperson

nel,equipm

ent,working

space,

storage

space,

trainedstaff,andmanagem

entsupp

ort.The

inform

ationprod

uced,as

aresult,

was

less

accurate,un

timely,

lackingconfidentiality,

and

incomplete.

Clayton

P.D

etal.[200

5][25]

Primary

care

Survey

Beforeandafter

implem

entatio

nDataob

tained

from

patient

medical

records

Todiscussthebenefitsof

usingEHRand

measure

physicianprod

uctiv

itybefore

andafter

implem

entatio

nof

theEHR.

The

find

ings

ofthisstud

yshow

that

currently,of

472em

ploy

edph

ysicians,32

1(68%

)routinely

entersomedata

directly

into

theEHRwith

out

coercion

.Twenty-fivepercent(80/32

1)of

the

physicians

usevo

icerecogn

ition

forsomedata

Tab

le2

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

ies

Finding

410 J Med Syst (2007) 31:397–432

entry.Tw

elve

of95

ambulatory

clinicshave

voluntarily

adoptedmeasuresto

elim

inatepaper

charts.O

fthe212physicians

who

entereddatain

2004,sixty-ninephysicians

(22%

)increasedtheir

levelo

fdataentry,while12

(6%)decreased.In

this

studyitwas

foundthattheaverageproductiv

ityincreasedover

theduratio

nof

theevaluatio

nperiod.

Mosto

fthesurvey

participantsreported

that

inform

ationretrievaltim

ewas

betterwith

theEHR

pilotsystem

than

theirprevious

system

andalso

nearlyallofthe

providersbelievedthatthey

provided

bettercare

fortheirpatientsby

usingtheEHRand

also

respondentsdidnotw

anttoreturn

totheir

previous

system

.Staffproductiv

ityim

proved

significantly.R

eportedbenefitsof

EHR

implem

entatio

nincluded

improved

intraoffice

communication,decreasedtranscriptions

costs,

perceivedim

proved

quality

ofpatient

care,and

improved

record

accessibility.

LisaPizziferri

etal.[200

5][65]

Primary

care

Tim

eseries

Beforeandafter

implem

entatio

n–Q

uestionn

aire

completed

by23

physicians

–Tim

emotionObservatio

nof

physicianactiv

ity

–Effectof

theEHR

ontheph

ysician’sov

erall

timein

clinical

sessions

–Evaluationof

physician’sperceptio

nsregarding

theEHR

The

results

ofthisstud

yshow

that

theov

eralltim

espentperpatient

during

clinical

sessions

decreased

with

thenew

system

.A

majority

ofph

ysicians

believedEHR

useresults

inqu

ality

improv

ement.

The

results

werecatego

rizedinto

twotypes:direct

patient

care

andindirectpatient

care.T

here

was

nosign

ificantchange

indirect

patient

care,for

exam

ple,

thetim

espentin

activ

ities

such

asthe

timespentin

exam

iningthepatient.The

physician

believedthat

theinpu

tof

thenew

system

oncommun

ication,

access,efficiency,workload,

and

quality

ofcare

indicatedthat

thenew

system

resultedin

anim

prov

ementin

manydo

mains.The

only

item

show

ingalower

ratin

gthan

averagewas

thenew

system

’sim

pact

onworkload.

Mandi

Why

teet

al.[200

5][26]

Hospital

Survey

After

implem

entatio

nDataob

tained

from

patient

medical

records

Toassess

thequ

ality

ofrecord

keepingby

nurses

inapaediatric

ward

Itwas

foun

dfrom

thestud

ythat

thecompu

terized

recordsarebetterforrecord

keepingin

thecontext

ofchild

protectio

n,as

alargeam

ount

ofinform

ationisstored

wheneverthenu

rses

logon

.How

ever,no

neof

therecordsthat

wereaudited

met

thepredetermined

standard.The

auditalso

indicatedthatbo

ththetwotypesof

record

keeping

intwowards

[HighDependencyUnit(H

DU)and

PaediatricIntensiveCareUnit(PICU)]hadtheir

faults.T

rainingandeducationissues

werefoun

dto

J Med Syst (2007) 31:397–432 411

bethebasicreason

behind

thesubstand

ardresults

achieved

bytheuseof

thetwosystem

s.Itwas

concludedthat

nursingtraining

prog

ramsmust

compriseof

such

ITtraining

too,

andtheresults

revealed

that

inthePICU,40

%of

thepatientsdid

notaccess

totheirprevious

data.

Tim

Scott

etal.[200

5][59]

Primary

care

and

hospital

Coh

ort

Beforeandafter

implem

entatio

nSem

istructureinterviews

with

senior

clinicians,

managers,projectteam

mem

bers

Toexam

ineusers’

attitud

estowards

implem

entatio

nof

anEMR

System

The

find

ings

ofthisstud

yshow

thattheadop

tionof

theelectron

icmedicalrecord

system

was

flaw

edanddetached

from

thelocalenv

iron

mentshow

ing

disapp

rovalof

theim

plem

entatio

n.Fou

rmajor

processeswereidentifiedin

theim

plem

entatio

nof

EMR:selectio

n,design

andearlytesting,adaptatio

nforwidespreaduse,andadaptatio

nof

the

organizatio

nto

thenewelectron

icmedicalrecords.

Regarding

theselectionof

theelectron

icmedical

record

system

,onlyon

erespon

dent

expressed

approv

alof

thesystem

choice,the

restfoun

dit

prob

lematicandrepo

rted

dissatisfactionwith

the

choice.T

heresistance

observed

towards

thedesign

andtestingwerethou

ghttobe

dueto

inadequate

earlytesting,

substantialsoftwareprob

lems,and

inadequateIT

know

ledg

e.Difficulties

werealso

noticed

regardingtheadop

tionof

theelectron

icmedicalrecordsdu

eto

differencesov

erprioritiesof

theusersanddu

eto

softwareprob

lemsitself.The

results

also

show

edthatthesystem

redu

cedclinical

prod

uctiv

itydu

etoextraworkinvo

lvingprocessing

labo

ratory

resultrepo

rts,entry’sordersand

navigatin

gthroug

hthesystem

.Maria

Bryson

etal.[200

5][66]

All domain

Survey

After

implem

entatio

nQuestionn

aire

completed

by20

20nu

rses,midwives

andhealth

visitors

workers

(Onlinesurvey)

Toinvestigatethenu

rses

perceptio

n,kn

owledg

eandexpectations

oftheNationalHealth

system

’sIT

prog

ram

Poo

raccess

andinadequate

know

ledg

eof

theplans

intheNationalH

ealth

Service

(NHS)was

observed

asaresultof

thisstud

y.The

survey

sugg

estedthat

thenu

rsingstaffs

wereno

tfully

awareof

the

currentIT

plans.How

ever,51

%of

thenu

rses

affirm

edthesign

ificantim

prov

ementbrou

ghtby

theintegrated

electron

ichealthcare

record

system

inclinical

care.Nohigh

levelof

enthusiasm

aspredictedwas

observed.Lackof

confidence

inusingadvanced

inform

ationtechno

logy

compo

undedwith

thelack

oftraining

andno

Tab

le2

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

ies

Finding

412 J Med Syst (2007) 31:397–432

invo

lvem

entof

thenu

rses

inthechange

process

werefoun

dto

bethereason

sbehind

thenegativ

erespon

ses.For

future

developm

ents,itwas

expected

that

earlyandregu

larinvo

lvem

entof

potentialuserswou

ldbe

madepo

ssible.Itwas

demon

stratedthatthenu

rses

wou

ldbe

ableto

share

decision

makingwith

clientsusingcredible,tim

ely,

andaccurate

inform

ation.

Wilcox

Aet

al[200

5][27]

Primary

care

Retrospectiv

ecoho

rtstud

y,Survey

Beforeandafter

implem

entatio

n–Interview

with

7ph

ysicians

–Trackingof

medicalrecord

Toanalyzetheusageof

summaryworksheets,

andtheirim

pact

onph

ysicianperformance.

Thisstud

yshow

sthat

four

physicians

forevery

patient

seen

usetheworksheet,andthreeof

them

stated

they

likeit.

Twoidentifiedho

wthey

useitto

trackeither

vitalsigns

orlabo

ratory

results

fortheir

patients.One

indicatedhe

only

used

itsometim

es,

butalso

stated

hisintentionto

useitmore.

One

physiciansaid

hereview

severyfield,

while

other

physicians

useiteither

forfollo

wingdisease-

specific

labvalues

orforreview

ingmedications

andtrending

bloo

dpressure

andweigh

t.One

physicianwho

indicatedlim

itedusesaid,“Hedo

esreview

thereminders

specifically”.

Alsothe

find

ingof

thisstud

yindicatedthat

usagehas

grow

nfrom

afew

hund

redto

over

25,000

unique

patientspermon

thdu

ring

a2-year

period

.The

find

ings

show

salso

diabetic

patientsforwho

mthe

worksheet

isaccessed

aresign

ificantly

morelik

ely

tobe

incompliancewith

accepted

testingregimens

forglycosolated

hemog

lobin.

KMäkelä

etal.[200

5][28]

Primary

care

Survey

After

implem

entatio

nQuestionn

aire

tohealthcare

profession

alin

15healthcare

center

Toevaluate

how

wellEPRsystem

smeetthe

needsof

physicians

andotherhealthcare

profession

alsandto

determ

ineifthereareany

sign

ificantdifferencesbetweenEPRsystem

s.

The

find

ings

show

that

despite

awideuseof

EPR

system

sin

healthcare

centres,thoseelectron

icdatabasescouldno

tyetcompetein

allaspectswith

conv

entio

nalpaper-basedrecords.The

fact

isa

reflectio

nof

thedifficulty

ofstructuringclinical

data

tomakethem

easily

accessible

while

still

retainingasimpleanduser-friendlymetho

dto

inpu

ttheinform

ationfor73

%of

thecases,allof

theinform

ationrequ

estedwas

madeavailablein

theEPR.Nosign

ificantcorrelationwas

observed

betweentheam

ount

ofpatientsforwho

mall

inform

ationwas

availablein

theEPRandthe

duratio

nforwhich

theEPRsystem

shadbeen

inuse.

Asign

ificantdifference

betweenthetype

ofEPRsystem

andthepercentage

ofpatientsfor

who

malltheinform

ationwas

availablethroug

htheEPRwas

demon

stratedthroug

hout

thestud

y.

J Med Syst (2007) 31:397–432 413

Tab

le3

Telem

edicineevaluatio

nstud

ies

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

B.Larcheri

etal.

[200

3][35]

Hospital

Survey

Beforeand

after

implem

entatio

n

Questionn

aire

completed

by59

physicians

and63

nurses

(open

andclosed

respon

sequ

estio

n)

Toexploreuser

satisfactionwith

thenewsystem

after6mon

thsof

experimentatio

n

The

find

ings

ofthisstud

yshow

that

therewas

anov

erallpo

sitiv

eattitud

etowards

thesystem

.A

sign

ificantdifference

was

noticed

betweenthe

user

opinionbefore

andaftertheuseof

the

system

,thus

wemight

conclude

that

theuser

satisfactionwas

generally

high

.A

qualitativ

eanalysisshow

edthat

86%

ofph

ysicianusersfelt

that

theteleconsultatio

nwas

useful

asitoffered

direct

commun

icationwith

thecolleagues.The

averageof

consultatio

ntim

efortheequivalent

outpatient

clinicin

thesameho

spitalfor

thesame

consultant

was

15minutes.In

additio

n,having

diagno

stic

inform

ationallowspatientsto

bescreened

andup

to50

%canbe

elim

inated

asno

trequ

estin

gtheho

spitalinvestigation.

Itwas

expected

that

atechno

logy

-based

approach

could

redu

cethenu

mberof

outpatient

clinicsby

75%,

redu

cetheconsultatio

ntim

eandby

makingthe

fullinform

ationavailable,

ensure

that

each

consultatio

niseffective.

Tho

mas

L.

etal.

[200

3][39]

Hom

eand

prim

ary

care

Con

trol

trial

andsurvey

After

implem

entatio

n–Q

uestionn

aire

completed

by14

7stud

ents(con

sultant

survey),94

scho

olnu

rseevaluatio

n,77

stud

ent

evaluatio

ns,51

parentsevaluatio

n–E

cono

mic

recordsexistencein

the

centers

Toevaluate

thequ

ality

andcost

effectivenessof

health

care

prov

ided

inurbanandrural

elem

entary

scho

ol-based

telehealth

centers

The

findings

ofthisstudyshow

thatprovider,nurse,

child,and

parent

satisfactionallw

erehigh.T

heproviders’andnurses’decision

confidence

scores

ranged

from

alowof

4toahigh

of4.8on

a5-point

scale.The

high

meanscores

onthedecision

confidence

scales

indicatedthatthetelehealth

protocolsspeededthetim

ebetweendiagnosisand

starto

ftreatm

entand

helped

with

thenurses’

decision

aboutthe

disposition

whenthechild’s

problem

was

outsideof

astandard

protocol.T

hestudentswerealso

positiveaboutthe

interactionwith

thephysicianor

thenursepractitioneraverage

family

savingsperencounterwere3.4hoursofwork

time($43)and$177

inem

ergencydepartmento

r$54in

physiciancosts.Includingtravel,savings

for

families

ranged

from

$101

to$224

perencounter.

Lynne

Bladw

inet

al.[2003]

[40]

Primary

care

Coh

ort

Beforeandafter

implem

entatio

n–T

imemeasurement

–Dataob

tained

from

hospital

records

Tomeasure

theeffect

oftelemedicineon

consultatio

ntim

e

The

results

ofthisstud

yshow

that

with

theuseof

telemedicine;

thenu

mberof

registered

outpatientsin

clinicsdecreased.

The

duratio

nof

theconsultatio

ntim

ewas

redu

cedfrom

23to

10

414 J Med Syst (2007) 31:397–432

minutes

becausediagno

stic

inform

ationwas

beingsent

inadvanceandin

theconsultatio

ntim

e,theusersdiscussthemanagem

entof

the

disease.

Inotherwords

consultatio

nchanged

from

diagno

sisto

managem

ent.With

diagno

stic

inform

ation,

patientscanbe

screened

andabou

thalfof

them

canbe

considered

tobe

with

outany

need

ofho

spitalinvestigation.

Moreover,

accordingto

theresults

ofthisstud

ytelemedicine

couldredu

cethenu

mberof

outpatient

clinicsby

75%

aswellas

redu

cethedu

ratio

nof

the

consultatio

n.Finkelstein

Jetal[200

3][41]

Hospital

and

Hom

ecare

Survey

After

implem

entatio

n–Q

uestionn

aire

completed

by41

patients

–Interview

with

41patients

Toassess

user

acceptance

ofthe

homeautomated

telemanagem

ent(H

AT)system

The

find

ings

show

that

inthepretestin

gph

aseat

hospital,patientsdemon

stratedahigh

levelof

acceptance

ofuser

interface.

90%

ofthepatients

stated

nodifficultiesin

usingthesystem

and93

%of

thepatientsclaimed

they

wou

ldusethesystem

atho

meandadvise

others

thesystem

.The

patientsnamed

saving

timeby

redu

cing

number

ofvisitsandgetting

results

ontim

e,andthe

possibility

ofhaving

morecomprehensive

diseasemanagem

entas

high

utility

oftheHAT

system

.Atthepilottestingaftereigh

tweeks

ofusingHATat

homeshow

edstatistically

sign

ificantim

prov

ementin

disease-specific

quality

oflifedimension

sof

generalsatisfaction,

selfefficacy,distress,andho

mehassles.

And

also

thestud

yshow

edsign

ificant

improv

ementin

patient

satisfactionwith

treatm

entprocessandhigh

acceptance

ofthe

HATsystem

.Michael

J.Breslow

etal.

[200

4][30]

Hospital

Coh

ort

Beforeandafter

implem

entatio

nDatacollected

from

patients

medical

records

Tomeasure

theeffectsof

telemedicineon

clinical

and

econ

omic

outcom

esin

amulti

site

intensivecare

unit

The

results

ofthisstud

ydemon

strate

improv

edclinical

outcom

esandfinancialperformance.

Patient

mortalityratesforbo

ththeICU

stay

and

hospitalizationwerelower

(9.4%

vs.12

.9%)

whenremoteintensivemanagem

ent

supp

lementedroutinecare.The

averageleng

thof

stay

(LOS)in

ICU

during

thesupp

lemental

care

period

was

lower

(3.63days

vs.4.35

days).

There

was

nochange

intheho

spital’sLOScosts

werefoun

dto

beredu

cedtoo,

asaresultof

both

aredu

ctionin

LOSandadecrease

inthe

daily

costsof

ICU

care.In

additio

n,the

redu

ctionin

ICU’sLOScreatednew

capacity

J Med Syst (2007) 31:397–432 415

that

enabledadditio

nalpatientsto

becaredforin

thetwoICUs.

Marily

nne

Aet

al.

[200

4][67]

Hom

ecare

Survey

After

implem

entatio

nFocus

grou

pswith

9clientsand13

nurses

Interviewswith

9ph

ysicians

and9managers

Toassess

thereadinessof

stakeholders

toadop

tteleho

mecareservices

foradult

diabetic

The

find

ings

ofthisstud

yshow

differencesin

stakeholders’conceptio

nsof

techno

logy,

includ

ingcommon

them

esam

ongclients,

prov

iders,andorganizatio

ns.Nurses,ph

ysicians,

andmanagersshared

twosubjects:on

erelatedto

thebenefitsof

increasedsystem

access,

particularly

forruralclientsandthosewith

mob

ility

limitatio

nsandalso

with

respectto

increasedacuity

andnu

mbers

ofho

mecare

clientsseekingservices,n

ursesandclientsshared

practical

concerns

abou

tthemechanics

oftelemedicineinclud

ingsystem

failu

reandease

ofuse.

Nursesfocusedon

diseasemanagem

entand

supp

ort.Managersandph

ysicians

focusedon

overallethics,system

structured,system

evaluatio

n,andsustainability.

Kaufm

anD.

R.et

al.

[200

4][68]

Hom

ecare

Cog

nitiv

ewalkthrou

gh(usability

testing)

After

implem

entatio

nFiledusability

testingin

25subjects’ho

me:

a4subjectsin

New

YorkCity

and11

inUpstate

New

York

Toidentifybarriers

toeffective

homeTelem

edicineuse

(IDEATel)

The

find

ings

ofthisstud

yshow

that

thereisa

rang

eof

both

cogn

itive

andno

ncog

nitiv

ebarriers

that

wereun

anticipated

bythecogn

itive

walkthrou

ghanalysis.The

evaluatio

ninclud

eda

cogn

itive

walkthrou

ghanalysisto

characterize

task

complexity

andidentifypo

tentialprob

lems

aswellas

fieldusability

testingin

patients’

homes.The

stud

yrevealed

dimension

sof

the

interfacethat

impededop

timal

access

tosystem

resources.Itwas

foun

dsign

ificantob

stacles

correspo

ndingto

perceptual-m

otoric

skills,

mentalmod

elsof

thesystem

,andhealth

literacy.

Won

gY.K

etal[200

5][44]

Hom

ecare

Clin

ical

trials

After

implem

entatio

n–Q

uestionn

aire

completed

by22

patients.–S

econ

dary

outcom

emeasuresby

medical

equipm

entto

assess

patient

outcom

es

Toexplorethefeasibility

and

efficacy

ofan

exercise

prog

ram

forelderlypeop

lewith

knee

pain

cond

uctedviavideocon

ferencing

The

find

ings

ofthestud

yshow

that

therewere

sign

ificantim

prov

ementsin

theprim

aryou

tcom

emeasures.Significant

improv

ementswere

recorded

inallsecond

aryou

tcom

esexcept

rang

eof

motionof

thekn

ee.Anincrease

ofstreng

thof

50%

and46

%(Po0

.001

)was

foun

din

therigh

tandleftqu

adriceps,respectiv

ely.

A21

%redu

ctionin

thetim

etakento

completethe

TUGT(tim

edup

-and

-gotest)was

observed.

There

was

also

sign

ificantim

prov

ementin

the

Tab

le3

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

416 J Med Syst (2007) 31:397–432

know

ledg

etest.Over80

%of

thesubjects

answ

ered

allthequ

estio

nscorrectly,with

anaverageim

prov

ementof

threemarks

inthefinal

evaluatio

n.Over80

%of

theelderlysubjectswho

joined

theprog

ram

agreed

orstrong

lyagreed

abou

tallaspectsof

usingvideocon

ferencing.

Mostof

them

feltthat

thesystem

was

user-

friend

lyandconv

enient.

BratanT

etal[200

5][36]

Hom

ecare

Clin

ical

trials

After

implem

entatio

nInterview

with

twoph

ysicians,4

managers,1career

and1nu

rse

Todeterm

inethefeasibility

and

acceptability

ofho

memon

itoring

The

find

ings

ofthestud

yshow

that

data

transm

ission

was

foun

dto

beun

reliableon

occasion

sandwas

improv

edby

extend

ingthe

wirelessnetworkin

theho

mes.Dataaccess

and

presentatio

nwereconsidered

acceptable,

althou

ghsugg

estio

nsforminor

changeswere

made.

Nodiscernibleim

pact

ondisease

managem

ent,diagno

sisor

care

was

observed

asa

resultof

themon

itoring

.Intervieweesno

ted

severalpo

tentialbenefits.Som

ekeyadvantages

from

interviewees’

pointof

view

that

were

mentio

nedwere:

facilitates

bettercommun

ication

with

thegeneralpractitioner;enablesearly

detectionof

deteriorations;canprov

idea

continuo

uspictureof

patients’

health;the

equipm

entwas

easy

touse.

The

mon

itoring

was

foun

dto

begenerally

acceptable

andfeasibility

was

considered

tohave

been

largelyprov

en.

MairF.S

etal[200

5][37]

Hom

ecare

Rando

mized

controlled

trial

After

implem

entatio

nQuestionn

aire

completed

by22

patientsand14

nurses

Toexplorepatient

andprov

ider

perspectives

onho

metelecare

The

results

ofthisstud

yshow

that

thereare

sign

ificantdifferencesin

perceptio

nbetween

patientsandprov

iderswith

regard

totelecare

encoun

ters

across

allthedo

mains

addressed.

Patientsdemon

stratedmorepo

sitiv

eview

sof

the

telecare

encoun

ters

than

prov

iders.The

prov

iders

weremoreconcernedabou

tthenegativ

eeffects

oftelecare

consultatio

nson

commun

ication.

With

regard

totheaccuracy

ofassessmentof

apatient’smedical

prob

lems,thepatientsbelieved

thatnu

rses

couldun

derstand

theirprob

lemsqu

itewellwhenusingthetelecare

system

,while

incontrastthenu

rses

werefarless

confidentabou

tthisaspect

ofthetelecare

encoun

ters.The

prelim

inarydata

obtained

inthepresentstud

yidentifyamarkeddifference

ofop

inionbetween

patientsandserviceprov

idersabou

ttheutility

andacceptability

oftheho

metelecare

consultatio

ns.

J Med Syst (2007) 31:397–432 417

SScalvini

etal.

[200

5][31]

Hom

ecare

Coh

ort

After

implem

entatio

nDatacollected

from

patient

medical

records

Toinvestigatetheuseand

econ

omic

aspectsof

home-based

telecardiology

inchronicheart

failu

repatients

The

find

ings

ofthestud

yshow

that

therewere

sign

ificantredu

ctionin

hospitalizationand

instability

intheho

me-basedtelecardiology

(HBT)grou

prelativ

eto

theusualcare

grou

p.There

was

aredu

ctionof

24%

inthetotalcosts

afteron

eyear

inthegrou

ps,which

underw

ent

telecardiology.Thiscostredu

ctionwas

presented

afteron

eyear.There

was

anincrease

inthe

quality

oflifein

HBTcomparedto

theusual-care

grou

ps(the

meanfortheMinnesota

living

Questionn

aire

scores

werefoun

dto

be29

and

23.5respectiv

ely).

Th

Von

tetsiano

set

al.

[200

5][42]

Hom

ecare

Quasi

experimental

Beforeandafter

implem

entatio

nThe

econ

omic

data

obtained

from

hospitalrecords

–Questionn

aire

topatients

–Toevaluate

theclinical

usefulness

ofatelemedicine

system

–Toestim

ateecon

omic

benefitsof

interventio

n

The

find

ings

show

asign

ificantim

prov

ement

regardingbo

thusefulness

andecon

omicbenefits.

There

was

adecrease

inho

spitalization,

which

declined

from

37to

only

6as

aconsequenceof

usethesystem

.A

similardecrease

inem

ergency

departmentvisitswas

also

observed;visits

drop

pedfrom

156to

86afterusingthesystem

.The

patient

know

ledg

eabou

tthediseaseand

self-m

anagem

entwerealso

observed

tobe

improv

ed.There

was

asign

ificantim

prov

ement

inpatient

quality

oflifeof

morethan

28%,while

thesatisfactionrate

was

32%

better.Empo

wered

patientsandcaregivers

wereable

tominim

ize

symptom

s,complications,anddisabilitiesandto

increase

autono

myin

everydayliv

ing.

The

total

costswerelower

during

thestud

ythan

before.

The

redu

cedcostsin

thepresentstud

ywere

mainlydu

eto

theredu

ctionin

hospitaladmission

andthelower

leng

thof

stay

ofpatients.

SianM

Nob

leet

al.

[200

5][32]

Hospital

Rando

mized

controlled

trial

Beforeandafter

implem

entatio

n–T

imemeasurement

–Interview

(exp

ertpanel)

–The

econ

omic

data

obtained

from

hospitalsrecords

Cost-consequences

analysisof

minor

injury

telemedicinein

aperiph

eral

emergency

department

The

stud

yshow

sthat

telemedicinehastheability

toredu

cecostssubstantially

only

where

transport

costsin

theabsenceof

telemedicineisvery

high

.The

find

ings

also

revealed

that

unlik

ethe

previous

stud

ies,itwas

foun

dto

have

been

more

costly

from

both

NHSandpatient

view

points.

The

mainreason

sbehind

itweretheneed

for

specialistequipm

ent,high

erstaffcosts,and

high

erfollo

w-upcosts.The

timetakenby

the

Tab

le3

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

418 J Med Syst (2007) 31:397–432

twoconsultatio

nswas

considerably

long

erthan

forgeneralph

ysicianconsultatio

n.Itwas

concludedthat

theprocedural

changesmay

berequ

ired

iftelemedicinewas

tobe

putinto

practice(e.g.theneed

totriage

telemedicine

patientsatthemainho

spital).Itw

ould

affectbo

thnu

rses

andgeneralph

ysicians,bu

titistooearly

topredicttheeffects.

AGary

Linassi

etal.

[200

5][38]

Hospital

Survey

After

implem

entatio

nQuestionn

aire

(open/closeended

questio

ns)to

15patients

Toassess

user

(patient)

satisfactionwith

telemedicine

From

theresultof

thestud

y,itwas

observed

that

telemedicineprov

ides

sign

ificantadv

antagesov

ermoretradition

almetho

dsof

ampu

teeevaluatio

nandcare.Overall,

thepatientssurveyed

were

satisfied

with

theorganizatio

nalstructure.

Telem

edicineseem

edto

representacost-

effective,

practical

solutio

n.Itisacceptable

topatientsandprov

ides

areliableassessmentof

the

ampu

tee.

App

roximately97

%of

therespon

ses

werein

thego

odto

excellent

rang

eshow

inggreat

user

satisfaction.

How

ever,concerns

wereraised

abou

tease

ofaccess

tolocaltelemedicinesites,

conn

ectio

nwaitin

gtim

esandlack

offamiliarity

with

telemedicinetechno

logy.Minor

difficulties

relatedto

technicalaspectswereob

served.

How

ever,itwas

sugg

estedthat

future

developm

entsmight

includ

emorecomprehensive

evaluatio

nsandtreatm

entservices

inthe

commun

ity.

DDavid

Persaud

etal.

[200

5][33]

Primary

care

Survey

Beforeandafter

implem

entatio

nThe

econ

omic

data

extractedfrom

expenserepo

rt–Q

uestionn

aire

(openended)

to21

5patient

Toexam

inethecostof

Telehealth

from

asocietal

perspective

The

find

ings

ofthisstud

yshow

that

thesocietal

costof

anaverageTelehealth

consultatio

nwas

sign

ificantly

high

erthan

thecostof

face-to-face

consultatio

n,even

thou

ghfrom

theperspectiveof

thepatient

Telehealth

services

werecheaper.

How

ever,itwas

also

observed

that

from

asocietal

perspective,theov

erallcostof

prov

iding

Telehealth

was

high

erthan

prov

idingface-to-face

services.F

orthemajority

oftheFCof

Telehealth

thelargestart-up

capitalcostsaccountwas

noted

tobe

morecostly

perconsultatio

n.EricDillon

etal.

[200

5][34]

Primary

care

Coh

ort

After

implem

entatio

n–D

ataob

tained

from

projects

repo

rtsInterview

Toevaluate

costsof

theTelehealth

projectin

Australia

The

authorsconclude

that

theWestern

Australia

Telehealth

projectachieved

itsob

jectives

that

weresign

ificantclinical

utilizatio

n,educational

andtraining

supp

ort,andim

prov

edinform

ation

access.H

owever,itw

asfoun

dthatno

tallclinical

Telehealth

projectsprov

ided

sign

ificantcost

J Med Syst (2007) 31:397–432 419

saving

sto

thehealth

department.Som

eprog

rams

representedanetcostto

thehealth

department

andthesaving

sofferedby

someotherswereon

lymod

est.How

ever,mostprog

rammes

offered

sign

ificantqu

antitativebenefitsandotherlong

erterm

benefitsthan

costsaving

salon

e.Clin

ical

usegrew

by30

%of

allTelehealth

activ

ity.

Edu

catio

nalusewas

increasedby

40%

and

managem

entusefoun

dto

grow

by30

%.The

stud

yalso

demon

stratedthat

theaverage

overhead

costof

each

Telehealth

eventwas

$AU19

2which

comparedfavo

urably

with

the

averagecostpertrip

of$A

279.

Itwas

also

observed

thatthecoordinatorswerecriticalof

the

implem

entatio

nof

individu

alprojectsandplayed

asign

ificantrole

intheov

erallsuccessof

the

projects.

Brian

Jetal.

[200

5][43]

Hospital

and

Hom

eCare

Retrospectiv

erecord

review

After

implem

entatio

nReview

of49

outpatient

records

Tocompare

specific

treatm

entand

outcom

evariablesbetween

mentalhealth

care

via

videocon

ferencingto

care

prov

ided

inperson

The

find

ings

ofthisstud

yshow

thaton

egrou

pwas

seen

viavideoconferencing

(telem

entalhealth

care

[TMHC])while

asecond

grou

pwas

seen

face-to-face

care

(FTFC)andserved

asacontrol

grou

pshow

edthat

theGlobalAssessm

entof

Fun

ctioning

was

sign

ificantly

moreim

prov

edfor

theTMHCgrou

p.Meanchange

inGlobal

Assessm

ento

fFun

ctioning

forTMHC(15.3)

was

sign

ificantly

morethan

meanchange

forFTFC

(8.4).There

wereno

sign

ificantdifferences

betweenthegrou

psin

thenu

mberof

labo

ratories

orstud

iesordered,

selected

mentalstatus

elem

ents,self-helprecommendatio

nsmade,

ornu

mberof

patientsprescribed

twoor

more

psycho

trop

icmedications.The

rate

offull

compliancewith

themedicationplan

andfollo

w-

upappo

intm

entswas

sign

ificantly

betterfor

TMHC.The

find

ingof

stud

yalso

show

edthat

Providers

usingTMHCtold

morepatientsto

return

forfollo

w-upappo

intm

entsin

30days.

Tab

le3

(con

tinued)

Autho

rsDom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

420 J Med Syst (2007) 31:397–432

Tab

le4

DSSevaluatio

nstud

ies

Autho

rsSystem

Dom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

Sho

rtD

etal

[200

3][50]

CDSS

Primary

care

Survey

Beforeand

after

implem

entatio

n

Questionn

aire

andinterview

with

15General

practitioner

Todevelopandevaluate

inprim

ary

care

acompu

terizeddecision

supp

ort

system

forthemanagem

entof

stroke

patients

The

find

ings

show

that

practitioners

werevery

positiv

efortheCDSSto

addressun

certaintyam

ongGPs.

Practition

ersstated

severalpo

tential

benefitsof

thesystem

,includ

ing

helpingthem

toim

prov

eGP–patient

dialog

ueandreinforcingtheir

recommendatio

nmessage

topatients,

theapplicability

andacceptability

ofincorporatingthesystem

into

aprim

arycare

consultatio

n.The

system

was

foun

deasier

touseandto

navigate.The

timeto

access

the

system

was

seen

asapo

sitiv

efactor.

Lim

itatio

nsto

applicability

ofthe

system

wereidentifiedsuch

aspractitionersneedingbasicinform

ation

techno

logy

(IT)skillsandbo

thGPs

andpatientsrequ

iringan

understand

ingof

simplerisk

concepts

arenecessary.

Quantitativ

eresults

furthersugg

estedthatthesystem

made

decision

makingeasier,im

prov

edfeelings

ofbeingsupp

orted,

improv

edthequ

ality

ofdecision

makingand

increasedsatisfaction.

Cornelia

Met

al[200

3][51]

CDSS

Hospital

Con

trol

trial

After

implem

entatio

nClin

icaltrialo

n52

patientswho

assign

edto

interventio

nor

controlcond

ition

Questionn

aire

completed

bythosepatients

–Toevaluate

prelim

inaryeffectsof

acompu

terizedsupp

ortsystem

oncong

ruence

betweenpatients’

repo

rted

symptom

s–T

oinvestigatethesystem

’sease

ofuse,

timerequ

irem

ents,andpatient

satisfaction.

The

find

ings

ofthestud

yreveal

that

therewereno

sign

ificantdifferencesin

numbers

ofrepo

rted

symptom

sin

the

interventio

nandcontrolgrou

ps.There

werelargevariations

inpatients’

repo

rtsof

frequency,

severity,degree

ofbo

ther

ofsymptom

s,and

impo

rtance.Alsothestud

yshow

sthat

while

patientsin

both

grou

pswere

equivalent

atbaselin

ein

symptom

characteristics,thereweresign

ificantly

greatercong

ruence

betweenpatients’

repo

rted

symptom

sandthose

addressedby

theirclinicians

inthe

J Med Syst (2007) 31:397–432 421

experimentalgrou

p.The

system

scored

high

onease

ofuse.There

were

nosign

ificantgrou

pdifferencesin

patient

satisfaction.

Beuscart-

zeph

irM.

Cet

al[200

4][45]

CPOE

Hospital

Survey

After

implem

entatio

nSem

istructured

andstructured

interviewsof

physicians

and

nurses

Observatio

non

users’

interactions

with

patient

recordsDocum

entanalysis

Tostud

yof

CPOE’smedication

administrationfunctio

nswith

focus

impact

onph

ysician–

nurse

coop

eration.

The

find

ings

ofthestud

yreveal

that

the

nursegetsthemedicationorderson

her

care

plan

andmedication

administrationrecord.Whentheorder

seem

sun

clearor

difficultto

administer,shemay

asktheph

ysician,

ifhe

isavailable.

On-site

observations

inthepaper-do

cumentedsituation

sugg

estedthat

thedo

ctorsareno

tinterested

anddo

n’tfeel

committed

with

theexactplanning

ofdrug

administration,

which

they

ordinarily

leaveto

thenu

rses.The

comparisonof

coop

erationmod

elsin

both

situation

show

sthat

userstend

toadop

ta

distribu

teddecision

makingparadigm

inthepaper-basedsituation,

while

the

CPOEsystem

supp

ortsacentralized

decision

makingprocess.The

usability

assessmentun

coveredusability

prob

lemsfortheentryof

medication

administrationtim

eschedu

lingby

the

physicianandrevealed

that

the

inform

ationcanbe

ambigu

ousforthe

nurse.

Joan

S.

Ash

etal.

[200

4][69]

CPOE

Hospital

Survey

After

implem

entatio

n–S

emistructuredinterviewwith

housestaff(phy

sicians,

nurses,ph

armacists,

administrators,inform

ation

techno

logy

staff)accompany

with

observation

Toidentifysuccessfactorsconcerning

theim

plem

entatio

nof

CPOE

The

find

ings

ofthestud

yreveal

that

the

direct

entryof

medical

orders

bythehealthcare

decision

maker

andits

otheru

sersincreasedyearby

year

during

the5-yearstudy.Itwasalso

noted

thatCPO

Eisperceivedto

reduce

medicalerrors.T

hestudy

hasproduced

detaileddescriptions

offactorsrelatedto

CPOE’ssuccesssuch

as(1)computertechnology

principles

(which

include:temporalconcerns;

Tab

le4

(con

tinued)

Autho

rsSystem

Dom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

422 J Med Syst (2007) 31:397–432

technology

andmeetin

ginform

ation

needs);m

ultid

imensionalintegration;

andcosts.(2)Personalprinciples:v

alue

tousers,training,and

support.(3)

Organizationalp

rinciplessuch

ascollaborativ

eprojectm

anagem

ent;

term

s,concept;im

provem

entthrough

evaluatio

n).(4)

Environmentalissues

provided

themotivationandcontextfor

implem

entin

gsuch

system

.How

ever,

good

training

andsupportw

eredefined

astheprerequisitesforbetterresults.

JosAarts

etal.

[200

4][60]

CPOE

Hospital

Survey

After

implem

entatio

n–U

nstructuredinterview

with

25staffmem

bers

–Observing

thesystem

inuse

Tostud

ytheInterrelationshipof

technicalandsocial

elem

entsand

how

CPOEfitinto

medical

work

practices

The

find

ings

ofthisstud

yweredivided

into

twocatego

ries:i.e.socioand

techno

.The

societal

aspectswere

prom

inentin

theexperience

ofthe

administrativeperson

alrelatedto

the

system

intheho

spitaland

theirattitud

etowards

it.Itwas

feared

that

greater

amou

ntsof

timeandcostwou

ldbe

requ

ired

intheim

plem

entatio

nprocess.Thisstud

yalso

show

sthatthe

implem

entatio

nof

CPOEin

twoDutch

hospitalsprov

edto

bevery

much

prob

lematic.In

oneho

spitalCPOE

neverbecamefunctio

nal,andin

the

other,CPOEwas

only

used

bythe

nursingandadministrativestaff.Toa

largeextent

socio-technicalissues

influenced

theim

plem

entatio

n,such

astheexistenceof

localinfrastructures

andworking

arrang

ements.These

are

noteasily

changedas

aresultof

new

techno

logies.Researchers

sugg

ested

that

introd

ucingnew

techno

logies

that

influencemedical

workpractices

and

organizatio

nalarrang

ementsshou

ldbe

carefully

plannedformeasurementof

organizatio

nalim

pact

tomakeiteasier

andaccurate.

Cho

iS.S

etal

[200

4][57]

CPOE

Hospital

Survey

Beforeandafter

implem

entatio

n–C

ompu

terandpaperbased

documentscomparison

–Interview

with

anu

rse

Toevaluatio

nof

aweb-based

nurse

orderentrysystem

fortuberculosis

patientsin

Peru

Inthisstud

yitwas

foun

dthatthereisno

sign

ificantdifference

inthecontrol

grou

ps’ou

tputs.The

interventio

ngrou

pshow

edasign

ificantdrop

inthe

J Med Syst (2007) 31:397–432 423

errorrate

afterthesametim

einterval.

Alsoin

thisstud

yitwas

foun

dthatthe

nurses

affirm

edthesystem

’susability

andease

touseandsimplicity

and

ability

toview

allof

theinform

ation

regardingapatient’smedication

historyon

onepage.The

web

system

show

sapatient’sentireregimen

historyfrom

theinitial

regimen

tothe

present.Add

ition

ally,theweb

system

allowed

thenu

rseto

seeallof

the

changesat

onetim

e.RobertL

.Ohsfeldt

etal.

[2004]

[54]

CPOE

Hospital

Sim

ulation

Before

implem

entatio

nQuestionn

aire

distribu

tedto

74generalho

spitals

–The

econ

omic

data

obtained

from

expenserepo

rts

–Toestim

atethecostof

implem

entin

gCPOEin

hospitalsin

aruralstate

–Toevaluate

thefinancialim

plications

ofCPOE’snatio

nwide

implem

entatio

n

The

findings

show

aconsistent

pattern

ofgreaterusein

urbanhospitals.In

additio

n,theextent

ofintegrationacross

clinicalfunctio

nswas

foundto

begreaterin

urbanhospitals.T

hefinancial

results

ofthisstudyshow

edthatcostsfor

CPO

Eim

plem

entatio

nwereabout$

1.3

millionforcriticalaccessandrural

hospitals,$2millionforruralreferral,

and$1.9millionforurbanhospitals

underthelowcostscenario.F

orrural

andcriticalaccesshospitals,theseCPOE

implem

entatio

ncostswould

representa

substantialexpansion

overall.CPO

Eim

plem

entatio

nwould

increase

operatingcostsforruraland

critical

access

hospitalsin

theabsenceof

substantialcostsavings

associated

with

improved

efficiency

orim

proved

patient

safety.Itw

asalso

observed

thatthecost

impactisless

dram

aticbutstill

substantialfor

theurbanandrural

referralhospitals.

Benjamin

Bog

ugki

etal.

[200

4][55]

CDSS

Hospital

Quasi-

experimental

Beforeandafter

implem

entatio

nDataob

tained

from

patient

medical

records

Toexam

inetheim

pact

ofacompu

ter-

basedreminderin

addressing

adrug

shortage

The

stud

yshow

sa55

%decrease

inthe

numberof

orders,thus

inthepo

stim

plem

entatio

nperiod

,thereisan

increase

inthealternativemedication.

Inadditio

n,thisstud

yshow

scost

Tab

le4

(con

tinued)

Autho

rsSystem

Dom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

424 J Med Syst (2007) 31:397–432

redu

ction(ann

ualsaving

:$3

6,55

2)for

theinstitu

tiondu

eto

the

implem

entatio

nof

thedrug

reminder

system

.Resultsshow

that

CDSSisan

effectivetool

inconserving

medications

during

times

ofnatio

nal

drug

shortagesby

rapidlychanging

prescribingpractices.

William

L.

Galanter

etal.

[200

4][46]

CPOE

Hospital

Coh

ort

Beforeandafter

implem

entatio

nDataob

tained

from

documentatio

nTomeasure

theeffectsof

alertsfor

inpatient

digo

xinuse(w

ithCPOE)

The

find

ings

ofthisstud

yshow

that

the

alertsgenerally

actedto

improv

ebo

ththespeedandov

erallmagnitude

ofapprop

riateclinicianrespon

seto

aclinical

situation.

Thisstud

yshow

sthat

alerts,generatedby

unsafe

medication,

improv

edthesafe

useof

digo

xin.

DuringCPOE,alerts

associated

with

missing

levelswere

effective.

For

hypo

kalemia

and

hypo

magnesaem

ia,thealertsgiven

during

CPOEwereno

tas

effectiveas

thosegivenat

thetim

eof

newly

repo

rted

low

electrolytes.During

CPOE,supp

lementatio

nfor

hypo

kalemia

was

notim

prov

edwhereas

supp

lementatio

nfor

hypo

magnesaem

iaim

prov

edin

one

hour.Itwas

also

observed

that

asyn

chrono

usalertsweresuccessful

inchanging

clinicianbehaviou

r,whereas

synchron

ouswereno

t.Itwas

observed

that

therespon

seto

thealertwas

depend

entup

onbo

ththemannerin

which

thealertwas

commun

icated

totheclinician,

synchron

icor

asyn

chrono

us,andtheclinical

context

ofthealert.

Joan

S.

Ash

etal.

[200

4][70]

CPOE

Hospital

Cross

sectional

After

implem

entatio

nQuestionn

aire

(openended

questio

n)distribu

ted96

4ho

spitals

Toassess

the

–Availabilityof

inpatient

compu

terized

physicianorderentry

–The

degree

towhich

physicians

are

usingit.

Thisstud

ydemon

stratesthat

CPOEstill

does

notem

ploy

widespread

implem

entatio

nacross

theU.S.

hospitals.The

results

ofthisstud

yshow

that

CPOEwas

notavailableto

83.7%

oftheho

spitals.Only9.7%

ofU.S.ho

spitalspresently

have

it.In

approx

imatelyhalfof

thoseho

spitals,

J Med Syst (2007) 31:397–432 425

morethan

90%

ofph

ysicians

use

CPOEandin

onethirdof

them

,more

than

90%

oftheorders

areenteredvia

CPOE.T

hedataalso

show

edthatthere

isatrendtowards

ahigh

erpercentage

ofho

spitalswith

CPOErequ

iringits

useov

erthepast5years,mov

ing

towards

concom

itant

trends

inhigh

erparticipationandsaturatio

nprop

ortio

ns.

Rob Shu

lman

etal.

[200

5][47]

CPOE

Hospital

Coh

ort

Beforeandafter

implem

entatio

nDatafrom

themedicationerrors

recorded

before

andafter

implem

entatio

nCPOE

Tocompare

theim

pact

ofCPOEwith

hand

writtenprescriptio

nsbasedon

thefrequency,

type

andou

tcom

eof

medicationerrors

inICU

Throu

ghthisstud

yitwas

demon

strated

that

theintrod

uctio

nof

CPOEwas

associated

with

:aredu

ctionin

the

prop

ortio

nof

medicationerrors

(from

6.7%

to4.8%

);an

increase

insome

typesof

errors

likedo

seerror,

omission

ofrequ

ired

drug

s.Itwas

notedthat

theprescriber’ssign

ature

mustbe

added.

There

was

anim

prov

ementin

theov

erallpatient

outcom

escore(the

meanAcute

Phy

siolog

yAnd

Chron

icHealth

Evaluation—

APA

CHE—IIscores)

with

CPOE.

ParkR.W

.et

al[200

5][52]

CPOE

Hospital

Survey

After

implem

entatio

n–Q

uestionn

aire

completed

bydirectorsof

hospital

inform

ationsystem

sin

122

hospitals(30teaching

hospitalsand92

general

hospitals)

Todeterm

inetheavailabilityof

CPOE

andEMRsystem

sin

teaching

and

generalho

spitalsin

therepu

blic

ofKorea

From

thisstud

y,itwas

foun

dthat

the

CPOEsystem

swereavailableat

all

locatio

nwith

intheho

spitalforall

typesof

orders

in80

.3%

ofallthe

hospitals.93

,3%

ofthesurveyed

teaching

hospitalshadcompleted

CPOEsystem

sbu

tgeneralho

spitals

didin

76,1%.86

%of

hospitalsreplied

that

theuseof

CPOEwas

mandatory;

9,3%

said

that

theho

spitalencouraged

itsuse,

and4,7%

said

that

theusewas

optio

nal.96

,7%

oftheteaching

hospitalsrequ

ired

theirph

ysicians

toorderby

CPOE.A

majority

ofph

ysicians

enteredmostof

their

medical

orders

into

CPOEsystem

.

Tab

le4

(con

tinued)

Autho

rsSystem

Dom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

426 J Med Syst (2007) 31:397–432

Only9%

oftheho

spitalsgetavailable

acompleteEMR

system

.Teaching

hospitalstend

edto

demon

strate

ahigh

erpercentage

ofEMRusethan

did

generalho

spitals.With

consideringall

form

sof

EMRs,includ

ingpartial,two

thirds

oftheteaching

hospitalshave

leastsomeform

ofEMRs.

Tejal

Ket

al[200

5][53]

CPOE

Primary

care

Prospectiv

ecoho

rtstud

yAfter

implem

entatio

n18

79Prescriptionreview

,66

1patient

survey,and65

3chart

review

Toassess

therates,types,andseverity

ofou

tpatient

prescribingerrors

and

understand

thepo

tentialim

pact

ofcompu

terizedprescribing.

The

results

ofthestud

yshow

thatof

the

prescriptio

ns,14

3(7.6%)containeda

prescribingerror.Three

errors

ledto

preventableADEsand62

(43%

;3%

ofallprescriptio

ns)hadpo

tentialfor

patient

injury

(potentialADEs);on

ewas

potentially

lifethreatening(2%)

and15

wereseriou

s(24%

).Errorsin

frequency(n=77

,54

%)anddo

se(n=

26,18

%)werecommon

.The

ratesof

medicationerrors

andpo

tentialADEs

wereno

tsignificantly

differentatbasic

compu

terizedprescribingsites(4.3%

vs11.0%,P=.31;

2.6%

vs4.0%

,P

=.16)

comparedto

hand

writtensites.

Adv

ancedchecks

(including

dose

and

frequencychecking

)couldhave

prevented95

%of

potentialADEs.

William

L.

Galanter

etal.

[200

5][48]

CPOE

Hospital

Coh

ort

Beforeandafter

implem

entatio

nDataob

tained

from

thepatient

records

Toevaluate

theutility

ofalertsas

well

asanalyzethefactorsthat

may

play

arole

inno

ncom

pliancewith

alert

recommendatio

nsam

ongho

usestaff

inorderto

improv

eCDSin

thefuture

The

results

ofthestud

yshow

that

the

alerts(generated

byun

safe

medication)

wereeffectivein

decreasing

theorder

andadministrationof

drug

scontradictions

dueto

renalinsuffiency.

The

likelihoo

dof

apatient

receivingat

leaston

edo

seof

contradicted

medicationdecreasedfrom

89%

to47

%afteralertim

plem

entatio

n,which

show

sa42

%redu

ctionin

administration.

Itwas

notedthat

this

redu

ctionwas

almostentirelydu

eto

thecancellatio

nof

theorderafter

view

ingthealert,which

occurred

in41

%of

themistakesin

which

analert

was

given.

The

timeof

thedaywas

not

foun

dto

besign

ificantly

associated

with

thelik

elihoo

dof

alert

J Med Syst (2007) 31:397–432 427

compliance.

The

stud

yalso

revealed

that

compliancewith

thealertswas

high

erin

malepatients,was

notedto

increase

with

thedu

ratio

nof

house

stafftraining

,andincreasedin

patients

with

moresevere

renaldy

sfun

ction.

Ross

Kop

pel

etal.

[200

5][61]

CPOE

Hospital

Survey

After

implem

entatio

n–Interview

with

housestaffand

inform

ationtechno

logists,

nurses,ph

armacists,and

physicians

–Observatio

n(hou

sestaff,

nurses

andph

ysicians)–

Questionn

aire

toho

usestaff

Toidentifytherole

ofCPOEin

facilitatingprescriptio

nerrorrisk

The

find

ings

ofthestud

ydemon

strate

that

awidelyused

CPOEsystem

facilitated

22typesof

errorrisks.For

exam

ple,

fragmentedCPOEdisplays

mistakesfordo

sage

guidelines,

igno

redantib

iotic

renewals,no

tices

placed

onpaperchartsrather

than

intheCPOEsystem

,separatio

nof

functio

nsthat

facilitatedo

uble

dosing

andincompatib

leorders,and

inflexible

ordering

form

atsgeneratin

gwrong

orders.Three

quarters

oftheho

use

staffrepo

rted

observingeach

ofthese

errorrisksandindicatedthatthey

were

occurringweeklyor

moreoften.

Itwas

also

observed

that

manyerrors

occurred

frequently.Itwas

sugg

ested

that

asCPOEsystem

sare

implem

ented,

clinicianandho

spitals

mustattend

toerrorsthatthesesystem

scausein

additio

nto

errors

that

they

prevent.

Jaap

L.

Van

den

brink

etal.

[200

5][56]

DSS

Hospital

Coh

ort

After

implem

entatio

n–Q

uestionn

aire

topatientsdata

obtained

from

automated

logg

ingon

system

Todeterm

ineuse,

appreciatio

n,and

effectivenessof

aninform

ation

system

The

results

ofthestudyshow

thepatient

used

thesystem

intensively(averaging

4–5tim

esaweek).Itw

asobserved

that

thepatient

appreciatedthesystem

highly.89%

werenoticed

toadvise

other

patients.Patient

satisfactionquestio

nsscored

high

percentagesbetween91–

94%.M

ostp

atientswerenotedto

learn

thesystem

easily

(with

apercentage

of61%).The

system

detected

patient

problemsthathadnotb

eendiscovered

during

regularoutpatient

visits.

Tab

le4

(con

tinued)

Autho

rsSystem

Dom

ain

Study

design

Tim

eof

evaluatio

nSou

rceof

evidence

Aim

ofstud

yFinding

428 J Med Syst (2007) 31:397–432

analysis should be used to investigate influences of the newsystems on the organisation and its personnel.

In our review we found that during the period 2003–2005, most of the evaluations aimed to include issues suchas the effectiveness of the systems, the quality of care, userand patient satisfaction, and the system’s usability. And alsothe evaluation studies tended to use subjective approachescombined with quantitative studies in order to analysingcost and benefits.

Most of the studies included in this paper have usedsurvey methodology as their research method. Some of thestudies focused upon financial analysis, such as a cost andbenefit or consequences approach. Some of the studies useda clinical trial or cohort study to research the systems’outputs. It is, however, important to note that it has beendifficult to find generalized models and methods to evaluateIT-based applications in clinical settings that cover all as-pects such as economic and inter- and intra-organizationalapproaches. It seems that there are no clear guidelines forevaluation studies to develop evaluations and obtain moreclear and exact feedback about the implemented systems.

A key requirement for analytical work is a cleardefinition of what constitutes an IT-based application.Today there are different terminologies relating to similarsystems. Analysis of the effects has predominantly takenplace at an organizational level. However, this does not takeaccount of the differences in the settings in which analysisis taking place.

A significant barrier to investment in IT in healthcare isthe widely recognized fact that any cost saving resultingfrom technology changes is not always seen by theimplementer, but is rather passed on to a third party. Inessence, benefits appear at one site and in one budget, whilea large share of the cost commitments appear at another siteand in another budget. To our best knowledge, theevaluation studies performed do not include any discussionabout this important issue, or how lack of incentive to adoptsystems can influence the organization and its personnel.

As in other areas, such as healthcare for the elderly [62,63], evaluation studies in general pay more attention to thetechnical aspects and/or direct costs of the technology inuse. Further, the potential effects of the implementation ofIT-based applications are identified without analysing themfrom an inter-organizational and economic perspective.

In addition to difficulties in quantifying the output of theuse of IT in healthcare, evaluative challenges in assessingthe impact of IT include isolating its impacts from others. Itis difficult to find a clear relationship between IT,organizational improvements, quality of care and benefitsrealization.

The most common type of analytical approach usedtakes the form of a case study, and usually retrospectiveones that try to measure the actual impact of an ITV

itali

Sintchenk

oet

al.

[200

5][49]

DSS

Hospital

Coh

ort

Beforeandafter

implem

entatio

nDataob

tained

from

thepatient

records

Toassess

theeffect

ofacompu

ter-

baseddecision

supp

ortsystem

onantib

iotic

useandpatient

outcom

ein

acriticalcare

unit

The

results

ofthisstud

ysugg

estthatthe

useof

theDSScontribu

tedto

the

redu

ctionof

patient

leng

thof

stay

intheICU.The

introd

uctio

nof

theDSS

was

foun

dto

have

been

associated

with

aredu

ctionin

antib

iotic

usagein

theICU.Itwas

also

foun

dto

have

brou

ghtachange

inpatternsof

antib

ioticsusein

theICU.Itwas

also

demon

stratedthroug

hthisstud

ythat

therewas

difficulty

with

theuseof

the

DSSam

ongclinicians

with

different

roles.

J Med Syst (2007) 31:397–432 429

investment or prospective assessments. The evaluationstherefore suffer two main weaknesses. First, it is difficult toattribute IT investments to observed effects, because it isusually not possible to apply case-control design to theevaluative study. Second, it is difficult to generalize fromthe particular study to the broader context. The results ofour review show that the study of the impact of IT inhealthcare is currently limited. No studies have beenconducted to explore the impact of IT on the system as awhole. Few organizational-level studies exist that focusexplicitly upon the impact of IT in a specific sector of care.This review therefore suggests that there is an increasingneed to share knowledge and find methods to evaluate theimpact of investments, and in parallel with this, formulateindicators for success. It is therefore interesting to developor extend evaluation methods that can be applied to thisarea, but also that have a multi-actor perspective in order tounderstand the effects, consequences and prerequisites thathave to be achieved for the successful implementation anduse of IT in healthcare.

References

1. Kaplan, B., Evaluating informatics applications—some alternativeapproaches: theory, social interactionism, and call for methodo-logical pluralism. Int. J. Med. Inform. 64:39–56, 2001.

2. Ammenwerth, E., Brender, J., Nykanen, P., Prokosch, H. U.,Rigby, M., and Talmon, J., Visions and strategies to improveevaluation of health information systems. Reflections and lessonsbased on the HIS-EVAL workshop in Innsbruck. Int. J. Med.Inform. 73:479–491, 2004.

3. Ammenwerth, E., Iller, C., and Mansmann, U., Can evaluationstudies benefit from triangulation? A case study. Int. J. Med.Inform. 70:237–248, 2003.

4. Andersen, J. G., Aydin, C. E., and Jay, S. J., EvaluatingHealthcare Information Systems: Methods and Applications.Thousand Oaks, CA: Sage publication, 1994.

5. Friedman, C. P., and Wyatt, J. C., Evaluation Methods in MedicalInformatics. New York: Springer, 1996.

6. Kaplan, B., and Shaw, N. T., Future directions in evaluationresearch: People, organizational, and social issues. Methods Inf.Med. 43:215–231, 2004.

7. Phil, W., An assessment of strategies for information systemevaluation: Lessons for education. Int. J. Educ. Res. 25:361–371,1996.

8. Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., andLang, T., A systematic review of computer-based patient recordsystems and quality of care: More randomized clinical trials or abroader approach. Int. J. Qual. Health Care. 16:407–416, 2004.

9. Wang, S. J., Middleton, B., Bardon, C. G., Spurr, C. D., Carchidi, P. J.,Kittler, A. f., Goldszar, R. C., Fairchild, D. G., Sussman, A. J.,Kuperman, G. J., andBates, D.W., A cost-benefit analysis of electronicmedical records in primary care. Am. J. Med. 114:397–403, 2003.

10. Laerum, H., Karlsen, T. H., and Faxvaag, A., Effects of scanning andeliminating paper-based medical records on hospital physicians’clinical work practice. J. Am. Med. Inform. Assoc. 10:160–588, 2003.

11. Ruland, C. M., and Ravn, I. H., Usefulness and effects on costsand staff management of a nursing resource managementinformation system. J. Nurs. Manag. 11:208–215, 2003.

12. Houston, T. K., Ray, M. N., Crawford, M. A., Giddens, T., andBerner, E. S., Patient perceptions of physician use of handheldcomputers. AMIA Annu. Symp. Proc. 299–303, 2003.

13. Ammenwerth, E., Mansmann, U., Iller, C., and Eichstädter, R.,Factors affecting and affected by user acceptance of computer-based nursing documentation: Results of a two-year study. J. Am.Med. Inform. Assoc. 10:69–84, 2003.

14. Zhang, W. P., Yamauchi, K., Mizuno, S., Zhang, R., and Huang,D. M., Analysis of cost and assessment of computerized patientrecord systems in Japan based on questionnaire survey. Med.Inform. Internet Med. 29:229–238, 2004.

15. Laerum, H., Karlsen, T. H., and Faxvaag, A., Use of and attitudesto a hospital information system by medical secretaries, nursesand physicians deprived of the paper-based medical record: Acase report BMC Med. Inform. Decis. Mak. 4:18, 2004.

16. Hier, D. B., Rothschild, A., LeMaistre, A., and Keeler, J.,Differing faculty and housestaff acceptance of an electronic healthrecord one year after implementation. Medinfo. 11:1300–1303,2004.

17. Rotich, J. K., Hannan, T. J., Smith, F. E., Bii, J., Odero, W. W.,Vu, N., Mamlin, B. W., Mamlin, J. J., Einterz, R. M., and Tierney,W. M., Installing and implementing a computer-based patientrecord system in sub-Saharan Africa: The Mosoriot MedicalRecord System. Med. Inform. Internet Med. 10:295–303, 2003.

18. Stengel, D., Bauwens, K., Walter, M., Kopfer, T., and Ekkernkamp,A., Comparison of handheld computer-assisted and conventionalpaper chart documentation of medical records. A randomized,controlled trial. J. Bone Joint Surg. Am. 86:553–560, 2004.

19. Likourezos, A., Chalfin, D. B., Murphy, D. G., Sommer, B.,Darcy, K., and Davidson, S. J., Physician and nurse satisfactionwith an Electronic Medical Record System. J. Emerg. Med.27:419–424, 2004.

20. Pagliari, C., Gilmour, M., and Sullivan, F., Evaluating nationalelectronic clinical communications implementation in Scotland.Medinfo. 11:1794, 2004.

21. Baum, A., Figar, S., Serverino, J., Assale, D., Schachner, B.,Otero, P., Luna, D., and de Quiros, F. G., Assessing the impact ofchange in the organization of a technical support system for ahealth information systems (HIS). Medinfo. 11:1367–1370, 2004.

22. Hobson, J. C., Khemani, S., and Singh, A., Prospective audit ofthe quality of ENT emergency clinic notes before and afterintroduction of a computerized template. J. Laryngol. Otol.119:264–266, 2005.

23. Kaushal, R., Blumenthal, D., Poon, E. G., Jha, A. K., Franz, C.,Middleton, B., Glaser, J., Kuperman, G., Christino, M., Fernandopulle,R., Newhouse, J. P., and Bates, D. W., The costs of a nationalhealth information network. Ann. Intern. Med. 143:165–173,2005.

24. Raja, E. E., Mahal, R., and Masih, V. B., An exploratory study toassess the computer knowledge, attitude and skill among nurses inhealthcare setting of a selected hospital in Ludhiana, Punjab,India. Medinfo. 11:1304–1307, 2004.

25. Clayton, P. D., Naus, S. P., Bowes, W. A., Madsen, T. S., Wilcox,A. B., Orsmond, G., Rocha, B., Thornton, S. N., Jones, S.,Jacobsen, C. A., Udall, M. R., Rhodes, M. L., Wallace, B. E.,Cannon, W., Gardner, J., Huff, S. M., and Leckman, L., Physicianuse of electronic medical records: issues and successes with directdata entry and physician productivity. AMIA Annu. Symp. Proc.141–145, 2005.

26. White, M., Computerised versus handwritten records. Paediatr.Nurs. 17:15–18, 2005.

27. Wilcox, A. B., Jones, S. S., Dorr, D. A., Cannon, W., Burns, L.,Radican, K., Christensen, K., Brunker, C., Larsen, A., Narus, S.P., Thornton, S. N., and Clayton, P. D., Use and impact of acomputer-generated patient summary worksheet for primary care.AMIA Annu. Symp. Proc. 824–828, 2005.

430 J Med Syst (2007) 31:397–432

28. Makela, K., Virjo, I., Aho, J., Kalliola, P., Koivukoski, A. M.,Kurunmaki, H., Kahara, M., Uusitalo, L., Valli, M., Vuotari, V.,and Ylinen, S., Electronic patient record systems and the generalpractitioner: an evaluation study. J. Telemed. Telecare. 11:S2:66–68, 2005.

29. Zurita, L., and Nohr, C., Patient opinion–EHR assessment fromthe users perspective. Medinfo. 11:1333–1336, 2004.

30. Breslow, M. J., Rosenfeld, B. A., Doerfler, M., Burke, G., Yates,G., Stone, D. J., Tomaszewicz, P., Hochman, R., and Plocher, D.W., Effect of a multiple-site intensive care unit telemedicineprogram on clinical and economic outcomes: an alternativeparadigm for intensivist staffing. Crit. Care Med. 32:31–38, 2004.

31. Scalvini, S., Capomolla, S., Zanelli, E., Benigno, M., Domenighini,D., Paletta, L., Glisenti, F., and Giordano, A., Effect of home-basedtelecardiology on chronic heart failure: Costs and outcomes. J.Telemed. Telecare. 11:S1:16–18, 2005.

32. Noble, S. M., Coast, J., and Benger, J. R., A cost-consequencesanalysis of minor injuries telemedicine. J. Telemed. Telecare.11:15–19, 2005.

33. Persaud, D. D., Jreige, S., Skedgel, C., Finley, J., Sargeant, J., andHanlon, N., An incremental cost analysis of telehealth in NovaScotia from a societal perspective. J. Telemed. Telecare. 11:77–84,2005.

34. Dillon, E., Loermans, J., Davis, D., and Xu, C., Evaluation of thewesterrn Australian Department of Health telehealth project.J. Telemed. Telecare. 11:S2:19–S2:121, 2005.

35. Larcher, B., Arisi, E., Berloffa, F., Demichelis, F., Eccher, C.,Galligioni, E., Galvagni, M., Martini, G., Sboner, A., Tomio, L.,Zumiani, G., Graiff, A., and Forti, S., Analysis of user-satisfactionwith the use of a teleconsultation system in oncology. Med.Inform. Internet. Med. 28:73–84, 2003.

36. Bratan, T., Clarke, M., Jones, R., Larkworthy, A., and Paul, R.,Evaluation of the practical feasibility and acceptability of homemonitoring in residential homes. J. Telemed. Telecare. 11(Suppl 1):29–31, 2005.

37. Mair, F. S., Goldstein, P., May, C., Angus, R., Shiels, C., Hibbert,D., O’Connor, J., Boland, A., Roberts, C., Haycox, A., andCapewell, S., Patient and provider perspectives on home telecare:preliminary results from a randomized controlled trial. J. Telemed.Telecare. 11(Suppl 1):95–97, 2005.

38. Linassi, A. G., and Li Pi Shan, R., User satisfaction with atelemedicine amputee clinic in Saskatchewan. J. Telemed. Tele-care. 11:414–418, 2005.

39. Young, T. L., and Ireson, C., Effectiveness of school-basedtelehealth care in urban and rural elementary schools. Pediatrics.112:1088–1094, 2003.

40. Baldwin, L., Clarke, M., Hands, L., Knott, M., and Jones, R., Theeffect of telemedicine on consultation time. J. Telemed. Telecare.9:S1:71–73, 2003.

41. Finkelstein, J., Khare, R., and Ansell, J., Feasibility and patients’acceptance of home automated telemanagement of oral anti-coagulation therapy. Proc. AMIA Symp. 230–234, 2003.

42. Vontetsianos, T., Giovas, P., Katsaras, T., Rigopoulou, A.,Mpirmpa, G., Giaboudakis, P., Koyrelea, S., Kontopyrgias, G.,and Tsoulkas, B., Telemedicine-assisted home support for patientswith advanced chronic obstructive pulmonary disease: preliminaryresults after nine-month follow-up. J. Telemed. Telecare. 11:S1:86–S1:88, 2005.

43. Grady, B. J., and Melcer, T., A retrospective evaluation ofTeleMental Healthcare services for remote military populations.Telemed J E Health. 11:551–558, 2005.

44. Wong, Y. K., Hui, E., and Woo, J., A community-based exerciseprogramme for older persons with knee pain using telemedicine.J. Telemed. Telecare. 11:310–315, 2005.

45. Beuscart-Zephir, M. C., Pelayo, S., Degoulet, P., Anceaux, F.,Guerlinger, S., and Meaux, J. J., A usability study of CPOE’s

medication administration functions: impact on physician-nursecooperation. Medinfo. 11:1018–1022, 2004.

46. Galanter, W. L., Polikaitis, A., and DiDomenico, R. J., A trial ofautomated safety alerts for inpatient digoxin use with computerizedphysician order entry. J. Am. Med. Inform. Assoc. 11:270–277, 2004.

47. Shulman, R., Singer, M., Goldstone, J., and Bellingan, G.,Medication errors: a prospective cohort study of hand-writtenand computerised physician order entry in the intensive care unit.Crit. Care. 9:R516–R521, 2005.

48. Galanter, W. L., Didomenico, R. J., and Polikaitis, A., A trial ofautomated decision support alerts for contraindicated medicationsusing computerized physician order entry. J. Am. Med. Inform.Assoc. 12:269–274, 2005.

49. Sintchenko, V., Iredell, J. R., Gilbert, G. L., and Coiera, E.,Handheld computer-based decision support reduces patient lengthof stay and antibiotic prescribing in critical care. J. Am. Med.Inform. Assoc. 12:398–402, 2005.

50. Short, D., Frischer, M., and Bashford, J., The development andevaluation of a computerised decision support system for primarycare based upon ‘patient profile decision analysis’. Inform. Prim.Care. 11:195–202, 2003.

51. Ruland, C. M., White, T., Stevens, M., Fanciullo, G., and Khilani,S. M., Effects of a computerized system to support shared decisionmaking in symptom management of cancer patients: Preliminaryresults. J. Am. Med. Inform. Assoc. 10:573–579, 2003.

52. Park, R. W., Shin, S. S., Choi, Y. I., Ahn, J. O., and Hwang, S. C.,Computerized physician order entry and electronic medical recordsystems in Korean teaching and general hospitals: Results of a2004 survey. J. Am. Med. Inform. Assoc. 12:642–647, 2005.

53. Gandhi, T. K., Weingart, S. N., Seger, A. C., Borus, J., Burdick,E., Poon, E. G., Leape, L. L., and Bates, D. W., Outpatientprescribing errors and the impact of computerized prescribing.J. Gen. Intern. Med. 20:837–841, 2005.

54. Ohsfeldt, R. L., Ward, M. M., Schneider, J. E., Jaana, M., Miller,T. R., Lei, Y., and Wakefield, S., Implementation of hospitalcomputerized physician order entry systems in a rural state:Feasibility and financial impact. J. Am. Med. Inform. Assoc.12:20–27, 2005, Jan.

55. Bogucki, B., Jacobs, B. R., and Hingle, J., Computerizedreminders reduce the use of medications during shortages.J. Am. Med. Inform. Assoc. 11:278–280, 2004.

56. Van den Brink, J. L., Moorman, P. W., de Boer, M. F., Pruyn, J. F.,Verwoerd, C. D., and van emmel, J. H., Involving the patient: aprospective study on use, appreciation and effectiveness of aninformation system in head and neck cancer care. Int. J. Med.Inform. 74:839–849, 2005.

57. Choi, S. S., Jazayeri, D. G., Mitnick, C. D., Chalco, K., Bayona,J., and Fraser, H. S., Implementation and initial evaluation of aWeb-based nurse order entry system for multidrug-resistanttuberculosis patients in Peru. Medinfo. 11:202–206, 2004.

58. Odhiambo-Otieno, G. W., Evaluation of existing district healthmanagement information systems a case study of the districthealth systems in Kenya. Int. J. Med Inform. 74:733–744, 2005.

59. Scott, J. T., Rundall, T. G., Vogt, T. M., and Hsu, J., KaiserPermanente’s experience of implementing an electronic medicalrecord: A qualitative study. BMJ. 331:1313–1316, 2005.

60. Aarts, J., and Berg, M., A tale of two hospitals: a sociotechnicalappraisal of the introduction of computerized physician orderentry in two Dutch hospitals. Medinfo. 11:999–1002, 2004.

61. Koppel, R., Metlay, J. P., Cohen, A., Abaluck, B., Localio, A. R.,Kimmel, S. E., and Strom, B. L., Role of computerized physicianorder entry systems in facilitating medication errors. JAMA.293:1197–1203, 2005.

62. Vimarlund, V., and Olve, N. G., Economic analyses for ICT inelderly healthcare: Questions and challenges. Health InformaticsJ. 11:309–321, 2005.

J Med Syst (2007) 31:397–432 431

63. Olve, N. G., Vimarlund, V., and Agerbo, M. Evaluation as multi-actor trade-off—A challenge in introducing ICT innovations in thehealth sector. Invited paper, 4th WSEAS International Conferenceon E-Activities. Miami, Florida, USA, November 17–19, 2005.

64. Adams, W. G., Mann, A. M., and Bauchner, H., Use of anelectronic medical record improves the quality of urban pediatricprimary care. Pediatrics. 111:626–632, 2003.

65. Pizziferri, L., Kittler, A. F., Volk, L. A., Honour, M. M., Gupta, S.,Wang, S., Wang, T., Lippincott, M., Li, Q., and Bates, D. W.,Primary care physician time utilization before and after imple-mentation of an electronic health record: A time-motion study.J. Biomed. Inform. 38:176–188, 2005.

66. Bryson, M., Tidy, N., Smith, M., and Levy, S., An online surveyof nurses’ perceptions, knowledge and expectations of theNational Health Service modernization programme. J. Telemed.Telecare. 11:S1:64–66, 2005.

67. Hebert, M. A., and Korabek, B., Stakeholder readiness fortelehomecare: implications for implementation. Telemed J EHealth. 10:85–92, 2004.

68. Kaufman, D. R., Starren, J., Patel, V. L., Morin, P. C., Hilliman,C., Pevzner, J., Weinstock, R. S., Goland, R., and Shea, S. Acognitive framework for understanding barriers to the productiveuse of a diabetes home telemedicine system. AMIA Annu. Symp.Proc. 356–360, 2003.

69. Ash, J. S., Sittig, D. F., Seshadri, V., Dykstra, R. H.,Carpenter, J. D., and Stavri, P. Z., Adding insight: aqualitative cross-site study of physician order entry. Medinfo.11:1013–1017, 2004.

70. Ash, J. S., Gorman, P. N., Seshadri, V., and Hersh, W. R.,Computerized physician order entry in U.S. hospitals: results of a2002 survey. J. Am. Med. Inform. Assoc. 11:95–99, 2004.

71. Ammenwerth, E., and de Keizer, N., An inventory ofevaluation studies of information technology in healthcaretrends in evaluation research 1982–2002. Methods Inf. Med.44:44–56, 2005.

72. Kaplan, B., Evaluating informatics applications—Clinical deci-sion support systems literature review. Int. J. Med. Inform. 64:39–56, 2001.

432 J Med Syst (2007) 31:397–432