View
11
Download
0
Category
Preview:
Citation preview
1
EHRUsabilityTestReportofExscribeEHR(Version6.0)ReportbasedonNISTIR7742CommonIndustryFormatforUsabilityTestReports
DateofUsabilityTest:February5thtoFebruary7th,2018DateofReport:February14,2018
ReportPreparedBy:
TheUsabilityPeople,LLC
4000LegatoRoad,Suite1100Fairfax,VA22033
www.TheUsabilityPeople.com
2
TABLEOFCONTENTSExecutiveSummary..................................................................................................................................................................3Introduction.................................................................................................................................................................................4Method............................................................................................................................................................................................5Participants.............................................................................................................................................................................5StudyDesign...........................................................................................................................................................................8Tasks..........................................................................................................................................................................................9TestLocation.......................................................................................................................................................................10TestEnvironment..............................................................................................................................................................11TestFormsandTools......................................................................................................................................................12ParticipantInstructions..................................................................................................................................................13Procedure.............................................................................................................................................................................14UsabilityMetrics................................................................................................................................................................15DataScoring...................................................................................................................................................................16
Results.........................................................................................................................................................................................18DataAnalysisandReporting........................................................................................................................................18Reliability.........................................................................................................................................................................18EffectivenessandEfficiency....................................................................................................................................19Satisfaction......................................................................................................................................................................20
SpecificTaskResultDetails(Includingcriteriaevaluated)............................................................................25DiscussionofFindings..........................................................................................................................................................50
Effectiveness..................................................................................................................................................................50Efficiency..........................................................................................................................................................................50Satisfaction......................................................................................................................................................................51SummaryofMajorFindings....................................................................................................................................52RiskAnalysis..................................................................................................................................................................53AreasforImprovement.............................................................................................................................................54
Appendices................................................................................................................................................................................55AppendixA:RecruitingScreener...............................................................................................................................55AppendixB:InformedConsentForm.......................................................................................................................57AppendixC:ParticipantGuide.....................................................................................................................................58AppendixD:SystemUsabilityScaleQuestionnaire............................................................................................73AppendixE:ComputerSystemUsabilityQuestionnaire..................................................................................74AppendixF.References..................................................................................................................................................76
3
ExecutiveSummary
OnFebruary5thtoFebruary7th,2017,TheUsabilityPeople,LLCconductedasummative
usabilitytestoftheExcribeEHR.ThetestwasconductedintheFairfax,VAofficeofTheUsability
Peopleoverremotetele-conferencingsessionsusingGotoMeeting.Thepurposewastotestand
validatetheusabilityofthecurrentuserinterfaceandprovideevidenceofusabilityoftheExcribe
EHRastheEHRUnderTest(EHRUT).Ten(10)healthcareprovidersmatchingthetarget
demographiccriteriaparticipatedintheusabilitytestusingtheEHRUTinsimulated,but
representativetasks.
Thestudyfocusedonmeasuringtheeffectivenessof,efficiencyof,andsatisfaction(ISO
9241-11)withExscribeamongasampleofparticipantsrepresentingpotentialusersofthesystem.
Performancedatawascollectedontwenty-five(25)taskstypicallyconductedonanEHR.Tasks
createdwerebaseduponthecriteriaspecifiedwithinthetestprocedurestructureforevaluating
conformanceofElectronicHealthRecord(EHR)technologytothecertificationcriteriadefinedin
certificationcriteriaidentifiedin45CFRPart170SubpartCoftheHealthInformationTechnology:
2015EditionHealthInformationTechnology(HealthIT)CertificationCriteria.
ResultsofthestudyindicatedthattheExscribeEHRsystemwasquitesatisfactorywith
regardstoeffectivenessandefficiencyandthattheparticipantsweresatisfiedwiththesystem.
4
Introduction
TheElectronicHealthRecordSystemUnderTest(EHRUT)testedforthisstudy,theExcribe
EHR(V6.0),wasspecificallydesignedtopresentmedicalinformationtohealthcareproviderson
desktopcomputersinstandardambulatoryhealthcaresettings.Thisstudytestedandvalidatedthe
usabilityoftheExscribeEHRsoftwareuserinterfaceandprovidesevidenceoftheusabilityofthe
ExscribeEHRwithrepresentativeexercisesandinrealisticuserconditions.Tothisend,measures
ofeffectivenessandefficiency,suchastimeontask,numberoferrorsmade,andcompletionrates
werecapturedduringusabilitytesting.Satisfactionwasassessed,andusercommentscollected
usingtwoindustry-standardquestionnaires:TheSystemUsabilityScale(SUS)andtheComputer
SystemUsabilityQuestionnaire(CSUQ).
5
MethodParticipants
Ten(10)individuals(7womenand3men)participatedintheEHRUT(s)usingtheExscribe
EHR.Participantswerephysicians,nurses,medicalassistants,and/orotherhealthcare/HealthIT
practitioners.Participantswererecruitedfromadatabaseofpotentialparticipantsmaintainedby
TheUsabilityPeople,LLC.Thecontactscontainedwithinthisdatabaseweregeneratedfrompast
participants,andviapotentialparticipantsresponsestopostingsinInternetandsocialmediasites,
andalinkatthebottomofTheUsabilityPeoplewebsite.Thosewhorespondedtotheinvitationto
takepartinthestudyweredirectedtoanonlinequestionnairethatservedastheparticipant
screener.(ThescreeningquestionnaireisprovidedasAppendix1.)Participantsmeetingthe
criteriaforparticipationinthestudywerecontactedandscheduledviatelephoneandemail.
ParticipantsintheusabilitytestoftheExscribeEHRhadavarietyofhealthcare
backgroundsanddemographiccharacteristics.
Table1presentsparticipantcharacteristics,includingdemographics,professional
experience,computingexperience,andpreviousEHRexperience.Participantcharacteristicsreflect
theaudienceofcurrentandfutureusersandmeetthecriteriadesignatedinthe2015Edition
CertificationCompanionGuideforSafety-enhanceddesign-45CFR170.315(g)(3).Noneofthe
participantswerefromthevendororganization(Exscribe)thatproducedandsuppliedthe
evaluatedsystemnordidanyparticipanthaveanydirectconnectiontothetestingorganization
(TheUsabilityPeople,LLC).Allparticipantswerecompensatedfortheirtime.
6
Table1.ParticipantCharacteristics
PartID Gender Age Education Role/Title
ProfessionalExperience(Months)
ComputerExperience(Months)
Experiencewith
Exscribe(Months)
AssistiveTechNeeds
P01 Male 50to59 Doctor MD 312 96 0 None
P02 Female 20to29
Tradetechnicalvocationaltraining MedicalAssistant 60 60 0 None
P03 Male 40to49Bachelor’sdegree RegisteredNurse 180 84 0 None
P04 Female 20to29
Tradetechnicalvocationaltraining LPN 36 36 0 None
P05 Female 50to59 Master’sdegreeInformatics
NurseSpecialist 72 96 0 None
P06 Female 40to49Bachelor’sdegree ClinicalAnalyst 252 192 0 None
P07 Male 40to49 AssociatedegreeChiropracticAssistant 144 36 0 None
P08 Female 40to49Bachelor’sdegree RegisteredNurse 36 36 0 None
P09 Female 50to59 Master’sdegree RegisteredNurse 72 60 0 None
P10 Female 30to39Bachelor’sdegree RegisteredNurse 144 180 0 None
7
SummaryofParticipantCharacteristics:Participantshadexperiencewiththeoccupationandexpertisethatalignswiththecapabilityundertesting.Thecohortofuserswhoareselectedasparticipantswasvariedwiththeproductanditsintendedusersandwasnotlimitedtoclinicians.Thedemographiccharacteristicsofthetestparticipantcharacteristicsreflectedtheaudienceofcurrentandfutureusers.Gender Male 3Female 7 AgeRange 20to29 230to39 140to49 450to59 360to69 070to79 0 Education Somecollegecredit,nodegree 0Tradetechnicalvocationaltraining 2Associatedegree 1Bachelor’sdegree 4Master’sdegree 2DoctorateDegree 1 YearsofExperiencewithExscribeNone 10Upto3years 03to5years 05to10years 0Morethan10years 0
8
StudyDesign
TheoverallobjectiveofthisusabilitytestwastouncoverareaswheretheExscribeEHR
systemperformedwell–thatis,effectively,efficiently,andwithsatisfaction–andareaswherethe
systemfailedtoservetheclinicaldocumentationandworkflowneedsofusers.Datafromthistest
maybeusedasabaselineforfuturetestsofupdatedversionsofExscribeand/orforcomparing
ExscribesoftwarewithotherEHRspresentingthesametasks.Inshort,thistestingservesasbotha
meanstorecordorbenchmarkcurrentusabilityandtoidentifyareaswhereimprovementsmustbe
made.
ParticipantshadarangeofexperiencewithEHRsingeneral,butnonehadanyprior
experiencewiththeExscribesystem.ParticipantscompletedtheExscribeusabilitystudyduring
individual60-90-minuteGotoMeetingsessions.Duringthetest,eachparticipantinteractedwith
variouscomponentsoftheExscribesoftwaresystem.Eachparticipantwasprovidedwiththesame
instructions.
TheExscribeEHRwasevaluatedforeffectiveness,efficiencyandsatisfactionasdefinedby
thefollowingmeasurescollectedandanalyzedforeachparticipant:
• Numberoftaskssuccessfullycompletedwithoutassistance
• Timetocompletethetasks
• Numberanddescriptionoferrors
• Pathdeviations
• Participant’sverbalizations(comments)
• Participant’ssatisfactionratingsofthesystem
9
TasksAtotaloftwenty-five(25)taskswereconstructedbyTheUsabilityPeople,LLC.(inclose
collaborationwithExscribestaff)toberealisticandrepresentativeoftheactivitiesausermight
engageinwhileusingExscribeinactualmedicalsettings.Thetwenty-five(25)taskswerecreated
baseduponthecriteriaspecifiedwithinthetestprocedurestructureforevaluatingconformanceof
ElectronicHealthRecord(EHR)technologytothecertificationcriteriaasdefinedin45CFRPart
170SubpartCoftheHealthInformationTechnology:Standards,ImplementationSpecifications,
andCertificationCriteriaforElectronicHealthRecordTechnology.Thetasksfocusedonthetwelve
2015EditioncertificationcriteriaspecifiedbyONC,specifically:
• Section170.315(a)(1)Computerizedproviderorderentry–medications
• Section170.315(a)(2)Computerizedproviderorderentry–laboratory
• Section170.315(a)(3)Computerizedproviderorderentry–diagnosticimaging
• Section170.315(a)(4)Drug-drug,drug-allergyinteractionchecks
• Section170.315(a)(5)Demographics
• Section170.315(a)(6)Problemlist
• Section170.315(a)(7)Medicationlist
• Section170.315(a)(8)Medicationallergylist
• Section170.315(a)(9)Clinicaldecisionsupport
• Section170.315(a)(14)Implantabledevicelist
• Section170.315(b)(2)Clinicalinformationreconciliationandincorporation
• Section170.315(b)(3)Electronicprescribing
AcopyofthetaskspresentedtoparticipantsintheusabilitytestoftheExscribeEHRcanbefound
inAppendixC.
10
TestLocation
AllparticipantsweretestedontheExscribeEHRsystemduringremoteconferencing
sessionsusingGotoMeeting.Eachparticipantwasrequestedinadvancetosecureaquietroom
withminimaldistractionsandadesktoporlaptopcomputerthatcouldconnecttotheInternetwith
aGotoMeetingsession.Althoughthetypeofcomputer,operatingsystemanddisplayresolutionof
theremoteparticipantsystemwasunknown,thesystemthatwasusedbythetestadministrator
andcontrolledbytheremoteparticipantwasaDellInspironLaptoprunningtheWindows10
professionaloperatingsystemataresolutionof1366x768pixels.DuringagivenGotoMeeting
session,onlythetestadministratorandparticipantcommunicatedwithoneanother.
TheGotoMeetingusabilitytestsessionwasconductedbyatestadministratorfromthe
testingorganization(TheUsabilityPeople,LLC)workingfromasmallconferenceroomatThe
UsabilityPeople’sFairfax,VAlocation.Adataloggerfromthetestingorganizationalsotook
detailednotesoneachsession,includingusercommentsandotherratingsfollowingeachtask.
Duringasessionboththetestadministratorandthedataloggercouldseeonlytheparticipant’s
screenandheartheparticipant’scomments,questions,andresponses.
11
TestEnvironment
WhiletheEHRUTtypicallywouldbeusedinahealthcareoffice,orambulatorycenter
facility,testingoftheExscribeEHRsystemwasconductedviaremoteconnectionduringindividual
GotoMeetingsessions.EachparticipantcalledintoaGotoMeetingsessionandwasconnectedby
thetestadministratortotheapplication.
TheExscribesystemitselfonaWindows™clientplatformonaLANconnectionusinga
sampledatabasethatwassetupspecificallyforthetest.Participantsusedamouseandkeyboard
wheninteractingwiththeEHRUTandweregivenremotecontroloftheadministrator’s
workstationtoperformthetasks.
12
TestFormsandTools
Aspartoftheusabilitytest,severaldocumentsandinstrumentswereused.Examplesofthe
documentsusedduringtheusabilitytest,includinganinformedconsentform,thetasks,andpost-
testquestionnaires,canbefoundinAppendicesBtoE,respectively.
Participants’interactionwiththeExscribeEHRsoftwarewascapturedandrecordeddigitally
usingtheMoraescreencapturesoftwarerunningonthetestadministrator’sworkstation.Verbal
responseswererecordedthrougheitherthemicrophoneintegratedintotheparticipant’scomputer
orthroughatelephoneconnection.Thisinformationwaselectronicallytransmittedtothe
administratorandtothedataloggerduringeachtestsession.
13
ParticipantInstructions
Theadministratorreadthefollowinginstructionsaloudtoeachparticipant:
Thankyouforparticipatinginthisstudy.Yourinputisveryimportant.Oursessiontodaywilllastabout60-90minutes.Duringthattime,youwilluseaninstanceofanelectronichealthrecord.Iwillaskyoutocompleteafewtasksusingthissystemandanswersomequestions.
Pleasenotethatwearenottestingyou;wearetestingthesystem.Therefore,ifyouhaveanydifficultythismaymeanthatsomethingneedstobeimprovedinthesystem.Iwillbehereincaseyouneedspecifichelp,butIamnotabletoinstructyouorprovidehelpinhowtousetheapplication.
Overall,weareinterestedinhoweasy(orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Ididnothaveanyinvolvementinitscreation,sopleasebehonestwithyouropinions.Alloftheinformationthatyouprovidewillbekeptconfidentialandyournamewillnotbeassociatedwithyourcommentsatanytime.Shouldyoufeelitnecessaryyouareabletowithdrawatanytimeduringthetesting.Participantswerethengiventaskstocomplete.
14
Procedure
Uponconnectiontotheonlinemeetingtool(GotoMeeting),eachparticipantwasgreeted,
hisorheridentityverified,andmatchedtoanameontheparticipantschedule.Participant
demographicinformationwasalsoverified,includinginformationregardingoverallcomputer
experienceandparticipantexperiencewiththeEHUT.Participantnameswerereplacedwith
participantIDssothatagivenindividual’sdatacannotbelinkedtohis/heridentity.Priorto
beginningtesting,eachparticipantreviewedandsignedaninformedconsentform(SeeAppendix
B).
StaffmembersoftheUsabilityPeople,ausabilitytestadministratoradministeredthetest.
Theadministratormoderatedthesessionbyprovidingbothverbalandwritteninstructionsforthe
overallusabilitytestandforeachofthetaskscomprisingthetest.Theadministratoralso
monitoredtasksuccess,pathdeviations,numberanddescriptionoferrors,andaudio-recorded
participantverbalcomments.Adataloggerloggedtasktimes,obtainedpost-taskratingdata,and
tooknotesonparticipantcommentsandadministratorfeedback.
Foreachofthetwenty-five(25)tasks,participantswereprovidedwritteninstructionsto
theircomputers.Followingtheadministrator’sinstructions,eachparticipantperformedeachtask
byfirstreadingthetaskthenstatinginhisorherownwordshisorherinterpretationofthetask
requirements.Whentheparticipant’sinterpretationmatchedtheactualgoalofthetask,the
administratorinstructedtheparticipanttobeginandtasktimingbegan.Tasktimewasstopped
andrecordedwhenthetestadministratorobservedontheirworkstationthattheparticipanthad
successfullycompletedthetask.Ifaparticipantfailedtocompleteataskbeforetheexpected
amountoftimeforeachtask,thattaskwasmarkedas“TimedOut.”Aftereachtask,thetest
administratoraskedtheparticipant,“Onascalefrom1to5,where1is‘VeryDifficultand5is‘Very
Easy,’howsatisfiedwereyouwiththeeaseofuseforthistask?”Thissameprocedurewas
conductedforeachofthetwenty-five(25)tasks.
15
Followingcompletionofthetwenty-five(25)EHRtasks,theadministratorelectronically
presentedtotheparticipanttwopost-testquestionnaires(SystemUsabilityScale(SUS),see
AppendixDandComputerSystemUsabilityQuestionnaire(CSUQ),seeAppendixE).Afterthe
participantcompletedbothquestionnaires,theadministratorthankedeachparticipantforhisor
hertimeandallowedtheparticipanttomakeanycommentsonoraskanyquestionsaboutthe
systemand/orthetaskspresented.
Foreachsession,theparticipant’sschedule,demographicinformation,tasksuccessrate,
timeontask,errors,deviations,verbalresponses,andpost-testquestionnaireweredigitally
recorded.Thesystemwasthenresettopropertestconditionsforthenextparticipant.
UsabilityMetrics
AccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronic
HealthRecords(NISTIR7741,November2010)EHRsshouldsupportaprocessthatprovidesahigh
levelofusabilityforallusers.Thegoalisforuserstointeractwiththesystemeffectively,efficiently,
andwithanacceptablelevelofsatisfaction.Tothisend,metricsforeffectiveness,efficiencyand
usersatisfactionwerecapturedduringtheusabilitytesting.Thegoalsofthetestweretoassess:
• EffectivenessoftheExscribeEHRsoftwarebymeasuringparticipantsuccessratesand
errors.
• EfficiencyoftheExscribeEHRsoftwareSoftwarebymeasuringtheaveragetasktimeand
pathdeviations.
• SatisfactionwiththeExscribeEHRsoftwarebymeasuringease-of-useratings.
16
DataScoringTable2detailshowtaskswerescored,errorsevaluated,andthetimedataanalyzed.Table2.ScoringProtocolsforEffectiveness,Efficiency,andSatisfaction
Measures RationaleandScoring
Effectiveness:• TaskSuccess
A task was counted as “Success” if the participant was able to achieve thecorrect outcome,without assistance,within the time allotted on a per taskbasis.ThetotalnumberofSuccesseswascalculatedforeachtaskandthendividedbythetotalnumberoftimesthattaskwasattempted.Resultsareprovidedasapercentage.
Effectiveness:• TaskFailures
If the participant abandoned the task, did not reach the correct answer orperformed it incorrectly, or reached the end of the allotted time beforesuccessfulcompletion,thetaskwascountedas“Fail.”Notasktimesweretakenforfailedattempts.Thetotalnumberoferrorswascalculatedforeachtaskanddividedbythetotalnumberoftimesthattaskwasattempted.Resultsarepresentedastheaverageerrorrate.Note:Notalldeviationsarecountedaserrors
Effectiveness:• Prompted
Successes
Becausesometasksaredependentuponthesuccessfulcompletionofprevioustasks,participantsmayreceivealimitednumberof“prompts”tohelppreparethesystemdataforthepre-requisitesforsubsequenttasks.Whenaparticipantwasable tocomplete thedataentryona taskwith3orfewer prompts, the taskwas counted as an “Assisted” completion. No tasktimeswererecordedforAssistedcompletions.
Efficiency:• Task
Deviations
The participant’s path (i.e., steps) through the application was recorded.Deviations occur if for example, the participant navigated to an incorrectscreen, clicked on an incorrect menu item, followed an incorrect link, orinteractedincorrectlywithanon-screencontrol.
Efficiency:• TaskTime
Each task was timed from the administrator’s prompt “Begin” until said,“Done.” If the participant failed to say, “Done,” timing stopped when theparticipantstoppedperformingthetask.Onlytasktimesfortasksthatweresuccessfullycompletedwereincludedintheaveragetasktimeanalysis.Averagetimepertaskwascalculatedforeachtask.
17
Satisfaction:• EaseofUse
ratings• System
Satisfaction
Participant’ssubjective impressionof theeaseofuseof theapplicationwasmeasured by administering both a single post-task question aswell as twopost-sessionquestionnaires.Aftereachtask,theparticipantdeterminedonascaleof1to5theirsubjectivesatisfaction with performance on the task. These data are averaged acrossparticipants.Tomeasureparticipants’confidenceinandlikeabilityoftheEHRoverall,thetesting team administered electronic versions of the SystemUsability Scale(SUS)andtheComputerSystemUsabilityQuestionnaire(CSUQ).SeetheSUSquestionnaireasAppendixD.,andtheCSUQasAppendixE.
18
Results
DataAnalysisandReporting
TheresultsoftheusabilitytestoftheExscribeEHRsoftwaresystemwereanalyzed
accordingtothemethodsdescribedintheUsabilityMetricssectionaboveandaredetailedbelow.
Notethattheresultsshouldbeevaluatedrelativetothestudyobjectivesandgoals,asoutlinedin
thestudydesignsectionabove.Thedatashouldyieldactionableresultsthat,ifcorrected,yield
material,positiveimpactonuserperformance.
Reliability
Duringthedatacollectionphase,itwasobservedthatthesystemprovidedausually
consistentandreliableinterfacetoeachparticipantastheycompletedtheirtasks.Aseach
participantcompletedtheirassignedtasks,thesystemprovidedthesameinformationand
respondedtotheirinputwiththesameverbiageandusingthesamemodeofcommunication(e.g.
Pop-upmessage,orembeddedassistance).
19
EffectivenessandEfficiency
Table3presentsasummaryofoveralltaskperformanceshowingtask,meantimeontask,
taskcompletionrates,meanpathdeviationsandmeantasksatisfaction:
Table3.UsabilityTestResults
TaskMeanTaskTime
SD
CompletionRate(%)
Mean#Path
Deviations
SD
MeanTaskSatisfaction
SD
Task1-Demographics 2:26 1:29 80% 2.30 2.10 3.70 0.64Task2-Demographics-Change 1:30 0:45 100% 0.70 1.00 4.10 0.54Task3-ClinicalInfoRecon.Incorp. 3:49 1:33 90% 2.00 1.41 4.20 0.40Task4-AddanAllergy 2:06 1:08 100% 1.90 1.14 3.90 0.54Task5-ViewAllergyList-ChangeAllergy
0:27 0:09 100% 0.00 0.00 4.80 0.40
Task6-ImplantableDeviceList 0:56 0:19 100% 0.10 0.30 4.70 0.46Task7-ImplantableDevice-Change 0:27 0:22 100% 0.40 0.66 4.80 0.40Task8-ProblemList–Add 0:55 0:10 100% 0.80 0.60 4.50 0.50Task9-ProblemList-Change 0:23 0:26 100% 0.10 0.30 4.60 0.66Task10-ProblemList-Views 0:28 0:12 100% 0.10 0.30 4.20 0.60Task11-MedicationList-AddMedication
0:50 0:12 100% 1.20 0.40 4.10 0.54
Task12-MedicationList-ChangeMedication
0:42 0:11 100% 0.40 0.66 4.50 0.50
Task13-DiagnosticImaging-AddOrder
1:07 0:15 100% 0.60 0.66 4.40 0.66
Task14-ChangeDiagnosticImagingOrder
0:17 0:05 100% 0.10 0.30 4.80 0.40
Task15-LabOrder-Add 1:11 0:39 90% 1.10 1.70 3.90 0.83Task16-LabOrder–Change 0:42 0:08 100% 0.10 0.30 4.20 0.75Task17-Drug-DrugInteractionCheck
2:17 1:00 80% 2.40 2.15 2.60 0.66
Task18-DrugAllergyInteractionCheck
1:20 0:52 90% 1.90 1.87 2.90 0.94
Task19-ElectronicRx-NewRx 1:37 0:30 90% 1.50 1.75 3.60 0.66Task20-ElectronicRx-ChangeRx 1:41 0:56 80% 1.70 2.24 3.60 0.66Task21-ElectronicRx-RefillRx 0:41 0:16 100% 1.10 0.94 4.20 0.60Task22-ElectronicRx-Views 0:37 0:27 100% 0.30 0.90 4.00 0.77Task23-ElectronicRx-CancelRx 0:31 0:16 100% 0.10 0.30 4.20 0.87Task24-CDS-EnableRule-ViewAttributes
1:18 0:25 100% 0.80 0.98 4.20 0.60
Task25-CCDAViewReconciledData
1:57 0:38 100% 0.70 0.64 4.70 0.46
20
AsTable3shows,relativetooptimalperformancestandardsasdefinedbyExscribeand
TheUsabilityPeople,participantperformanceintheExscribeEHRsoftwareusabilitytestwasquite
satisfactory.Theoverallaveragetaskcompletionratewasninety-six(96)percent.
Satisfaction
IndividualTaskSatisfaction
Participantsverballyindicatedtheirsatisfactionwiththeeaseofuseforeachtaskusinga
scaleof“1”(“VeryDifficult”)to“5,”(“VeryEasy”).AsFigure1showsindividualtasksatisfaction
rangedfromalowof2.6outof5onTask17(Drug-Druginteractioncheck)toahighof4.8outof5
onTasks5,7,and14(Viewallergylist-ChangeAllergy,ImplantableDevice–Change,andChange
DiagnosticImagingOrder).
21
Figure1.SatisfactionRatingsofIndividualTasks
0.00 1.00 2.00 3.00 4.00 5.00
Task1-Demographics
Task2-Demographics-Change
Task3-ClinicalInfoRecon.Incorp.
Task4-AddanAllergy
Task5-ViewAllergyList-ChangeAllergy
Task6-ImplantableDeviceList
Task7-ImplantableDevice-Change
Task8-ProblemList- Add
Task9-ProblemList- Change
Task10-ProblemList- Views
Task11-MedicationList-AddMedication
Task12-MedicationList-ChangeMedication
Task13-DiagnosticImaging-AddOrder
Task14-ChangeDiagnosticImagingOrder
Task15-LabOrder-Add
Task16-LabOrder- Change
Task17-Drug-DrugInteractionCheck
Task18-DrugAllergyInteractionCheck
Task19-ElectronicRx- NewRx
Task20-ElectronicRx-ChangeRx
Task21-ElectronicRx-RefillRx
Task22-ElectronicRx-Views
Task23-ElectronicRx- CancelRx
Task24-CDS-EnableRule-ViewAttributes
Task25-CCDAViewReconciledData
AverageTaskSatisfaction
22
IndividualParticipantSatisfaction
Ingeneral,theparticipantsweresatisfiedwiththeeaseofuseoftheExscribeEHRsoftwaresystem.Thefollowingchartdisplaysoverallsatisfactionforeachparticipant:
Theaverageoveralltasksatisfactionratewas4.14outof5.
3.64
4.20
4.16
4.80
4.08
4.24
4.12
4.04
3.80
4.28
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
P01
P02
P03
P04
P05
P06
P07
P08
P09
P10
OverallSatisfaction(average)
23
SystemUsabilityScale
TheSystemUsabilityScale(SUS)isasimple,10-itemLikert-typeattitudescaleprovidinga
globalsubjectiveassessmentofusabilityfromtheuser’sperspective(JohnBrookeatDigital
EquipmentCompanydevelopedtheSUSin1986).TheSUSscaleisscoredfrom0to100;scores
under60representsystemswithlessthanoptimalusability,scoresover80areconsideredbetter
thanaverage.SeeAppendixDforacopyoftheSUS.
ThemeantotalSUSscoretheExscribeEHRwassixty-one(61)andrangedfromalowof
forty-five(45)andahighofseventy-five(75).Overall,participant-usersratedtheirsatisfaction
withtheExscribesoftwaresystemtobewithintherangeofausableandsatisfyingEHR.
ThefollowingchartshowstheSUSscorebyeachparticipant:
0 10 20 30 40 50 60 70 80
Participant1
Participant2
Participant3
Participant4
Participant5
Participant6
Participant7
Participant8
Participant9
Participant10
55
72.5
75
62.5
45
65
50
70
60
57.5
SUSScorebyParticipant
24
ComputerSystemUsabilityQuestionnaire
UsingtheComputerSystemUsabilityQuestionnaire(CSUQ;Lewis,J.R.(1995).(See:IBM
ComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationandInstructionsforUse.
InternationalJournalofHuman-ComputerInteraction,7:1,57-78).),participantsratedeachof19
itemsoftheCSUQquestionnaireonascalefrom1to7,witharatingof7beingmostinagreement
withthepositively-wordeditem.Responsesforeachitemweresummedandaveragedtofour
scales–InterfaceQuality,InformationQuality,SystemUsefulness-andanoverallscale.See
AppendixEforacopyoftheCSUQ.
Figure2displaysCUSQratingsforeachofthefourscales.Ingeneral,participantsinthe
Exscribestudyratedsystemusabilitytobehigh.OnInterfaceQualitytheaveragescoreforthe
participantswas5.47/7;onInformationQualitytheaveragescore4.86/7;onSystemUsefulness
theaveragescorewas4.90/7;andtheoverallaverageCUSQscorewas4.96/7.
3.00 3.50 4.00 4.50 5.00 5.50 6.00 6.50 7.00
Overall Score
System Usefulness
Information Quality
Interface Quality
4.96
4.90
4.86
5.47
User Rating (Out of 7)
Computer System Usability Questionnaire
25
SpecificTaskResultDetails
ParticipantNumber
Task1-Demographics
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 4:06 Fail 6 2P02 1:09 Success 1 4P03 5:37 Success 2 4P04 3:28 Success 1 4P05 1:10 Success 0 4P06 1:18 Success 1 4P07 1:31 Success 1 4P08 3:31 Success 4 4P09 3:51 Fail 6 3P10 1:43 Success 1 4
ExpectedTimeonTask 2:00 (SD) AverageTimeonTask 2:26 1:29 AverageTaskSatisfaction 3.70 0.64 Average#PathDeviations 2.30 2.10 PercentSuccess 80%
26
ParticipantNumber
Task2-Demographics-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 3:32Success 3 3P02 0:51Success 0 4P03 1:54Success 0 4P04 1:05Success 0 5P05 1:24Success 1 4P06 1:01Success 1 4P07 1:18Success 0 4P08 1:22Success 0 4P09 1:43Success 2 4P10 0:53Success 0 5
ExpectedTimeonTask 0:30 (SD)AverageTimeonTask 1:30 0:45AverageTaskSatisfaction 4.10 0.54Average#PathDeviations 0.70 1.00PercentSuccess 100%
27
ParticipantNumber
Task3-ClinicalInfoRecon.Incorp.
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 6:25Success 3 5P02 3:34Success 1 4P03 7:35TimeOut 4 4P04 2:41Success 0 5P05 2:51Success 0 4P06 2:53Success 2 4P07 3:02Success 2 4P08 4:08Success 3 4P09 4:33Success 4 4P10 4:11Success 1 4
ExpectedTimeonTask 4:00 (SD)AverageTimeonTask 3:49 1:33AverageTaskSatisfaction 4.20 0.40Average#PathDeviations 2.00 1.41PercentSuccess 90%
28
ParticipantNumber
Task4-AddanAllergy
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 4:38 Success 4 3P02 1:10 Success 2 3P03 2:56 Success 1 4P04 2:13 Success 2 4P05 1:11 Success 1 5P06 1:22 Success 2 4P07 1:43 Success 1 4P08 1:32 Success 1 4P09 3:19 Success 4 4P10 0:54 Success 1 4
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 2:06 1:08 AverageTaskSatisfaction 3.90 0.54 Average#PathDeviations 1.90 1.14 PercentSuccess 100%
29
ParticipantNumber
Task5-ViewAllergyList-ChangeAllergy
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:27 Success 0 4P02 0:18 Success 0 5P03 0:21 Success 0 5P04 0:51 Success 0 5P05 0:20 Success 0 5P06 0:22 Success 0 5P07 0:23 Success 0 5P08 0:33 Success 0 5P09 0:31 Success 0 4P10 0:20 Success 0 5
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:27 0:09 AverageTaskSatisfaction 4.80 0.40 Average#PathDeviations 0.00 0.00 PercentSuccess 100%
30
ParticipantNumber
Task6-ImplantableDeviceList
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:39 Success 0 4P02 0:40 Success 0 4P03 1:45 Success 0 4P04 0:51 Success 0 5P05 1:13 Success 1 5P06 0:51 Success 0 5P07 0:49 Success 0 5P08 0:55 Success 0 5P09 0:48 Success 0 5P10 0:51 Success 0 5
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 0:56 0:19 AverageTaskSatisfaction 4.70 0.46 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
31
ParticipantNumber
Task7-ImplantableDevice-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:28 Success 2 4P02 0:16 Success 0 5P03 0:34 Success 1 5P04 0:12 Success 0 5P05 0:13 Success 0 5P06 0:16 Success 0 5P07 0:15 Success 0 5P08 0:18 Success 0 5P09 0:41 Success 1 4P10 0:18 Success 0 5
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:27 0:22 AverageTaskSatisfaction 4.80 0.40 Average#PathDeviations 0.40 0.66 PercentSuccess 100%
32
ParticipantNumber
Task8-ProblemList-Add
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:47 Success 0 5P02 0:49 Success 0 5P03 1:19 Success 0 5P04 0:47 Success 1 5P05 1:02 Success 1 4P06 0:51 Success 1 5P07 0:48 Success 1 4P08 0:44 Success 1 4P09 0:58 Success 2 4P10 1:03 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 0:55 0:10 AverageTaskSatisfaction 4.50 0.50 Average#PathDeviations 0.80 0.60 PercentSuccess 100%
33
ParticipantNumber
Task9-ProblemList-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:33 Success 0 4P02 0:14 Success 0 5P03 1:38 Success 1 3P04 0:13 Success 0 5P05 0:08 Success 0 5P06 0:09 Success 0 5P07 0:10 Success 0 5P08 0:11 Success 0 5P09 0:18 Success 0 4P10 0:11 Success 0 5
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:23 0:26 AverageTaskSatisfaction 4.6 0.66 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
34
ParticipantNumber
Task10-ProblemList-Views
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:01 Success 1 5P02 0:19 Success 0 4P03 0:18 Success 0 5P04 0:26 Success 0 5P05 0:35 Success 0 4P06 0:26 Success 0 4P07 0:22 Success 0 4P08 0:20 Success 0 4P09 0:31 Success 0 3P10 0:25 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:28 0:12 AverageTaskSatisfaction 4.2 0.60 Average#PathDeviations 0.1 0.30 PercentSuccess 100%
35
ParticipantNumber
Task11-MedicationList-AddMedication
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:42 Success 1 4P02 0:55 Success 1 4P03 1:13 Success 1 5P04 1:05 Success 1 5P05 0:38 Success 1 3P06 0:44 Success 1 4P07 0:33 Success 2 4P08 0:45 Success 1 4P09 0:55 Success 2 4P10 0:53 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 0:50 0:12 AverageTaskSatisfaction 4.10 0.54 Average#PathDeviations 1.20 0.40 PercentSuccess 100%
36
ParticipantNumber
Task12-MedicationList-ChangeMedication
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:11 Success 2 4P02 0:47 Success 0 5P03 0:33 Success 0 5P04 0:44 Success 1 5P05 0:40 Success 0 4P06 0:31 Success 0 4P07 0:42 Success 1 4P08 0:33 Success 0 4P09 0:35 Success 0 5P10 0:44 Success 0 5
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 0:42 0:11 AverageTaskSatisfaction 4.50 0.50 Average#PathDeviations 0.40 0.66 PercentSuccess 100%
37
ParticipantNumber
Task13-DiagnosticImaging-AddOrder
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:35 Success 1 3P02 1:14 Success 1 4P03 0:56 Success 0 4P04 0:53 Success 0 5P05 0:47 Success 0 5P06 0:58 Success 1 5P07 1:03 Success 0 4P08 1:18 Success 0 5P09 1:26 Success 1 5P10 1:01 Success 2 4
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 1:07 0:15 AverageTaskSatisfaction 4.40 0.66 Average#PathDeviations 0.60 0.66 PercentSuccess 100%
38
ParticipantNumber
Task14-ChangeDiagnosticImagingOrder
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:23 Success 0 4P02 0:20 Success 1 4P03 0:16 Success 0 5P04 0:18 Success 0 5P05 0:08 Success 0 5P06 0:10 Success 0 5P07 0:15 Success 0 5P08 0:22 Success 0 5P09 0:14 Success 0 5P10 0:19 Success 0 5
ExpectedTimeonTask 0:30 (SD) AverageTimeonTask 0:17 0:05 AverageTaskSatisfaction 4.80 0.40 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
39
ParticipantNumber
Task15-LabOrder-Add
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:35 Success 1 3P02 0:41 Success 0 4P03 1:07 Success 0 5P04 0:44 Success 0 5P05 1:25 Success 1 3P06 1:13 Success 1 3P07 1:24 Success 1 4P08 1:17 Success 1 4P09 3:11 Fail 6 3P10 1:15 Success 0 5
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 1:11 0:39 AverageTaskSatisfaction 3.90 0.83 Average#PathDeviations 1.10 1.70 PercentSuccess 90%
40
ParticipantNumber
Task16-LabOrder-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:38 Success 0 3P02 0:24 Success 0 4P03 0:51 Success 1 5P04 0:45 Success 0 5P05 0:46 Success 0 3P06 0:38 Success 0 4P07 0:41 Success 0 4P08 0:55 Success 0 4P09 0:44 Success 0 5P10 0:39 Success 0 5
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 0:42 0:08 AverageTaskSatisfaction 4.20 0.75 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
41
ParticipantNumber
Task17-Drug-DrugInteractionCheck
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:33 Success 1 3P02 1:53 Success 2 3P03 2:40 Success 2 2P04 2:01 Success 0 4P05 3:02 Success 1 2P06 2:52 Success 1 2P07 2:28 Success 2 3P08 4:23 Fail 7 2P09 4:33 Fail 6 2P10 1:44 Success 2 3
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 2:17 1:00 AverageTaskSatisfaction 2.60 0.66 Average#PathDeviations 2.40 2.15 PercentSuccess 80%
42
ParticipantNumber
Task18-DrugAllergyInteractionCheck
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:12 Success 2 2P02 0:32 Success 0 4P03 1:22 Success 2 2P04 1:04 Success 0 5P05 0:51 Success 0 3P06 0:46 Success 1 3P07 0:57 Success 1 3P08 2:46 Success 4 2P09 3:13 Fail 6 2P10 1:31 Success 3 3
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 1:20 0:52 AverageTaskSatisfaction 2.90 0.94 Average#PathDeviations 1.90 1.87 PercentSuccess 90%
43
ParticipantNumber
Task19-ElectronicRx-NewRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:12 Success 1 4P02 1:14 Success 0 4P03 2:32 Success 2 4P04 1:22 Success 0 4P05 1:01 Success 0 3P06 1:25 Success 1 4P07 1:19 Success 1 4P08 1:58 Success 3 3P09 2:19 Fail 6 2P10 1:34 Success 1 4
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 1:37 0:30 AverageTaskSatisfaction 3.60 0.66 Average#PathDeviations 1.50 1.75 PercentSuccess 90%
44
ParticipantNumber
Task20-ElectronicRx-ChangeRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:14 Success 1 3P02 0:58 Success 0 5P03 3:25 Fail 6 3P04 2:23 Success 1 4P05 2:04 Success 0 3P06 0:45 Success 0 4P07 0:54 Success 0 4P08 3:30 Fail 6 3P09 1:39 Success 2 3P10 2:34 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 1:41 0:56 AverageTaskSatisfaction 3.60 0.66 Average#PathDeviations 1.70 2.24 PercentSuccess 80%
45
ParticipantNumber
Task21-ElectronicRx-RefillRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:15 Success 1 3P02 0:35 Success 0 4P03 0:20 Success 0 5P04 1:04 Success 2 5P05 0:43 Success 1 5P06 0:27 Success 1 4P07 0:34 Success 1 4P08 0:37 Success 2 4P09 0:41 Success 3 4P10 0:38 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:41 0:16 AverageTaskSatisfaction 4.20 0.60 Average#PathDeviations 1.10 0.94 PercentSuccess 100%
46
ParticipantNumber
Task22-ElectronicRx-Views
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:30 Success 0 5P02 0:21 Success 0 4P03 1:46 Success 3 2P04 0:23 Success 0 5P05 0:19 Success 0 4P06 0:22 Success 0 4P07 0:12 Success 0 4P08 1:04 Success 0 4P09 0:33 Success 0 4P10 0:41 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:37 0:27 AverageTaskSatisfaction 4.0 0.77 Average#PathDeviations 0.30 0.90 PercentSuccess 100%
47
ParticipantNumber
Task23-ElectronicRx-CancelRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:02 Success 0 2P02 0:09 Success 0 5P03 0:12 Success 0 5P04 0:20 Success 0 5P05 0:32 Success 0 4P06 0:18 Success 0 5P07 0:22 Success 0 4P08 0:43 Success 1 4P09 0:44 Success 0 4P10 0:45 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:31 0:16 AverageTaskSatisfaction 4.2 0.87 Average#PathDeviations 0.1 0.30 PercentSuccess 100%
48
ParticipantNumber
Task24-CDS-EnableRule-ViewAttributes
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:05 Success 0 4P02 0:51 Success 1 4P03 1:34 Success 0 4P04 1:01 Success 0 5P05 0:48 Success 0 5P06 0:53 Success 0 5P07 1:23 Success 1 4P08 1:44 Success 2 4P09 2:03 Success 3 3P10 1:42 Success 1 4
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 1:18 0:25 AverageTaskSatisfaction 4.20 0.60 Average#PathDeviations 0.80 0.98 PercentSuccess 100%
49
ParticipantNumber
Task25-CCDAViewReconciledData
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:11 Success 0 5P02 2:35 Success 0 4P03 1:46 Success 0 5P04 1:23 Success 0 5P05 0:58 Success 1 5P06 0:55 Success 1 5P07 2:01 Success 1 4P08 2:47 Success 1 5P09 2:38 Success 2 5P10 2:15 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 1:57 0:38 AverageTaskSatisfaction 4.70 0.46 Average#PathDeviations 0.70 0.64 PercentSuccess 100%
50
DiscussionofFindings
Ingeneral,theparticipantsperformedverywellandweresatisfiedwithExscribeEHR
softwaresystem.Afewoftheparticipantsstruggledwithsomeportionsofafewtasksbutin
generalmostwereabletosuccessfullycompletealargemajorityofthetaskswithlittleorno
difficulty.Giventhatnoparticipantshadanypriorexperiencewiththesystem,thisshowsthatthe
ExcribeEHRisaneasy-to-useandaneasy-to-learnEHR.Participantsweremostlyabletoperform
alltaskssuccessfullyontheirownwithnoassistanceorexternaldocumentation.Theparticipant
averageperformanceratewashigh.TheExscribesystemappearstobeausableEHR.
Effectiveness
Ofthetwenty-five(25)taskspresented,alargemajorityofthetasksweresuccessfully
completedbyalloftheparticipants.Overallofparticipants,themeansuccessfultaskcompetition
ratewashighwithanoverallaveragerateofninety-six(96)percentindicatingthatingeneralthe
participantshadlittleornodifficultycompletingthetasks.
TheamountofpriorexperiencewithEHRsystemswasonlyslightlyrelatedtosuccessful
taskperformanceanderrorrates;participantswithmorepriorexperienceweremoreslightly
likelytosuccessfullycompletetaskswithlesserrorsthanthosewithlesspriorexperience.
Efficiency
Participantswhosuccessfullycompletedtasksgenerallycompletedthosetaskswithinan
acceptabletime.Sometaskswerecompletedmorequicklythanthecalculatedoptimaltime,while
severaltaskstookslightlylongerthanexpected.Thetasksthattookthelongestrequiredthe
participantstonavigatetoanunfamiliarportionofapage,interactwithaworkflow,locateand
selectspecificactions.Thosetasksmaybeperformedmorequicklywithaminorsetofupdatesto
theuserinterfaceand/ortheuserexperience.
51
Someparticipantsmadeerrorswhenattemptingtonavigatetowardsolvingtheirassigned
tasks.Theseerrorsmaybeassociatedwiththoseparticipantsnotbeingfamiliarwithfeaturesand
notunderstandingthepresentedworkflowoftheExscribesoftwaresystem.Asnotedabove,prior
experiencewithEHRsystemswasrelatedtosuccessfultaskcompletion.
Satisfaction
ParticipantsweresatisfiedwiththeExscribeSoftwaresystem;ratingsontheSUS(mean=
61outofapossible100)andtheCSUQ(Overallscore=4.96outofapossible7.0)demonstrated
satisfactionwiththesystem.
OntheCSUQ,participantsrankedthescale“(InterfaceQuality)”highestofthethreescales,
suggestingthatthesystemprovidedaneffectiveandefficientuserinterface.Individualtask
satisfactionratingswererelatedtoindividualuserperformance.Thoseparticipantswhowereable
tosuccessfullycompletetaskswerealsomorelikelytorankthosetasksassatisfying,whilethose
participantswhodidpoorlyorwerenotabletocompleteataskrankedthosetasksasless
satisfying.
52
SummaryofMajorFindings
ThisevaluationdemonstratedthattheExscribeEHRsystemisausablesystemwitha
relativelyshortlearningcurve.Participantswithnoexperienceusingthesystemexperiencedvery
littledifficultyunderstandingthenavigationandinformationarchitecture.Participantswithmore
overallEHRexperiencewereslightlymoreabletosolvemosttaskswithoutdifficultyorerror.
53
RiskAnalysisThefollowingtablepresentsalistoftaskspresentedwiththeriskoferrorasobservedduringthetesting.Table5.RiskAnalysis
Task DescriptionPercentComplete
RiskStatus
Task1-Demographics Verifyandmodifypatientdemographics
80% Moderate
Task2-Demographics-Change Modifypatientdemographics 100% None
Task3-ClinicalInfoRecon.Incorp. Access,reconcileandincorporateC-CDAInformation
90% Low
Task4-AddanAllergy AddanallergytoallergyList 100% None
Task5-ViewAllergyList-ChangeAllergy Changeanallergyintheallergylist 100% None
Task6-ImplantableDeviceList EnterimplantabledeviceintoEHR 100% None
Task7-ImplantableDevice-Change Modifyimplantabledeviceinformation
100% None
Task8-ProblemList-Add AddanDiagnosistotheproblemlist 100% None
Task9-ProblemList-Change Modifyaproblemintheproblemlist 100% None
Task10-ProblemList-Views Viewhistoricalprobleminformation 100% None
Task11-MedicationList-AddMedication Addamedicationtomedicationlist 100% None
Task12-MedicationList-ChangeMedication
Modifymedicationinthemedicationlist
100% None
Task13-DiagnosticImaging-AddOrder Createadiagnosticimagingorder 100% None
Task14-ChangeDiagnosticImagingOrder Modifyimagingorder 100% None
Task15-LabOrder-Add Createalaboratoryorder 90% Low
Task16-LabOrder-Change Modifyalaboratoryorder 100% None
Task17-Drug-DrugInteractionCheck Demonstratedrug-druginteractionwarning
80% Moderate
Task18-DrugAllergyInteractionCheck Demonstratedrug-allergyinteractionwarning
90% Low
Task19-ElectronicRx-NewRx Electronicallyprescribeanewdrug 90% Low
Task20-ElectronicRx-ChangeRx Modifyanelectronicprescription 80% Moderate
Task21-ElectronicRx-RefillRx Refillaprescriptionelectronically 100% None
Task22-ElectronicRx-Views Viewhistoricalprescriptiondata 100% None
Task23-ElectronicRx-CancelRx Cancelanelectronicprescription 100% None
Task24-CDS-EnableRule-ViewAttributes MakeadministrativechangetoCDSsystemtoenablerule;Viewextendedattributes
100% None
Task25-CCDAViewReconciledData GenerateandviewanupdatedCCDAdocument
100% None
54
AreasforImprovement
Thefollowingisapartiallistofpotentialareasforimprovement.Makingtheseandotherminor
enhancementswillimprovetheoveralluserexperienceoftheExscribesystemandincreasethe
effectiveness,efficiency,andsatisfactionforbothexperiencedandnoviceusers.
• IndicationofRequiredFields
o Afrequenterrorwascausedwhenparticipantsattemptedtosubmitaform
withintheinterfacethatcontainedrequiredfieldsthatwerenotcompleted.
Thisislikelybecausethesystemdidnotprovideaclearindicationofwhich
fieldsarerequired.Addingaconsistentvisualindicationofrequiredfields
wouldlikelyeliminatemanyoftheseerrorsobserved.
• Inconsistent“Add,“Search”andotherselectfunctions
o Manyerrorsoccurredwhenusersattemptedtoperforman“Add”or“Update”or
“Search”function.Thesystemprovidedseveraldifferentuserinteraction
patternsacrossseveraldifferentfunctionalareas.Providingaconsistentuser
interfaceanduserexperienceacrosstheentireapplicationmayhelptoreduce
usererrors.
• InconsistentUserInterfaceandUserExperience
o Severalerrorsoccurredwhentheusersattemptedtoperformanactionusinga
userexperiencepatternthatexistedinonefunctionalarea,butthispatterndid
notworkinthecurrentfunctionalarea.CreatingaUserExperienceandauser
interfacethatisconsistentacrossallfunctionalareaswouldhelptoimprovethe
overalluserexperienceofthesystem.
55
Appendices
AppendixA:RecruitingScreener1.Areyoumaleorfemale? 2.Haveyouparticipatedinafocusgrouporusabilitytestinthepast6months?3.Doyou,ordoesanyoneinyourhomeworkinmarketingresearch,usabilityresearch,and/orwebdesign?4.Doyou,ordoesanyoneinyourhome,haveacommercialorresearchinterestinanelectronichealthrecordsoftwareorconsultingcompany?5.Whichofthefollowingbestdescribesyourage?
20-29,30-39,40-49,50-59,60-69,70-79,80-89,90-99,100andolder.6.Whichofthefollowingbestdescribesyoureducationlevel?
• Nohighschooldiploma• Highschoolgraduate,diplomaortheequivalent• Somecollegecredit,nodegree• Tradetechnicalvocationaltraining• Associatedegree• Bachelor’sdegree• Master’sdegree• Doctoratedegree(e.g.,MD,DNP,DMD,PhD)
7.Doyourequireanyassistivetechnologiestouseacomputer?8.Pleasedescribeyourmedicalornursingcredentials9.Whatisyourcurrentjobtitle?10.Howlonghaveyouheldthisposition?(numberofyears):11.Whattypeoffacilitydoyouworkinandwhatisyourrolethere?12.Howaremedicalrecordshandledatyour(main)workplace?
_____AllPaper _____SomePaper/SomeElectronic ___AllElectronic13.HowmanyEHRsdoyouuseorhaveyouworkedwith?14.Howmanyyearshaveyouusedanelectronichealthrecord?15.HowmanyyearshaveyouusedtheExscribesystem?
56
16.Howmanyyearshaveyouusedcomputers?17Abouthowmanyhoursperweekdoyouspendusingacomputer?18.Whatcomputerplatform(s)doyouusuallyuse?19.Inthelastmonth,abouthowoftenhaveyouusedanelectronichealthrecord?
_____Didnotuselastmonth ___Everyday _____Afewtimesaweek.
57
AppendixB:InformedConsentFormTheUsabilityPeoplewouldliketothankyouforparticipatinginthisstudy.Thepurposeofthisstudyistoevaluateanelectronichealthrecordssystem.Ifyoudecidetoparticipate,youwillbeaskedtoperform several tasks using the prototype and give your feedback. The studywill last about 60minutes.AgreementI understand and agree that as a voluntary participant in the present study conducted by TheUsabilityPeople.Iamfreetowithdrawconsentordiscontinueparticipationatanytime.IunderstandandagreetoparticipateinthestudyconductedandrecordedbyTheUsabilityPeople.Iunderstandandconsenttotheuseandreleaseof thevideorecordingbyTheUsabilityPeople. Iunderstandthattheinformationandvideoisforresearchpurposesonlyandthatmynameandimagewill not be used for any purpose other than research. I relinquish any rights to the video andunderstand thevideorecordingmaybecopiedandusedbyTheUsabilityPeoplewithout furtherpermission.Iunderstandandagreethatthepurposeofthisstudyistomakesoftwareapplicationsmoreusefulandusableinthefuture.IunderstandandagreethatthedatacollectedfromthisstudymaybesharedoutsideofTheUsabilityPeople.Iunderstandandagreethatdataconfidentialityisassured,becauseonlyde-identifieddata–i.e.,identificationnumbersnotnames–willbeusedinanalysisandreportingoftheresults.I agree to immediately raise any concerns or areas of discomfortwith the study administrator. IunderstandthatIcanleaveatanytime.Pleasecheckoneofthefollowing:____YES,Ihavereadtheabovestatementandagreetobeaparticipant.____NO,Ichoosenottoparticipateinthisstudy.Signature:_____________________________________Date_____________________
58
AppendixC:ParticipantGuideORIENTATIONandINTRODUCTIONThankyouforparticipatinginthisstudy.Thesessiontodaywilllastapproximately60-90minutes.Duringthissession,youwilllookataspecialversionoftheExscribeEHR.Theproductyouwillbeusingtodaymaynotbeexactlylikethesoftwarethatyoumaybeusedto.Someofthedatamaynotmakesense,asitismerelyplaceholderdata.Thepurposeofthisstudyistogaugetheusabilityofthesoftware.Weareinterestedinhoweasy(orhowdifficult)thisversionistouse,andlearningaboutwhichofthefeaturesandbenefitswouldbemostusefultoyou.Wealsowanttoknowhowwecouldimproveit.Themoderatorwillguideyouthroughthetestingprocess.Pleaseusethesoftwareasyouwouldinarealclinicalsetting.Youwillbeaskedtosaveyourcommentsuntiltheendofataskortheendofthesessiontodiscussyourthoughtsfreely.Herearesomethingsyoushouldknowaboutyourparticipationinthissession
• TheModeratorwillguideyouthrougheachtask.• Pleasedonotworkahead.• Ifyounoticeanerror,justcanceloutandcontinuetothetask.Ifyoufeellostor
havedifficultycompletingthescenario,pleaseinformtheModerator.• TheTestingsessionwillbeaudiorecordedtheaudioforfurtherstudy.• Allinformationwillbekeptconfidential.Yournamewillnotbeassociatedwith
yourcommentsatanytime.Fictitiouspatientscenarioshavebeencreatedandpre-loadedinthesystemsoftware.TheModeratorwillaskyoutocompleteseveraltasksusingthesystem.Youwillbeaskedtoanswersomequestionsandtocompletesometasksonyourown.Trytocompletetasksasquicklyaspossible,withthefewestpossibleerrorsordeviations.Pleasedonotdoanythingmorethanasked.
59
DemographicInformationYourpatientCandiceTaylorhasarrivedfortheirscheduledappointment.TheyhavebeenreferredtoourorganizationbyanoutsideprimarycarefacilityTask1(a.5)DemographicsBeforebeginningtheirappointment,youneedtoverifyorentersomeofthedemographicinformationthatisstoredwithinthesystemPreferredLanguage ‘FrenchPatientRace ‘Asian’
PatientEthnicity ‘NotHispanicorLatino’
BirthSex MaleSexualOrientation ‘Heterosexual’
GenderIdentity ‘IdentifiesasMale'
Switchtotheeditviewandverifyand/orenterandsavethisinformationintotheEHR.
Verifythatthesechangeshavebeensaved.
60
Task2(a.5)Demographics–ChangesDuringyourconsultationwiththepatient,theyaskyoutomakethefollowingchangestotheirdemographicrecords:PreferredLanguage ‘EnglishPatientRace ‘Asian’
PatientEthnicity ‘NotHispanicorLatino’
DateofBirth 06/17/1950BirthSex MaleSexualOrientation ‘Unknown
GenderIdentity ‘IdentifiesasNon-ConformingGender
Switchtotheeditviewandverifyand/orenterandsavethisinformationintotheEHR.
Verifythatthechangeshavebeensaved.
61
Task3(b.2)ClinicalInformationReconciliationandIncorporationYourpatienthascometotheirappointmentwitha“ThumbDrive”withClinicalinformationfromanotherprovider.Forthepurposesoftesting,wehaveplacedaHealthExchangeDocumentontheWindowsDesktop.ReconcileandincorporatetheMedicalInfoincluding:Medications,DiagnosesandAllergiesthatarepresentintheCCDAfilewithpatient’sexistingrecords
Medication
• AddtheAccupriltotheFinalReconciledMedications.• AddtheKeflextotheFinalReconciledMedications.• AddtheFlexeriltotheFinalReconciledMedications.
Allergy• AddtheSulfaAllergytotheFinalReconciledAllergyList.
Problems• AddHypertensiontotheFinalReconciledProblemList.• AddHeartDiseasetotheFinalReconciledProblemList.• AddHypercholseteromiatotheFinalReconciledProblemList.• AddArthritistotheFinalReconciledProblemList.
Confirmthatthereconciledlisthasbeenincorporatedintothecurrentpatientrecord.VerballyreportanyClinicalDecisionSupportmessagesthatmayappear.
62
Task4(a.8)AddanallergyThepatientisreportingsomehivesaftertakingPenicillin.AddPenicillinasaMilddrugallergywithactivestatusandthesymptomofHives.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task5(a.8)ViewAllergyListandmakeachangeNoticethatthecurrentallergylistnowhastwoitems,ThePenicillinthatwasjustaddedandanallergytoSulfaDrugs.YounoticethatthereisnoinformationabouttheseverityandreactiontotheSulfadrug.ChangetheSulfaAllergysothatitisrecordedasanActiveMildDrugAllergywithareactionofRashVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
63
Task6(a.14)ImplantableDeviceListTheclienthashadanhistoricalsurgeryinwhichamedicaldevicewasimplanted.ThepatienthasprovidedyouwiththefollowingUniqueDeviceIdentifier(UDI):(01)10884521062856(11)141231(17)150707(10)A213B1(21)1234 (ThisUDInumbershouldbeinyour‘clipboard’andhasbeensavedasadesktopfile)PleaserecordthisMedicalInfointotheImplantableDevices(DeviceImplantHistory)list
Verifyand/orEnterandsavethisinformationintotheEHR.Verballystatethedevicedescription,identifiers,andotherattributes(brandetc.)ofthisdevice.Task7(a.14)ImplantableDeviceList-ChangeTheclientreportstheimplantabledevicewasremoved.ChangethedevicestatustoanInactivestatus.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved
64
Task8(a.6)ProblemList.Duringtheevaluationofthepatient’sleftelbow,youchoosetoaddanotheritemtotheirlistofMedicalConditions.EnteranOrthopaedictypediagnosisofOsteoarthritisVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task9(a6)ProblemList-ChangeUsethesystemtoupdatethelistofmedicalconditionsothattheMedicalConditionthatwasjustenteredhasaStatussettoNEW
Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task10(a6)ProblemList–ViewsPleaseviewtheactiveproblemlistandconfirmthatitnowindicates:HypertensionHeartDiseaseHypercholesterolemiaArthritisAndOsteoarthritisModifytheviewsothatthesystemdisplaysonlyResolveditems.Viewtheproblemlistandverifythatthehistoricalproblemlisthasbeendisplayed.
65
Task11(a.1)RecordmedicationYourpatientreportstakingTylenolforsomepaintheyhavebeenfeelingintheirleftelbow.Viewthepatient’scurrentmedslistandaddthefollowingmedication:Tylenol325mgtakenonce-a-dayevery4hours.Verifyand/orEnterandsavethisinformationintotheEHR.
Task12(a.1)ChangemedicationUponfurtherconversationwiththepatient,theyinformyouthattheyarenotinfacttakingTylenolbuthavebeentakingAleve.ChangethemedicationlistsothattheactualdrugtheyaretakingislistedasAleve(Naproxen)220mgcapsuleonceevery6to8hoursVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
66
Task13(a.3)DiagnosticImagingorderYouwanttofollowuponthepainthatyourpatienthasreportedintheirleftelbow.Usingthe“Superbill”functionsoftheEHR,Selectthepatient’sleftelbowandorderthefollowingprocedure:Orderanimagingstudy:73070ElbowLeft–2ViewsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
UponsavingtheimagingorderwithintheSuperbillfeature–theClinicalDecisionSupport(CDS)systemwilldisplayimportantinformation.VerballyidentifytheCDSitemthatappearsandoptionallyviewtheadditionalinformationthatisavailablethatsupplementstheinformationalmessage.
Task14(a.3)ChangeDiagnosticImagingorderOnsecondthought,youdecidethatyouwanttodoaslightlymorecomprehensiveimagingstudy.Changetheimagingorderthatwasjustenteredto:Orderanimagingstudy:73080ElbowLeft–3ViewsVerifyand/orEnterandsavethisinformationintotheEHR.
Uponsavingtheupdatedimagingorder-theClinicalDecisionSupport(CDS)systemwilldisplayimportantinformation.VerballyidentifytheCDSitemthatappearsandviewtheadditionalinformationthatisavailablethatsupplementstheinformationalmessage.
67
Task15(a.2)RecordaLaborderBecauseyourpatienthasseveraldiagnosesassociatedwiththeiroverallhealth,youdecidethatyouwouldalsoliketoordersomelabwork.Again,usingthe“SuperBill”“PlanItems”feature,completeaLabRequestformforthefollowinglabs:Selectthefollowinglabtests:
• Glucose2hrpp• Creatinine
Verifyand/orEnterandsavethislabrequestintotheEHR.Verifythatanychangeshavebeensaved.
Task16(a.2)ChangeLaborderUponreviewingthelaborderjustentered,yourealizeyouwouldalsoliketorunafullUrinalysisMakethenecessarychangessuchthataUrinalysishadbeenaddedtothelaborder.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
68
Task17(a.4)Drug-drug,drug-allergyinteractionchecksYourpatienthashypertension.YouwanttoprescribeadrugusingtheePrescribefeaturetohelptreatthem.PrescribeLisinopril10mg,onceadayfor30daysDuringtheCompleteRXReviewthesystemwilldisplayamoderatewarningmessageregardingthisnewmedication.ReviewthewarningdetailsbutdoNOTcontinuewiththeselection/prescriptionofthisdrug.RemovethisRXfromthelistof“Pending”drugsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task18(a.4)Drug-drug,drug-allergyinteractionThepatientpresentswithinfectedsoresontheirleftforearm.Addthefollowingmedication:PenicillinVpotassium125mg/5mLoralsolution3timesperdayfortheinfection.Thesystemwilldisplayawarningmessageregardingthisnewmedication.Reviewthewarningandcanceltheorderofthisdrug.Additionally,removethisRXfromthelistof“Pending”drugsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
69
Task19(b.3)e-Prescribing-NewRXYouneedtocreateaprescriptionforyourpatientforsomethingthattheyarenotallergicto.CreatenewprescriptionforDoxycycline100mgwithaSIG:Take1capsulebyoralrouteonetimeperdayfor5days.Includeaquantityof5pillsand0refills.Finalizeandreviewthecompleteprescriptionandthenelectronicallysend(Using“TestPharmacyat100FakeAddress”astheselectedpharmacy.)tothepharmacyTask20(b.3)e-Prescribing-ChangeprescriptionUponfurtherthoughtandreviewoftheprescriptionabove,youhavedecidedthata150mgcapsuleisbetterforthispatient.Youwillneedtomakeachangetotheelectronicprescriptionthatyouhavejustordered.ViewtheirupdatedPrescriptionlistandMakethenecessarychangessothat:Doxycycline150MGwithaSIG:Take1capsulebyoralrouteonetimeperdayfor5days.Includeaquantityof5pillsand0refills.isinsteadprescribed.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved
70
Task21(b.3)eRX-RefillprescriptionYourpatienthasrequestedthattheirpharmacyrefillsomeoftheirmedications.AccessAlloftheirprescriptionsandrefilltheDoxycyclineprescriptionforanadditional2refillsVerifyand/orEnterandsavethisinformationintotheEHR.VerifythattherefillhasbeensavedTask22(b.3)eRX–ViewprescriptioninformationYoualsowanttoexaminethispatient’shistoricalmedications.FiltertheviewofthemedicationslisttoseethefullhistoryofprescribedmedicationsVerballyidentifyandhistoricalmedicationsthataredisplayedTask23(b.3)eRxCancelprescriptionDiscontinueorcanceltheDoxycyclineprescription.VerifythattheDoxycyclinehasbeenremovedfromthepatientcurrentprescriptions.
71
Task24(a.4)CDSAdjustmentYouwanttomakeachangetothewarningmessagesand/orclinicalinformationsupportthatappearsinthesystem.AccesstheadministrativefunctionsforClinicalrulesandperformthefollowing:LocateandopenineditmodetheClinicalrulethathasbeencreatedfor“PQRS#39OsteoporosisforWomen“EnsurethatthisruleissettoActiveNavigatetoviewtheattributesofthisruleandverballystatethe:BibliographiclinkDeveloperFundingSourceVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
72
Task25(b.2)CCDAViewreconcileddataThepatientisdonewiththeirappointment.Thepatientgoestothecheckoutdesk.Whilemakingtheirnextappointment,thepatientaskstohaveafullyup-to-dateContinuityofCareDocument(CCD)thatincludesthechangesmadeasaresultoftoday'svisit.YouneedtogenerateaCCDHealthExchangeDocumentforyourclient.Viewtheclient’sContinuityofCareDocumentForthepurposesofthistesting
• settheDocumentTypetoContinuityofCareDocument”• andselecttheoptiontoDisplaythedocumenttothescreen.
ViewtheCCDdocumentthatcanbesharedwiththepatient.
73
AppendixD:SystemUsabilityScaleQuestionnaire
74
AppendixE:ComputerSystemUsabilityQuestionnairePleaseprovideyourimpressionoftheusabilityofthesystembyansweringeachofthequestionsbelow:1.Overall,IamsatisfiedwithhoweasyitistousethissystemStrongly 1234567 NA StronglyDisagree Agree2.ItwassimpletousethissystemStrongly 1234567 NA StronglyDisagree Agree3.IcaneffectivelycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree4.IamabletocompletemyworkquicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree5.IamabletoefficientlycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree6.IfeelcomfortableusingthissystemStrongly 1234567 NA StronglyDisagree Agree7.ItwaseasytolearntousethissystemStrongly 1234567 NA StronglyDisagree Agree8.IbelieveIbecameproductivequicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree9.ThesystemgiveserrormessagesthatclearlytellmehowtofixproblemsStrongly 1234567 NA StronglyDisagree Agree10.WheneverImakeamistakeusingthesystem,IrecovereasilyandquicklyStrongly 1234567 NA StronglyDisagree Agree11.Theinformation(suchasonlinehelp,on-screenmessages,andotherdocumentation)providedwiththissystemisclearStrongly1 234567 NA StronglyDisagree Agree12.ItiseasytofindtheinformationIneeded
75
Strongly 1234567 NA StronglyDisagree Agree13.TheinformationprovidedforthesystemiseasytounderstandStrongly 1234567 NA StronglyDisagree Agree14.TheinformationiseffectiveinhelpingmecompletethetasksandscenariosStrongly 1234567 NA StronglyDisagree Agree15.TheorganizationofinformationonthesystemscreensisclearStrongly 1234567 NA StronglyDisagree Agree16.TheinterfaceofthissystemispleasantStrongly 1234567 NA StronglyDisagree Agree17.IlikeusingtheinterfaceofthissystemStrongly 1234567 NA StronglyDisagree Agree18.ThissystemhasallthefunctionsandcapabilitiesIexpectittohaveStrongly 1234567 NA StronglyDisagree Agree19.Overall,IamsatisfiedwiththissystemStrongly 1234567 NA StronglyDisagree Agree
76
AppendixF.ReferencesNISTIR7741-NISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronicHealthRecords,RobertM.Schumacher,UserCentric,Inc.,SvetlanaZ.Lowry,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Nov.2010.NISTIR7742-NISTCustomizedCommonIndustryFormatTemplateforElectronicHealthRecordUsabilityTesting,RobertM.Schumacher,UserCentric,Inc.,SvetlanaZ.Lowry,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Nov.2010.NISTIR7804-TechnicalEvaluation,Testing,andValidationoftheUsabilityofElectronicHealthRecords,SvetlanaZ.Lowry,MatthewT.Quinn,MalaRamaiah,RobertM.Schumacher,Emilys.Patterson,RobertNorth,,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Feb.2012.ONC-2015EditionCertificationCompanionGuideSafety-enhanceddesign-45CFR170.315(g)(3b)3/30/2016ONC-2015EditionCertification-TestProcedurefor§170.315(g)(3)SafetyEnhancedDesignCSUQ-Lewis,J.R.(1995).IBMComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationandInstructionsforUse.InternationalJournalofHuman-ComputerInteraction,7,57-78.SUSBrooke,J.(1996)SUS:a"quickanddirty"usabilityscale.InP.W.Jordan,B.Thomas,B.A.Weerdmeester&A.L.McClelland(eds.)UsabilityEvaluationinIndustry.London:TaylorandFrancis.
Recommended