EHR Usability Test Report of Exscribe EHR (Version 6.0)

1

EHRUsabilityTestReportofExscribeEHR(Version6.0)ReportbasedonNISTIR7742CommonIndustryFormatforUsabilityTestReports

DateofUsabilityTest:February5thtoFebruary7th,2018DateofReport:February14,2018

ReportPreparedBy:

TheUsabilityPeople,LLC

4000LegatoRoad,Suite1100Fairfax,VA22033

www.TheUsabilityPeople.com

2

TABLEOFCONTENTSExecutiveSummary..................................................................................................................................................................3Introduction.................................................................................................................................................................................4Method............................................................................................................................................................................................5Participants.............................................................................................................................................................................5StudyDesign...........................................................................................................................................................................8Tasks..........................................................................................................................................................................................9TestLocation.......................................................................................................................................................................10TestEnvironment..............................................................................................................................................................11TestFormsandTools......................................................................................................................................................12ParticipantInstructions..................................................................................................................................................13Procedure.............................................................................................................................................................................14UsabilityMetrics................................................................................................................................................................15DataScoring...................................................................................................................................................................16

Results.........................................................................................................................................................................................18DataAnalysisandReporting........................................................................................................................................18Reliability.........................................................................................................................................................................18EffectivenessandEfficiency....................................................................................................................................19Satisfaction......................................................................................................................................................................20

SpecificTaskResultDetails(Includingcriteriaevaluated)............................................................................25DiscussionofFindings..........................................................................................................................................................50

Effectiveness..................................................................................................................................................................50Efficiency..........................................................................................................................................................................50Satisfaction......................................................................................................................................................................51SummaryofMajorFindings....................................................................................................................................52RiskAnalysis..................................................................................................................................................................53AreasforImprovement.............................................................................................................................................54

Appendices................................................................................................................................................................................55AppendixA:RecruitingScreener...............................................................................................................................55AppendixB:InformedConsentForm.......................................................................................................................57AppendixC:ParticipantGuide.....................................................................................................................................58AppendixD:SystemUsabilityScaleQuestionnaire............................................................................................73AppendixE:ComputerSystemUsabilityQuestionnaire..................................................................................74AppendixF.References..................................................................................................................................................76

3

ExecutiveSummary

OnFebruary5thtoFebruary7th,2017,TheUsabilityPeople,LLCconductedasummative

usabilitytestoftheExcribeEHR.ThetestwasconductedintheFairfax,VAofficeofTheUsability

Peopleoverremotetele-conferencingsessionsusingGotoMeeting.Thepurposewastotestand

validatetheusabilityofthecurrentuserinterfaceandprovideevidenceofusabilityoftheExcribe

EHRastheEHRUnderTest(EHRUT).Ten(10)healthcareprovidersmatchingthetarget

demographiccriteriaparticipatedintheusabilitytestusingtheEHRUTinsimulated,but

representativetasks.

Thestudyfocusedonmeasuringtheeffectivenessof,efficiencyof,andsatisfaction(ISO

9241-11)withExscribeamongasampleofparticipantsrepresentingpotentialusersofthesystem.

Performancedatawascollectedontwenty-five(25)taskstypicallyconductedonanEHR.Tasks

createdwerebaseduponthecriteriaspecifiedwithinthetestprocedurestructureforevaluating

conformanceofElectronicHealthRecord(EHR)technologytothecertificationcriteriadefinedin

certificationcriteriaidentifiedin45CFRPart170SubpartCoftheHealthInformationTechnology:

2015EditionHealthInformationTechnology(HealthIT)CertificationCriteria.

ResultsofthestudyindicatedthattheExscribeEHRsystemwasquitesatisfactorywith

regardstoeffectivenessandefficiencyandthattheparticipantsweresatisfiedwiththesystem.

4

Introduction

TheElectronicHealthRecordSystemUnderTest(EHRUT)testedforthisstudy,theExcribe

EHR(V6.0),wasspecificallydesignedtopresentmedicalinformationtohealthcareproviderson

desktopcomputersinstandardambulatoryhealthcaresettings.Thisstudytestedandvalidatedthe

usabilityoftheExscribeEHRsoftwareuserinterfaceandprovidesevidenceoftheusabilityofthe

ExscribeEHRwithrepresentativeexercisesandinrealisticuserconditions.Tothisend,measures

ofeffectivenessandefficiency,suchastimeontask,numberoferrorsmade,andcompletionrates

werecapturedduringusabilitytesting.Satisfactionwasassessed,andusercommentscollected

usingtwoindustry-standardquestionnaires:TheSystemUsabilityScale(SUS)andtheComputer

SystemUsabilityQuestionnaire(CSUQ).

5

MethodParticipants

Ten(10)individuals(7womenand3men)participatedintheEHRUT(s)usingtheExscribe

EHR.Participantswerephysicians,nurses,medicalassistants,and/orotherhealthcare/HealthIT

practitioners.Participantswererecruitedfromadatabaseofpotentialparticipantsmaintainedby

TheUsabilityPeople,LLC.Thecontactscontainedwithinthisdatabaseweregeneratedfrompast

participants,andviapotentialparticipantsresponsestopostingsinInternetandsocialmediasites,

andalinkatthebottomofTheUsabilityPeoplewebsite.Thosewhorespondedtotheinvitationto

takepartinthestudyweredirectedtoanonlinequestionnairethatservedastheparticipant

screener.(ThescreeningquestionnaireisprovidedasAppendix1.)Participantsmeetingthe

criteriaforparticipationinthestudywerecontactedandscheduledviatelephoneandemail.

ParticipantsintheusabilitytestoftheExscribeEHRhadavarietyofhealthcare

backgroundsanddemographiccharacteristics.

Table1presentsparticipantcharacteristics,includingdemographics,professional

experience,computingexperience,andpreviousEHRexperience.Participantcharacteristicsreflect

theaudienceofcurrentandfutureusersandmeetthecriteriadesignatedinthe2015Edition

CertificationCompanionGuideforSafety-enhanceddesign-45CFR170.315(g)(3).Noneofthe

participantswerefromthevendororganization(Exscribe)thatproducedandsuppliedthe

evaluatedsystemnordidanyparticipanthaveanydirectconnectiontothetestingorganization

(TheUsabilityPeople,LLC).Allparticipantswerecompensatedfortheirtime.

6

Table1.ParticipantCharacteristics

PartID Gender Age Education Role/Title

ProfessionalExperience(Months)

ComputerExperience(Months)

Experiencewith

Exscribe(Months)

AssistiveTechNeeds

P01 Male 50to59 Doctor MD 312 96 0 None

P02 Female 20to29

Tradetechnicalvocationaltraining MedicalAssistant 60 60 0 None

P03 Male 40to49Bachelor’sdegree RegisteredNurse 180 84 0 None

P04 Female 20to29

Tradetechnicalvocationaltraining LPN 36 36 0 None

P05 Female 50to59 Master’sdegreeInformatics

NurseSpecialist 72 96 0 None

P06 Female 40to49Bachelor’sdegree ClinicalAnalyst 252 192 0 None

P07 Male 40to49 AssociatedegreeChiropracticAssistant 144 36 0 None

P08 Female 40to49Bachelor’sdegree RegisteredNurse 36 36 0 None

P09 Female 50to59 Master’sdegree RegisteredNurse 72 60 0 None

P10 Female 30to39Bachelor’sdegree RegisteredNurse 144 180 0 None

7

SummaryofParticipantCharacteristics:Participantshadexperiencewiththeoccupationandexpertisethatalignswiththecapabilityundertesting.Thecohortofuserswhoareselectedasparticipantswasvariedwiththeproductanditsintendedusersandwasnotlimitedtoclinicians.Thedemographiccharacteristicsofthetestparticipantcharacteristicsreflectedtheaudienceofcurrentandfutureusers.Gender Male 3Female 7 AgeRange 20to29 230to39 140to49 450to59 360to69 070to79 0 Education Somecollegecredit,nodegree 0Tradetechnicalvocationaltraining 2Associatedegree 1Bachelor’sdegree 4Master’sdegree 2DoctorateDegree 1 YearsofExperiencewithExscribeNone 10Upto3years 03to5years 05to10years 0Morethan10years 0

8

StudyDesign

TheoverallobjectiveofthisusabilitytestwastouncoverareaswheretheExscribeEHR

systemperformedwell–thatis,effectively,efficiently,andwithsatisfaction–andareaswherethe

systemfailedtoservetheclinicaldocumentationandworkflowneedsofusers.Datafromthistest

maybeusedasabaselineforfuturetestsofupdatedversionsofExscribeand/orforcomparing

ExscribesoftwarewithotherEHRspresentingthesametasks.Inshort,thistestingservesasbotha

meanstorecordorbenchmarkcurrentusabilityandtoidentifyareaswhereimprovementsmustbe

made.

ParticipantshadarangeofexperiencewithEHRsingeneral,butnonehadanyprior

experiencewiththeExscribesystem.ParticipantscompletedtheExscribeusabilitystudyduring

individual60-90-minuteGotoMeetingsessions.Duringthetest,eachparticipantinteractedwith

variouscomponentsoftheExscribesoftwaresystem.Eachparticipantwasprovidedwiththesame

instructions.

TheExscribeEHRwasevaluatedforeffectiveness,efficiencyandsatisfactionasdefinedby

thefollowingmeasurescollectedandanalyzedforeachparticipant:

• Numberoftaskssuccessfullycompletedwithoutassistance

• Timetocompletethetasks

• Numberanddescriptionoferrors

• Pathdeviations

• Participant’sverbalizations(comments)

• Participant’ssatisfactionratingsofthesystem

9

TasksAtotaloftwenty-five(25)taskswereconstructedbyTheUsabilityPeople,LLC.(inclose

collaborationwithExscribestaff)toberealisticandrepresentativeoftheactivitiesausermight

engageinwhileusingExscribeinactualmedicalsettings.Thetwenty-five(25)taskswerecreated

baseduponthecriteriaspecifiedwithinthetestprocedurestructureforevaluatingconformanceof

ElectronicHealthRecord(EHR)technologytothecertificationcriteriaasdefinedin45CFRPart

170SubpartCoftheHealthInformationTechnology:Standards,ImplementationSpecifications,

andCertificationCriteriaforElectronicHealthRecordTechnology.Thetasksfocusedonthetwelve

2015EditioncertificationcriteriaspecifiedbyONC,specifically:

• Section170.315(a)(1)Computerizedproviderorderentry–medications

• Section170.315(a)(2)Computerizedproviderorderentry–laboratory

• Section170.315(a)(3)Computerizedproviderorderentry–diagnosticimaging

• Section170.315(a)(4)Drug-drug,drug-allergyinteractionchecks

• Section170.315(a)(5)Demographics

• Section170.315(a)(6)Problemlist

• Section170.315(a)(7)Medicationlist

• Section170.315(a)(8)Medicationallergylist

• Section170.315(a)(9)Clinicaldecisionsupport

• Section170.315(a)(14)Implantabledevicelist

• Section170.315(b)(2)Clinicalinformationreconciliationandincorporation

• Section170.315(b)(3)Electronicprescribing

AcopyofthetaskspresentedtoparticipantsintheusabilitytestoftheExscribeEHRcanbefound

inAppendixC.

10

TestLocation

AllparticipantsweretestedontheExscribeEHRsystemduringremoteconferencing

sessionsusingGotoMeeting.Eachparticipantwasrequestedinadvancetosecureaquietroom

withminimaldistractionsandadesktoporlaptopcomputerthatcouldconnecttotheInternetwith

aGotoMeetingsession.Althoughthetypeofcomputer,operatingsystemanddisplayresolutionof

theremoteparticipantsystemwasunknown,thesystemthatwasusedbythetestadministrator

andcontrolledbytheremoteparticipantwasaDellInspironLaptoprunningtheWindows10

professionaloperatingsystemataresolutionof1366x768pixels.DuringagivenGotoMeeting

session,onlythetestadministratorandparticipantcommunicatedwithoneanother.

TheGotoMeetingusabilitytestsessionwasconductedbyatestadministratorfromthe

testingorganization(TheUsabilityPeople,LLC)workingfromasmallconferenceroomatThe

UsabilityPeople’sFairfax,VAlocation.Adataloggerfromthetestingorganizationalsotook

detailednotesoneachsession,includingusercommentsandotherratingsfollowingeachtask.

Duringasessionboththetestadministratorandthedataloggercouldseeonlytheparticipant’s

screenandheartheparticipant’scomments,questions,andresponses.

11

TestEnvironment

WhiletheEHRUTtypicallywouldbeusedinahealthcareoffice,orambulatorycenter

facility,testingoftheExscribeEHRsystemwasconductedviaremoteconnectionduringindividual

GotoMeetingsessions.EachparticipantcalledintoaGotoMeetingsessionandwasconnectedby

thetestadministratortotheapplication.

TheExscribesystemitselfonaWindows™clientplatformonaLANconnectionusinga

sampledatabasethatwassetupspecificallyforthetest.Participantsusedamouseandkeyboard

wheninteractingwiththeEHRUTandweregivenremotecontroloftheadministrator’s

workstationtoperformthetasks.

12

TestFormsandTools

Aspartoftheusabilitytest,severaldocumentsandinstrumentswereused.Examplesofthe

documentsusedduringtheusabilitytest,includinganinformedconsentform,thetasks,andpost-

testquestionnaires,canbefoundinAppendicesBtoE,respectively.

Participants’interactionwiththeExscribeEHRsoftwarewascapturedandrecordeddigitally

usingtheMoraescreencapturesoftwarerunningonthetestadministrator’sworkstation.Verbal

responseswererecordedthrougheitherthemicrophoneintegratedintotheparticipant’scomputer

orthroughatelephoneconnection.Thisinformationwaselectronicallytransmittedtothe

administratorandtothedataloggerduringeachtestsession.

13

ParticipantInstructions

Theadministratorreadthefollowinginstructionsaloudtoeachparticipant:

Thankyouforparticipatinginthisstudy.Yourinputisveryimportant.Oursessiontodaywilllastabout60-90minutes.Duringthattime,youwilluseaninstanceofanelectronichealthrecord.Iwillaskyoutocompleteafewtasksusingthissystemandanswersomequestions.

Pleasenotethatwearenottestingyou;wearetestingthesystem.Therefore,ifyouhaveanydifficultythismaymeanthatsomethingneedstobeimprovedinthesystem.Iwillbehereincaseyouneedspecifichelp,butIamnotabletoinstructyouorprovidehelpinhowtousetheapplication.

Overall,weareinterestedinhoweasy(orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Ididnothaveanyinvolvementinitscreation,sopleasebehonestwithyouropinions.Alloftheinformationthatyouprovidewillbekeptconfidentialandyournamewillnotbeassociatedwithyourcommentsatanytime.Shouldyoufeelitnecessaryyouareabletowithdrawatanytimeduringthetesting.Participantswerethengiventaskstocomplete.

14

Procedure

Uponconnectiontotheonlinemeetingtool(GotoMeeting),eachparticipantwasgreeted,

hisorheridentityverified,andmatchedtoanameontheparticipantschedule.Participant

demographicinformationwasalsoverified,includinginformationregardingoverallcomputer

experienceandparticipantexperiencewiththeEHUT.Participantnameswerereplacedwith

participantIDssothatagivenindividual’sdatacannotbelinkedtohis/heridentity.Priorto

beginningtesting,eachparticipantreviewedandsignedaninformedconsentform(SeeAppendix

B).

StaffmembersoftheUsabilityPeople,ausabilitytestadministratoradministeredthetest.

Theadministratormoderatedthesessionbyprovidingbothverbalandwritteninstructionsforthe

overallusabilitytestandforeachofthetaskscomprisingthetest.Theadministratoralso

monitoredtasksuccess,pathdeviations,numberanddescriptionoferrors,andaudio-recorded

participantverbalcomments.Adataloggerloggedtasktimes,obtainedpost-taskratingdata,and

tooknotesonparticipantcommentsandadministratorfeedback.

Foreachofthetwenty-five(25)tasks,participantswereprovidedwritteninstructionsto

theircomputers.Followingtheadministrator’sinstructions,eachparticipantperformedeachtask

byfirstreadingthetaskthenstatinginhisorherownwordshisorherinterpretationofthetask

requirements.Whentheparticipant’sinterpretationmatchedtheactualgoalofthetask,the

administratorinstructedtheparticipanttobeginandtasktimingbegan.Tasktimewasstopped

andrecordedwhenthetestadministratorobservedontheirworkstationthattheparticipanthad

successfullycompletedthetask.Ifaparticipantfailedtocompleteataskbeforetheexpected

amountoftimeforeachtask,thattaskwasmarkedas“TimedOut.”Aftereachtask,thetest

administratoraskedtheparticipant,“Onascalefrom1to5,where1is‘VeryDifficultand5is‘Very

Easy,’howsatisfiedwereyouwiththeeaseofuseforthistask?”Thissameprocedurewas

conductedforeachofthetwenty-five(25)tasks.

15

Followingcompletionofthetwenty-five(25)EHRtasks,theadministratorelectronically

presentedtotheparticipanttwopost-testquestionnaires(SystemUsabilityScale(SUS),see

AppendixDandComputerSystemUsabilityQuestionnaire(CSUQ),seeAppendixE).Afterthe

participantcompletedbothquestionnaires,theadministratorthankedeachparticipantforhisor

hertimeandallowedtheparticipanttomakeanycommentsonoraskanyquestionsaboutthe

systemand/orthetaskspresented.

Foreachsession,theparticipant’sschedule,demographicinformation,tasksuccessrate,

timeontask,errors,deviations,verbalresponses,andpost-testquestionnaireweredigitally

recorded.Thesystemwasthenresettopropertestconditionsforthenextparticipant.

UsabilityMetrics

AccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronic

HealthRecords(NISTIR7741,November2010)EHRsshouldsupportaprocessthatprovidesahigh

levelofusabilityforallusers.Thegoalisforuserstointeractwiththesystemeffectively,efficiently,

andwithanacceptablelevelofsatisfaction.Tothisend,metricsforeffectiveness,efficiencyand

usersatisfactionwerecapturedduringtheusabilitytesting.Thegoalsofthetestweretoassess:

• EffectivenessoftheExscribeEHRsoftwarebymeasuringparticipantsuccessratesand

errors.

• EfficiencyoftheExscribeEHRsoftwareSoftwarebymeasuringtheaveragetasktimeand

pathdeviations.

• SatisfactionwiththeExscribeEHRsoftwarebymeasuringease-of-useratings.

16

DataScoringTable2detailshowtaskswerescored,errorsevaluated,andthetimedataanalyzed.Table2.ScoringProtocolsforEffectiveness,Efficiency,andSatisfaction

Measures RationaleandScoring

Effectiveness:• TaskSuccess

A task was counted as “Success” if the participant was able to achieve thecorrect outcome,without assistance,within the time allotted on a per taskbasis.ThetotalnumberofSuccesseswascalculatedforeachtaskandthendividedbythetotalnumberoftimesthattaskwasattempted.Resultsareprovidedasapercentage.

Effectiveness:• TaskFailures

If the participant abandoned the task, did not reach the correct answer orperformed it incorrectly, or reached the end of the allotted time beforesuccessfulcompletion,thetaskwascountedas“Fail.”Notasktimesweretakenforfailedattempts.Thetotalnumberoferrorswascalculatedforeachtaskanddividedbythetotalnumberoftimesthattaskwasattempted.Resultsarepresentedastheaverageerrorrate.Note:Notalldeviationsarecountedaserrors

Effectiveness:• Prompted

Successes

Becausesometasksaredependentuponthesuccessfulcompletionofprevioustasks,participantsmayreceivealimitednumberof“prompts”tohelppreparethesystemdataforthepre-requisitesforsubsequenttasks.Whenaparticipantwasable tocomplete thedataentryona taskwith3orfewer prompts, the taskwas counted as an “Assisted” completion. No tasktimeswererecordedforAssistedcompletions.

Efficiency:• Task

Deviations

The participant’s path (i.e., steps) through the application was recorded.Deviations occur if for example, the participant navigated to an incorrectscreen, clicked on an incorrect menu item, followed an incorrect link, orinteractedincorrectlywithanon-screencontrol.

Efficiency:• TaskTime

Each task was timed from the administrator’s prompt “Begin” until said,“Done.” If the participant failed to say, “Done,” timing stopped when theparticipantstoppedperformingthetask.Onlytasktimesfortasksthatweresuccessfullycompletedwereincludedintheaveragetasktimeanalysis.Averagetimepertaskwascalculatedforeachtask.

17

Satisfaction:• EaseofUse

ratings• System

Satisfaction

Participant’ssubjective impressionof theeaseofuseof theapplicationwasmeasured by administering both a single post-task question aswell as twopost-sessionquestionnaires.Aftereachtask,theparticipantdeterminedonascaleof1to5theirsubjectivesatisfaction with performance on the task. These data are averaged acrossparticipants.Tomeasureparticipants’confidenceinandlikeabilityoftheEHRoverall,thetesting team administered electronic versions of the SystemUsability Scale(SUS)andtheComputerSystemUsabilityQuestionnaire(CSUQ).SeetheSUSquestionnaireasAppendixD.,andtheCSUQasAppendixE.

18

Results

DataAnalysisandReporting

TheresultsoftheusabilitytestoftheExscribeEHRsoftwaresystemwereanalyzed

accordingtothemethodsdescribedintheUsabilityMetricssectionaboveandaredetailedbelow.

Notethattheresultsshouldbeevaluatedrelativetothestudyobjectivesandgoals,asoutlinedin

thestudydesignsectionabove.Thedatashouldyieldactionableresultsthat,ifcorrected,yield

material,positiveimpactonuserperformance.

Reliability

Duringthedatacollectionphase,itwasobservedthatthesystemprovidedausually

consistentandreliableinterfacetoeachparticipantastheycompletedtheirtasks.Aseach

participantcompletedtheirassignedtasks,thesystemprovidedthesameinformationand

respondedtotheirinputwiththesameverbiageandusingthesamemodeofcommunication(e.g.

Pop-upmessage,orembeddedassistance).

19

EffectivenessandEfficiency

Table3presentsasummaryofoveralltaskperformanceshowingtask,meantimeontask,

taskcompletionrates,meanpathdeviationsandmeantasksatisfaction:

Table3.UsabilityTestResults

TaskMeanTaskTime

SD

CompletionRate(%)

Mean#Path

Deviations

SD

MeanTaskSatisfaction

SD

Task1-Demographics 2:26 1:29 80% 2.30 2.10 3.70 0.64Task2-Demographics-Change 1:30 0:45 100% 0.70 1.00 4.10 0.54Task3-ClinicalInfoRecon.Incorp. 3:49 1:33 90% 2.00 1.41 4.20 0.40Task4-AddanAllergy 2:06 1:08 100% 1.90 1.14 3.90 0.54Task5-ViewAllergyList-ChangeAllergy

0:27 0:09 100% 0.00 0.00 4.80 0.40

Task6-ImplantableDeviceList 0:56 0:19 100% 0.10 0.30 4.70 0.46Task7-ImplantableDevice-Change 0:27 0:22 100% 0.40 0.66 4.80 0.40Task8-ProblemList–Add 0:55 0:10 100% 0.80 0.60 4.50 0.50Task9-ProblemList-Change 0:23 0:26 100% 0.10 0.30 4.60 0.66Task10-ProblemList-Views 0:28 0:12 100% 0.10 0.30 4.20 0.60Task11-MedicationList-AddMedication

0:50 0:12 100% 1.20 0.40 4.10 0.54

Task12-MedicationList-ChangeMedication

0:42 0:11 100% 0.40 0.66 4.50 0.50

Task13-DiagnosticImaging-AddOrder

1:07 0:15 100% 0.60 0.66 4.40 0.66

Task14-ChangeDiagnosticImagingOrder

0:17 0:05 100% 0.10 0.30 4.80 0.40

Task15-LabOrder-Add 1:11 0:39 90% 1.10 1.70 3.90 0.83Task16-LabOrder–Change 0:42 0:08 100% 0.10 0.30 4.20 0.75Task17-Drug-DrugInteractionCheck

2:17 1:00 80% 2.40 2.15 2.60 0.66

Task18-DrugAllergyInteractionCheck

1:20 0:52 90% 1.90 1.87 2.90 0.94

Task19-ElectronicRx-NewRx 1:37 0:30 90% 1.50 1.75 3.60 0.66Task20-ElectronicRx-ChangeRx 1:41 0:56 80% 1.70 2.24 3.60 0.66Task21-ElectronicRx-RefillRx 0:41 0:16 100% 1.10 0.94 4.20 0.60Task22-ElectronicRx-Views 0:37 0:27 100% 0.30 0.90 4.00 0.77Task23-ElectronicRx-CancelRx 0:31 0:16 100% 0.10 0.30 4.20 0.87Task24-CDS-EnableRule-ViewAttributes

1:18 0:25 100% 0.80 0.98 4.20 0.60

Task25-CCDAViewReconciledData

1:57 0:38 100% 0.70 0.64 4.70 0.46

20

AsTable3shows,relativetooptimalperformancestandardsasdefinedbyExscribeand

TheUsabilityPeople,participantperformanceintheExscribeEHRsoftwareusabilitytestwasquite

satisfactory.Theoverallaveragetaskcompletionratewasninety-six(96)percent.

Satisfaction

IndividualTaskSatisfaction

Participantsverballyindicatedtheirsatisfactionwiththeeaseofuseforeachtaskusinga

scaleof“1”(“VeryDifficult”)to“5,”(“VeryEasy”).AsFigure1showsindividualtasksatisfaction

rangedfromalowof2.6outof5onTask17(Drug-Druginteractioncheck)toahighof4.8outof5

onTasks5,7,and14(Viewallergylist-ChangeAllergy,ImplantableDevice–Change,andChange

DiagnosticImagingOrder).

21

Figure1.SatisfactionRatingsofIndividualTasks

0.00 1.00 2.00 3.00 4.00 5.00

Task1-Demographics

Task2-Demographics-Change

Task3-ClinicalInfoRecon.Incorp.

Task4-AddanAllergy

Task5-ViewAllergyList-ChangeAllergy

Task6-ImplantableDeviceList

Task7-ImplantableDevice-Change

Task8-ProblemList- Add

Task9-ProblemList- Change

Task10-ProblemList- Views

Task11-MedicationList-AddMedication




Task15-LabOrder-Add

Task16-LabOrder- Change

Task17-Drug-DrugInteractionCheck


Task19-ElectronicRx- NewRx

Task20-ElectronicRx-ChangeRx

Task21-ElectronicRx-RefillRx

Task22-ElectronicRx-Views

Task23-ElectronicRx- CancelRx

Task24-CDS-EnableRule-ViewAttributes


AverageTaskSatisfaction

22

IndividualParticipantSatisfaction

Ingeneral,theparticipantsweresatisfiedwiththeeaseofuseoftheExscribeEHRsoftwaresystem.Thefollowingchartdisplaysoverallsatisfactionforeachparticipant:

Theaverageoveralltasksatisfactionratewas4.14outof5.

3.64

4.20

4.16

4.80

4.08

4.24

4.12

4.04

3.80

4.28

1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00

P01

P02

P03

P04

P05

P06

P07

P08

P09

P10

OverallSatisfaction(average)

23

SystemUsabilityScale

TheSystemUsabilityScale(SUS)isasimple,10-itemLikert-typeattitudescaleprovidinga

globalsubjectiveassessmentofusabilityfromtheuser’sperspective(JohnBrookeatDigital

EquipmentCompanydevelopedtheSUSin1986).TheSUSscaleisscoredfrom0to100;scores

under60representsystemswithlessthanoptimalusability,scoresover80areconsideredbetter

thanaverage.SeeAppendixDforacopyoftheSUS.

ThemeantotalSUSscoretheExscribeEHRwassixty-one(61)andrangedfromalowof

forty-five(45)andahighofseventy-five(75).Overall,participant-usersratedtheirsatisfaction

withtheExscribesoftwaresystemtobewithintherangeofausableandsatisfyingEHR.

ThefollowingchartshowstheSUSscorebyeachparticipant:

0 10 20 30 40 50 60 70 80

Participant1

Participant2

Participant3

Participant4

Participant5

Participant6

Participant7

Participant8

Participant9

Participant10

55

72.5

75

62.5

45

65

50

70

60

57.5

SUSScorebyParticipant

24

ComputerSystemUsabilityQuestionnaire

UsingtheComputerSystemUsabilityQuestionnaire(CSUQ;Lewis,J.R.(1995).(See:IBM

ComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationandInstructionsforUse.

InternationalJournalofHuman-ComputerInteraction,7:1,57-78).),participantsratedeachof19

itemsoftheCSUQquestionnaireonascalefrom1to7,witharatingof7beingmostinagreement

withthepositively-wordeditem.Responsesforeachitemweresummedandaveragedtofour

scales–InterfaceQuality,InformationQuality,SystemUsefulness-andanoverallscale.See

AppendixEforacopyoftheCSUQ.

Figure2displaysCUSQratingsforeachofthefourscales.Ingeneral,participantsinthe

Exscribestudyratedsystemusabilitytobehigh.OnInterfaceQualitytheaveragescoreforthe

participantswas5.47/7;onInformationQualitytheaveragescore4.86/7;onSystemUsefulness

theaveragescorewas4.90/7;andtheoverallaverageCUSQscorewas4.96/7.

3.00 3.50 4.00 4.50 5.00 5.50 6.00 6.50 7.00

Overall Score

System Usefulness

Information Quality

Interface Quality

4.96

4.90

4.86

5.47

User Rating (Out of 7)

Computer System Usability Questionnaire

25

SpecificTaskResultDetails

ParticipantNumber

Task1-Demographics

TaskTime Outcome#Path

DeviationsTask

SatisfactionP01 4:06 Fail 6 2P02 1:09 Success 1 4P03 5:37 Success 2 4P04 3:28 Success 1 4P05 1:10 Success 0 4P06 1:18 Success 1 4P07 1:31 Success 1 4P08 3:31 Success 4 4P09 3:51 Fail 6 3P10 1:43 Success 1 4

ExpectedTimeonTask 2:00 (SD) AverageTimeonTask 2:26 1:29 AverageTaskSatisfaction 3.70 0.64 Average#PathDeviations 2.30 2.10 PercentSuccess 80%

26

ParticipantNumber

Task2-Demographics-Change


DeviationsTask

SatisfactionP01 3:32Success 3 3P02 0:51Success 0 4P03 1:54Success 0 4P04 1:05Success 0 5P05 1:24Success 1 4P06 1:01Success 1 4P07 1:18Success 0 4P08 1:22Success 0 4P09 1:43Success 2 4P10 0:53Success 0 5

ExpectedTimeonTask 0:30 (SD)AverageTimeonTask 1:30 0:45AverageTaskSatisfaction 4.10 0.54Average#PathDeviations 0.70 1.00PercentSuccess 100%

27

ParticipantNumber

Task3-ClinicalInfoRecon.Incorp.


DeviationsTask

SatisfactionP01 6:25Success 3 5P02 3:34Success 1 4P03 7:35TimeOut 4 4P04 2:41Success 0 5P05 2:51Success 0 4P06 2:53Success 2 4P07 3:02Success 2 4P08 4:08Success 3 4P09 4:33Success 4 4P10 4:11Success 1 4

ExpectedTimeonTask 4:00 (SD)AverageTimeonTask 3:49 1:33AverageTaskSatisfaction 4.20 0.40Average#PathDeviations 2.00 1.41PercentSuccess 90%

28

ParticipantNumber

Task4-AddanAllergy


DeviationsTask

SatisfactionP01 4:38 Success 4 3P02 1:10 Success 2 3P03 2:56 Success 1 4P04 2:13 Success 2 4P05 1:11 Success 1 5P06 1:22 Success 2 4P07 1:43 Success 1 4P08 1:32 Success 1 4P09 3:19 Success 4 4P10 0:54 Success 1 4

ExpectedTimeonTask 1:00 (SD)

AverageTimeonTask 2:06 1:08 AverageTaskSatisfaction 3.90 0.54 Average#PathDeviations 1.90 1.14 PercentSuccess 100%

29

ParticipantNumber

Task5-ViewAllergyList-ChangeAllergy


DeviationsTask




30

ParticipantNumber

Task6-ImplantableDeviceList


DeviationsTask




31

ParticipantNumber

Task7-ImplantableDevice-Change


DeviationsTask




32

ParticipantNumber

Task8-ProblemList-Add


DeviationsTask




33

ParticipantNumber

Task9-ProblemList-Change


DeviationsTask




34

ParticipantNumber

Task10-ProblemList-Views


DeviationsTask




35

ParticipantNumber

Task11-MedicationList-AddMedication


DeviationsTask




36

ParticipantNumber



DeviationsTask




37

ParticipantNumber



DeviationsTask




38

ParticipantNumber



DeviationsTask


ExpectedTimeonTask 0:30 (SD) AverageTimeonTask 0:17 0:05 AverageTaskSatisfaction 4.80 0.40 Average#PathDeviations 0.10 0.30 PercentSuccess 100%

39

ParticipantNumber

Task15-LabOrder-Add


DeviationsTask

SatisfactionP01 1:35 Success 1 3P02 0:41 Success 0 4P03 1:07 Success 0 5P04 0:44 Success 0 5P05 1:25 Success 1 3P06 1:13 Success 1 3P07 1:24 Success 1 4P08 1:17 Success 1 4P09 3:11 Fail 6 3P10 1:15 Success 0 5



40

ParticipantNumber

Task16-LabOrder-Change


DeviationsTask




41

ParticipantNumber

Task17-Drug-DrugInteractionCheck


DeviationsTask

SatisfactionP01 1:33 Success 1 3P02 1:53 Success 2 3P03 2:40 Success 2 2P04 2:01 Success 0 4P05 3:02 Success 1 2P06 2:52 Success 1 2P07 2:28 Success 2 3P08 4:23 Fail 7 2P09 4:33 Fail 6 2P10 1:44 Success 2 3



42

ParticipantNumber



DeviationsTask




43

ParticipantNumber

Task19-ElectronicRx-NewRx


DeviationsTask




44

ParticipantNumber

Task20-ElectronicRx-ChangeRx


DeviationsTask

SatisfactionP01 2:14 Success 1 3P02 0:58 Success 0 5P03 3:25 Fail 6 3P04 2:23 Success 1 4P05 2:04 Success 0 3P06 0:45 Success 0 4P07 0:54 Success 0 4P08 3:30 Fail 6 3P09 1:39 Success 2 3P10 2:34 Success 1 4



45

ParticipantNumber

Task21-ElectronicRx-RefillRx


DeviationsTask




46

ParticipantNumber

Task22-ElectronicRx-Views


DeviationsTask




47

ParticipantNumber

Task23-ElectronicRx-CancelRx


DeviationsTask




48

ParticipantNumber

Task24-CDS-EnableRule-ViewAttributes


DeviationsTask




49

ParticipantNumber



DeviationsTask




50

DiscussionofFindings

Ingeneral,theparticipantsperformedverywellandweresatisfiedwithExscribeEHR

softwaresystem.Afewoftheparticipantsstruggledwithsomeportionsofafewtasksbutin

generalmostwereabletosuccessfullycompletealargemajorityofthetaskswithlittleorno

difficulty.Giventhatnoparticipantshadanypriorexperiencewiththesystem,thisshowsthatthe

ExcribeEHRisaneasy-to-useandaneasy-to-learnEHR.Participantsweremostlyabletoperform

alltaskssuccessfullyontheirownwithnoassistanceorexternaldocumentation.Theparticipant

averageperformanceratewashigh.TheExscribesystemappearstobeausableEHR.

Effectiveness

Ofthetwenty-five(25)taskspresented,alargemajorityofthetasksweresuccessfully

completedbyalloftheparticipants.Overallofparticipants,themeansuccessfultaskcompetition

ratewashighwithanoverallaveragerateofninety-six(96)percentindicatingthatingeneralthe

participantshadlittleornodifficultycompletingthetasks.

TheamountofpriorexperiencewithEHRsystemswasonlyslightlyrelatedtosuccessful

taskperformanceanderrorrates;participantswithmorepriorexperienceweremoreslightly

likelytosuccessfullycompletetaskswithlesserrorsthanthosewithlesspriorexperience.

Efficiency

Participantswhosuccessfullycompletedtasksgenerallycompletedthosetaskswithinan

acceptabletime.Sometaskswerecompletedmorequicklythanthecalculatedoptimaltime,while

severaltaskstookslightlylongerthanexpected.Thetasksthattookthelongestrequiredthe

participantstonavigatetoanunfamiliarportionofapage,interactwithaworkflow,locateand

selectspecificactions.Thosetasksmaybeperformedmorequicklywithaminorsetofupdatesto

theuserinterfaceand/ortheuserexperience.

51

Someparticipantsmadeerrorswhenattemptingtonavigatetowardsolvingtheirassigned

tasks.Theseerrorsmaybeassociatedwiththoseparticipantsnotbeingfamiliarwithfeaturesand

notunderstandingthepresentedworkflowoftheExscribesoftwaresystem.Asnotedabove,prior

experiencewithEHRsystemswasrelatedtosuccessfultaskcompletion.

Satisfaction

ParticipantsweresatisfiedwiththeExscribeSoftwaresystem;ratingsontheSUS(mean=

61outofapossible100)andtheCSUQ(Overallscore=4.96outofapossible7.0)demonstrated

satisfactionwiththesystem.

OntheCSUQ,participantsrankedthescale“(InterfaceQuality)”highestofthethreescales,

suggestingthatthesystemprovidedaneffectiveandefficientuserinterface.Individualtask

satisfactionratingswererelatedtoindividualuserperformance.Thoseparticipantswhowereable

tosuccessfullycompletetaskswerealsomorelikelytorankthosetasksassatisfying,whilethose

participantswhodidpoorlyorwerenotabletocompleteataskrankedthosetasksasless

satisfying.

52

SummaryofMajorFindings

ThisevaluationdemonstratedthattheExscribeEHRsystemisausablesystemwitha

relativelyshortlearningcurve.Participantswithnoexperienceusingthesystemexperiencedvery

littledifficultyunderstandingthenavigationandinformationarchitecture.Participantswithmore

overallEHRexperiencewereslightlymoreabletosolvemosttaskswithoutdifficultyorerror.

53

RiskAnalysisThefollowingtablepresentsalistoftaskspresentedwiththeriskoferrorasobservedduringthetesting.Table5.RiskAnalysis

Task DescriptionPercentComplete

RiskStatus

Task1-Demographics Verifyandmodifypatientdemographics

80% Moderate

Task2-Demographics-Change Modifypatientdemographics 100% None

Task3-ClinicalInfoRecon.Incorp. Access,reconcileandincorporateC-CDAInformation

90% Low

Task4-AddanAllergy AddanallergytoallergyList 100% None

Task5-ViewAllergyList-ChangeAllergy Changeanallergyintheallergylist 100% None

Task6-ImplantableDeviceList EnterimplantabledeviceintoEHR 100% None

Task7-ImplantableDevice-Change Modifyimplantabledeviceinformation

100% None

Task8-ProblemList-Add AddanDiagnosistotheproblemlist 100% None

Task9-ProblemList-Change Modifyaproblemintheproblemlist 100% None

Task10-ProblemList-Views Viewhistoricalprobleminformation 100% None

Task11-MedicationList-AddMedication Addamedicationtomedicationlist 100% None


Modifymedicationinthemedicationlist

100% None

Task13-DiagnosticImaging-AddOrder Createadiagnosticimagingorder 100% None

Task14-ChangeDiagnosticImagingOrder Modifyimagingorder 100% None

Task15-LabOrder-Add Createalaboratoryorder 90% Low

Task16-LabOrder-Change Modifyalaboratoryorder 100% None

Task17-Drug-DrugInteractionCheck Demonstratedrug-druginteractionwarning

80% Moderate

Task18-DrugAllergyInteractionCheck Demonstratedrug-allergyinteractionwarning

90% Low

Task19-ElectronicRx-NewRx Electronicallyprescribeanewdrug 90% Low

Task20-ElectronicRx-ChangeRx Modifyanelectronicprescription 80% Moderate

Task21-ElectronicRx-RefillRx Refillaprescriptionelectronically 100% None

Task22-ElectronicRx-Views Viewhistoricalprescriptiondata 100% None

Task23-ElectronicRx-CancelRx Cancelanelectronicprescription 100% None

Task24-CDS-EnableRule-ViewAttributes MakeadministrativechangetoCDSsystemtoenablerule;Viewextendedattributes

100% None

Task25-CCDAViewReconciledData GenerateandviewanupdatedCCDAdocument

100% None

54

AreasforImprovement

Thefollowingisapartiallistofpotentialareasforimprovement.Makingtheseandotherminor

enhancementswillimprovetheoveralluserexperienceoftheExscribesystemandincreasethe

effectiveness,efficiency,andsatisfactionforbothexperiencedandnoviceusers.

• IndicationofRequiredFields

o Afrequenterrorwascausedwhenparticipantsattemptedtosubmitaform

withintheinterfacethatcontainedrequiredfieldsthatwerenotcompleted.

Thisislikelybecausethesystemdidnotprovideaclearindicationofwhich

fieldsarerequired.Addingaconsistentvisualindicationofrequiredfields

wouldlikelyeliminatemanyoftheseerrorsobserved.

• Inconsistent“Add,“Search”andotherselectfunctions

o Manyerrorsoccurredwhenusersattemptedtoperforman“Add”or“Update”or

“Search”function.Thesystemprovidedseveraldifferentuserinteraction

patternsacrossseveraldifferentfunctionalareas.Providingaconsistentuser

interfaceanduserexperienceacrosstheentireapplicationmayhelptoreduce

usererrors.

• InconsistentUserInterfaceandUserExperience

o Severalerrorsoccurredwhentheusersattemptedtoperformanactionusinga

userexperiencepatternthatexistedinonefunctionalarea,butthispatterndid

notworkinthecurrentfunctionalarea.CreatingaUserExperienceandauser

interfacethatisconsistentacrossallfunctionalareaswouldhelptoimprovethe

overalluserexperienceofthesystem.

55

Appendices

AppendixA:RecruitingScreener1.Areyoumaleorfemale? 2.Haveyouparticipatedinafocusgrouporusabilitytestinthepast6months?3.Doyou,ordoesanyoneinyourhomeworkinmarketingresearch,usabilityresearch,and/orwebdesign?4.Doyou,ordoesanyoneinyourhome,haveacommercialorresearchinterestinanelectronichealthrecordsoftwareorconsultingcompany?5.Whichofthefollowingbestdescribesyourage?

20-29,30-39,40-49,50-59,60-69,70-79,80-89,90-99,100andolder.6.Whichofthefollowingbestdescribesyoureducationlevel?

• Nohighschooldiploma• Highschoolgraduate,diplomaortheequivalent• Somecollegecredit,nodegree• Tradetechnicalvocationaltraining• Associatedegree• Bachelor’sdegree• Master’sdegree• Doctoratedegree(e.g.,MD,DNP,DMD,PhD)

7.Doyourequireanyassistivetechnologiestouseacomputer?8.Pleasedescribeyourmedicalornursingcredentials9.Whatisyourcurrentjobtitle?10.Howlonghaveyouheldthisposition?(numberofyears):11.Whattypeoffacilitydoyouworkinandwhatisyourrolethere?12.Howaremedicalrecordshandledatyour(main)workplace?

_____AllPaper _____SomePaper/SomeElectronic ___AllElectronic13.HowmanyEHRsdoyouuseorhaveyouworkedwith?14.Howmanyyearshaveyouusedanelectronichealthrecord?15.HowmanyyearshaveyouusedtheExscribesystem?

56

16.Howmanyyearshaveyouusedcomputers?17Abouthowmanyhoursperweekdoyouspendusingacomputer?18.Whatcomputerplatform(s)doyouusuallyuse?19.Inthelastmonth,abouthowoftenhaveyouusedanelectronichealthrecord?

_____Didnotuselastmonth ___Everyday _____Afewtimesaweek.

57

AppendixB:InformedConsentFormTheUsabilityPeoplewouldliketothankyouforparticipatinginthisstudy.Thepurposeofthisstudyistoevaluateanelectronichealthrecordssystem.Ifyoudecidetoparticipate,youwillbeaskedtoperform several tasks using the prototype and give your feedback. The studywill last about 60minutes.AgreementI understand and agree that as a voluntary participant in the present study conducted by TheUsabilityPeople.Iamfreetowithdrawconsentordiscontinueparticipationatanytime.IunderstandandagreetoparticipateinthestudyconductedandrecordedbyTheUsabilityPeople.Iunderstandandconsenttotheuseandreleaseof thevideorecordingbyTheUsabilityPeople. Iunderstandthattheinformationandvideoisforresearchpurposesonlyandthatmynameandimagewill not be used for any purpose other than research. I relinquish any rights to the video andunderstand thevideorecordingmaybecopiedandusedbyTheUsabilityPeoplewithout furtherpermission.Iunderstandandagreethatthepurposeofthisstudyistomakesoftwareapplicationsmoreusefulandusableinthefuture.IunderstandandagreethatthedatacollectedfromthisstudymaybesharedoutsideofTheUsabilityPeople.Iunderstandandagreethatdataconfidentialityisassured,becauseonlyde-identifieddata–i.e.,identificationnumbersnotnames–willbeusedinanalysisandreportingoftheresults.I agree to immediately raise any concerns or areas of discomfortwith the study administrator. IunderstandthatIcanleaveatanytime.Pleasecheckoneofthefollowing:____YES,Ihavereadtheabovestatementandagreetobeaparticipant.____NO,Ichoosenottoparticipateinthisstudy.Signature:_____________________________________Date_____________________

58

AppendixC:ParticipantGuideORIENTATIONandINTRODUCTIONThankyouforparticipatinginthisstudy.Thesessiontodaywilllastapproximately60-90minutes.Duringthissession,youwilllookataspecialversionoftheExscribeEHR.Theproductyouwillbeusingtodaymaynotbeexactlylikethesoftwarethatyoumaybeusedto.Someofthedatamaynotmakesense,asitismerelyplaceholderdata.Thepurposeofthisstudyistogaugetheusabilityofthesoftware.Weareinterestedinhoweasy(orhowdifficult)thisversionistouse,andlearningaboutwhichofthefeaturesandbenefitswouldbemostusefultoyou.Wealsowanttoknowhowwecouldimproveit.Themoderatorwillguideyouthroughthetestingprocess.Pleaseusethesoftwareasyouwouldinarealclinicalsetting.Youwillbeaskedtosaveyourcommentsuntiltheendofataskortheendofthesessiontodiscussyourthoughtsfreely.Herearesomethingsyoushouldknowaboutyourparticipationinthissession

• TheModeratorwillguideyouthrougheachtask.• Pleasedonotworkahead.• Ifyounoticeanerror,justcanceloutandcontinuetothetask.Ifyoufeellostor

havedifficultycompletingthescenario,pleaseinformtheModerator.• TheTestingsessionwillbeaudiorecordedtheaudioforfurtherstudy.• Allinformationwillbekeptconfidential.Yournamewillnotbeassociatedwith

yourcommentsatanytime.Fictitiouspatientscenarioshavebeencreatedandpre-loadedinthesystemsoftware.TheModeratorwillaskyoutocompleteseveraltasksusingthesystem.Youwillbeaskedtoanswersomequestionsandtocompletesometasksonyourown.Trytocompletetasksasquicklyaspossible,withthefewestpossibleerrorsordeviations.Pleasedonotdoanythingmorethanasked.

59

DemographicInformationYourpatientCandiceTaylorhasarrivedfortheirscheduledappointment.TheyhavebeenreferredtoourorganizationbyanoutsideprimarycarefacilityTask1(a.5)DemographicsBeforebeginningtheirappointment,youneedtoverifyorentersomeofthedemographicinformationthatisstoredwithinthesystemPreferredLanguage ‘FrenchPatientRace ‘Asian’

PatientEthnicity ‘NotHispanicorLatino’

BirthSex MaleSexualOrientation ‘Heterosexual’

GenderIdentity ‘IdentifiesasMale'

Switchtotheeditviewandverifyand/orenterandsavethisinformationintotheEHR.

Verifythatthesechangeshavebeensaved.

60

Task2(a.5)Demographics–ChangesDuringyourconsultationwiththepatient,theyaskyoutomakethefollowingchangestotheirdemographicrecords:PreferredLanguage ‘EnglishPatientRace ‘Asian’

PatientEthnicity ‘NotHispanicorLatino’

DateofBirth 06/17/1950BirthSex MaleSexualOrientation ‘Unknown

GenderIdentity ‘IdentifiesasNon-ConformingGender

Switchtotheeditviewandverifyand/orenterandsavethisinformationintotheEHR.

Verifythatthechangeshavebeensaved.

61

Task3(b.2)ClinicalInformationReconciliationandIncorporationYourpatienthascometotheirappointmentwitha“ThumbDrive”withClinicalinformationfromanotherprovider.Forthepurposesoftesting,wehaveplacedaHealthExchangeDocumentontheWindowsDesktop.ReconcileandincorporatetheMedicalInfoincluding:Medications,DiagnosesandAllergiesthatarepresentintheCCDAfilewithpatient’sexistingrecords

Medication

• AddtheAccupriltotheFinalReconciledMedications.• AddtheKeflextotheFinalReconciledMedications.• AddtheFlexeriltotheFinalReconciledMedications.

Allergy• AddtheSulfaAllergytotheFinalReconciledAllergyList.

Problems• AddHypertensiontotheFinalReconciledProblemList.• AddHeartDiseasetotheFinalReconciledProblemList.• AddHypercholseteromiatotheFinalReconciledProblemList.• AddArthritistotheFinalReconciledProblemList.

Confirmthatthereconciledlisthasbeenincorporatedintothecurrentpatientrecord.VerballyreportanyClinicalDecisionSupportmessagesthatmayappear.

62

Task4(a.8)AddanallergyThepatientisreportingsomehivesaftertakingPenicillin.AddPenicillinasaMilddrugallergywithactivestatusandthesymptomofHives.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

Task5(a.8)ViewAllergyListandmakeachangeNoticethatthecurrentallergylistnowhastwoitems,ThePenicillinthatwasjustaddedandanallergytoSulfaDrugs.YounoticethatthereisnoinformationabouttheseverityandreactiontotheSulfadrug.ChangetheSulfaAllergysothatitisrecordedasanActiveMildDrugAllergywithareactionofRashVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

63

Task6(a.14)ImplantableDeviceListTheclienthashadanhistoricalsurgeryinwhichamedicaldevicewasimplanted.ThepatienthasprovidedyouwiththefollowingUniqueDeviceIdentifier(UDI):(01)10884521062856(11)141231(17)150707(10)A213B1(21)1234 (ThisUDInumbershouldbeinyour‘clipboard’andhasbeensavedasadesktopfile)PleaserecordthisMedicalInfointotheImplantableDevices(DeviceImplantHistory)list

Verifyand/orEnterandsavethisinformationintotheEHR.Verballystatethedevicedescription,identifiers,andotherattributes(brandetc.)ofthisdevice.Task7(a.14)ImplantableDeviceList-ChangeTheclientreportstheimplantabledevicewasremoved.ChangethedevicestatustoanInactivestatus.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved

64

Task8(a.6)ProblemList.Duringtheevaluationofthepatient’sleftelbow,youchoosetoaddanotheritemtotheirlistofMedicalConditions.EnteranOrthopaedictypediagnosisofOsteoarthritisVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

Task9(a6)ProblemList-ChangeUsethesystemtoupdatethelistofmedicalconditionsothattheMedicalConditionthatwasjustenteredhasaStatussettoNEW

Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

Task10(a6)ProblemList–ViewsPleaseviewtheactiveproblemlistandconfirmthatitnowindicates:HypertensionHeartDiseaseHypercholesterolemiaArthritisAndOsteoarthritisModifytheviewsothatthesystemdisplaysonlyResolveditems.Viewtheproblemlistandverifythatthehistoricalproblemlisthasbeendisplayed.

65

Task11(a.1)RecordmedicationYourpatientreportstakingTylenolforsomepaintheyhavebeenfeelingintheirleftelbow.Viewthepatient’scurrentmedslistandaddthefollowingmedication:Tylenol325mgtakenonce-a-dayevery4hours.Verifyand/orEnterandsavethisinformationintotheEHR.

Task12(a.1)ChangemedicationUponfurtherconversationwiththepatient,theyinformyouthattheyarenotinfacttakingTylenolbuthavebeentakingAleve.ChangethemedicationlistsothattheactualdrugtheyaretakingislistedasAleve(Naproxen)220mgcapsuleonceevery6to8hoursVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

66

Task13(a.3)DiagnosticImagingorderYouwanttofollowuponthepainthatyourpatienthasreportedintheirleftelbow.Usingthe“Superbill”functionsoftheEHR,Selectthepatient’sleftelbowandorderthefollowingprocedure:Orderanimagingstudy:73070ElbowLeft–2ViewsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

UponsavingtheimagingorderwithintheSuperbillfeature–theClinicalDecisionSupport(CDS)systemwilldisplayimportantinformation.VerballyidentifytheCDSitemthatappearsandoptionallyviewtheadditionalinformationthatisavailablethatsupplementstheinformationalmessage.

Task14(a.3)ChangeDiagnosticImagingorderOnsecondthought,youdecidethatyouwanttodoaslightlymorecomprehensiveimagingstudy.Changetheimagingorderthatwasjustenteredto:Orderanimagingstudy:73080ElbowLeft–3ViewsVerifyand/orEnterandsavethisinformationintotheEHR.

Uponsavingtheupdatedimagingorder-theClinicalDecisionSupport(CDS)systemwilldisplayimportantinformation.VerballyidentifytheCDSitemthatappearsandviewtheadditionalinformationthatisavailablethatsupplementstheinformationalmessage.

67

Task15(a.2)RecordaLaborderBecauseyourpatienthasseveraldiagnosesassociatedwiththeiroverallhealth,youdecidethatyouwouldalsoliketoordersomelabwork.Again,usingthe“SuperBill”“PlanItems”feature,completeaLabRequestformforthefollowinglabs:Selectthefollowinglabtests:

• Glucose2hrpp• Creatinine

Verifyand/orEnterandsavethislabrequestintotheEHR.Verifythatanychangeshavebeensaved.

Task16(a.2)ChangeLaborderUponreviewingthelaborderjustentered,yourealizeyouwouldalsoliketorunafullUrinalysisMakethenecessarychangessuchthataUrinalysishadbeenaddedtothelaborder.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

68

Task17(a.4)Drug-drug,drug-allergyinteractionchecksYourpatienthashypertension.YouwanttoprescribeadrugusingtheePrescribefeaturetohelptreatthem.PrescribeLisinopril10mg,onceadayfor30daysDuringtheCompleteRXReviewthesystemwilldisplayamoderatewarningmessageregardingthisnewmedication.ReviewthewarningdetailsbutdoNOTcontinuewiththeselection/prescriptionofthisdrug.RemovethisRXfromthelistof“Pending”drugsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

Task18(a.4)Drug-drug,drug-allergyinteractionThepatientpresentswithinfectedsoresontheirleftforearm.Addthefollowingmedication:PenicillinVpotassium125mg/5mLoralsolution3timesperdayfortheinfection.Thesystemwilldisplayawarningmessageregardingthisnewmedication.Reviewthewarningandcanceltheorderofthisdrug.Additionally,removethisRXfromthelistof“Pending”drugsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

69

Task19(b.3)e-Prescribing-NewRXYouneedtocreateaprescriptionforyourpatientforsomethingthattheyarenotallergicto.CreatenewprescriptionforDoxycycline100mgwithaSIG:Take1capsulebyoralrouteonetimeperdayfor5days.Includeaquantityof5pillsand0refills.Finalizeandreviewthecompleteprescriptionandthenelectronicallysend(Using“TestPharmacyat100FakeAddress”astheselectedpharmacy.)tothepharmacyTask20(b.3)e-Prescribing-ChangeprescriptionUponfurtherthoughtandreviewoftheprescriptionabove,youhavedecidedthata150mgcapsuleisbetterforthispatient.Youwillneedtomakeachangetotheelectronicprescriptionthatyouhavejustordered.ViewtheirupdatedPrescriptionlistandMakethenecessarychangessothat:Doxycycline150MGwithaSIG:Take1capsulebyoralrouteonetimeperdayfor5days.Includeaquantityof5pillsand0refills.isinsteadprescribed.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved

70

Task21(b.3)eRX-RefillprescriptionYourpatienthasrequestedthattheirpharmacyrefillsomeoftheirmedications.AccessAlloftheirprescriptionsandrefilltheDoxycyclineprescriptionforanadditional2refillsVerifyand/orEnterandsavethisinformationintotheEHR.VerifythattherefillhasbeensavedTask22(b.3)eRX–ViewprescriptioninformationYoualsowanttoexaminethispatient’shistoricalmedications.FiltertheviewofthemedicationslisttoseethefullhistoryofprescribedmedicationsVerballyidentifyandhistoricalmedicationsthataredisplayedTask23(b.3)eRxCancelprescriptionDiscontinueorcanceltheDoxycyclineprescription.VerifythattheDoxycyclinehasbeenremovedfromthepatientcurrentprescriptions.

71

Task24(a.4)CDSAdjustmentYouwanttomakeachangetothewarningmessagesand/orclinicalinformationsupportthatappearsinthesystem.AccesstheadministrativefunctionsforClinicalrulesandperformthefollowing:LocateandopenineditmodetheClinicalrulethathasbeencreatedfor“PQRS#39OsteoporosisforWomen“EnsurethatthisruleissettoActiveNavigatetoviewtheattributesofthisruleandverballystatethe:BibliographiclinkDeveloperFundingSourceVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.

72

Task25(b.2)CCDAViewreconcileddataThepatientisdonewiththeirappointment.Thepatientgoestothecheckoutdesk.Whilemakingtheirnextappointment,thepatientaskstohaveafullyup-to-dateContinuityofCareDocument(CCD)thatincludesthechangesmadeasaresultoftoday'svisit.YouneedtogenerateaCCDHealthExchangeDocumentforyourclient.Viewtheclient’sContinuityofCareDocumentForthepurposesofthistesting

• settheDocumentTypetoContinuityofCareDocument”• andselecttheoptiontoDisplaythedocumenttothescreen.

ViewtheCCDdocumentthatcanbesharedwiththepatient.

73

AppendixD:SystemUsabilityScaleQuestionnaire

74

AppendixE:ComputerSystemUsabilityQuestionnairePleaseprovideyourimpressionoftheusabilityofthesystembyansweringeachofthequestionsbelow:1.Overall,IamsatisfiedwithhoweasyitistousethissystemStrongly 1234567 NA StronglyDisagree Agree2.ItwassimpletousethissystemStrongly 1234567 NA StronglyDisagree Agree3.IcaneffectivelycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree4.IamabletocompletemyworkquicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree5.IamabletoefficientlycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree6.IfeelcomfortableusingthissystemStrongly 1234567 NA StronglyDisagree Agree7.ItwaseasytolearntousethissystemStrongly 1234567 NA StronglyDisagree Agree8.IbelieveIbecameproductivequicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree9.ThesystemgiveserrormessagesthatclearlytellmehowtofixproblemsStrongly 1234567 NA StronglyDisagree Agree10.WheneverImakeamistakeusingthesystem,IrecovereasilyandquicklyStrongly 1234567 NA StronglyDisagree Agree11.Theinformation(suchasonlinehelp,on-screenmessages,andotherdocumentation)providedwiththissystemisclearStrongly1 234567 NA StronglyDisagree Agree12.ItiseasytofindtheinformationIneeded

75

Strongly 1234567 NA StronglyDisagree Agree13.TheinformationprovidedforthesystemiseasytounderstandStrongly 1234567 NA StronglyDisagree Agree14.TheinformationiseffectiveinhelpingmecompletethetasksandscenariosStrongly 1234567 NA StronglyDisagree Agree15.TheorganizationofinformationonthesystemscreensisclearStrongly 1234567 NA StronglyDisagree Agree16.TheinterfaceofthissystemispleasantStrongly 1234567 NA StronglyDisagree Agree17.IlikeusingtheinterfaceofthissystemStrongly 1234567 NA StronglyDisagree Agree18.ThissystemhasallthefunctionsandcapabilitiesIexpectittohaveStrongly 1234567 NA StronglyDisagree Agree19.Overall,IamsatisfiedwiththissystemStrongly 1234567 NA StronglyDisagree Agree

76

AppendixF.ReferencesNISTIR7741-NISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronicHealthRecords,RobertM.Schumacher,UserCentric,Inc.,SvetlanaZ.Lowry,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Nov.2010.NISTIR7742-NISTCustomizedCommonIndustryFormatTemplateforElectronicHealthRecordUsabilityTesting,RobertM.Schumacher,UserCentric,Inc.,SvetlanaZ.Lowry,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Nov.2010.NISTIR7804-TechnicalEvaluation,Testing,andValidationoftheUsabilityofElectronicHealthRecords,SvetlanaZ.Lowry,MatthewT.Quinn,MalaRamaiah,RobertM.Schumacher,Emilys.Patterson,RobertNorth,,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Feb.2012.ONC-2015EditionCertificationCompanionGuideSafety-enhanceddesign-45CFR170.315(g)(3b)3/30/2016ONC-2015EditionCertification-TestProcedurefor§170.315(g)(3)SafetyEnhancedDesignCSUQ-Lewis,J.R.(1995).IBMComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationandInstructionsforUse.InternationalJournalofHuman-ComputerInteraction,7,57-78.SUSBrooke,J.(1996)SUS:a"quickanddirty"usabilityscale.InP.W.Jordan,B.Thomas,B.A.Weerdmeester&A.L.McClelland(eds.)UsabilityEvaluationinIndustry.London:TaylorandFrancis.

Documents

EHR Usability Test Report of Exscribe EHR (Version 6.0)