Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
© 2017 Epic Systems Corporation.
1
EHR Usability Test Report
for EpicCare Inpatient
EHR Suite Epic 2017
Report based on NISTIR 7742 Customized Common Industry Format Template for Electronic Health Record Usability Testing
Dates of Usability Study: January, 2017
Date of Report: March 28, 2017
Report Prepared by: Epic
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 2
Table of Contents
1 Executive Summary .........................................................................................................................................4
2 Introduction .......................................................................................................................................................5
3 Method.................................................................................................................................................................6
3.1 Participants................................................................................................................ 6
3.2 Study Design ........................................................................................................... 10
3.3 Tasks ....................................................................................................................... 11
3.4 Procedures............................................................................................................... 18
3.5 Test Location ........................................................................................................... 18
3.6 Test Environment .................................................................................................... 18
3.7 Test Forms and Tools ............................................................................................... 19
3.8 Participant Instructions ............................................................................................ 19
3.9 Usability Metrics...................................................................................................... 19
4 Results ............................................................................................................................................................... 21
4.1 §170.315(a)(9) Clinical Decision Support ................................................................... 21
4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation ........................ 23
4.3 §170.315(a)(1) Computerized Provider Order Entry - Medications............................. 25
4.4 §170.315(a)(2) Computerized Provider Order Entry - Laboratory .............................. 27
4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging ................. 29
4.6 §170.315(a)(5) Demographics.................................................................................... 31
4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks ..................................... 33
4.8 §170.315(b)(3) Electronic Prescribing......................................................................... 35
4.9 §170.315(a)(14) Implantable Device List .................................................................... 37
4.10 §170 315(a)(8) Medication Allergy List .................................................................... 39
4.11 §170 315(a)(7) Medication List ................................................................................ 41
4.12 §170.315(a)(6) Problem List ..................................................................................... 43
5 Results Conclusion....................................................................................................................................... 45
Appendices .......................................................................................................................................................... 46
Appendix 1 ................................................................................................................... 46
Appendix 2 ................................................................................................................... 51
Appendix 3 ................................................................................................................... 52
Appendix 4 ................................................................................................................... 53
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 3
Appendix 5 ................................................................................................................... 57
Appendix 6 ................................................................................................................... 58
Appendix 7 ................................................................................................................... 60
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 4
1 Executive Summary Epic staff conducted a usability study of the Epic 2017 version of the EpicCare Inpatient EHR Suite1 in January
2017 at multiple healthcare organizations. The purpose of this study was to evaluate the usability of the user
interface and provide quantitative analysis of the usability of EpicCare Inpatient. During the usability test, 72
healthcare providers used EpicCare Inpatient in simulated, representative tasks. Each task was analyzed for
risk using the methods detailed in section 3.3.
This study collected performance data on various tasks typically conducted by physicians and nurses. The tasks correspond to certification criteria identified in 45 CFR Part 170 Subpart C of the Health Information
Technology: 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic
Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications. For information about the
overall User-Centered Design (UCD) process at Epic, see the User Centered Design document as submitted
for 9.1.4.
Participants in the usability study had experience with a prior version of EpicCare Inpatient. The moderators
introduced the test and instructed participants to complete a series of tasks, given one at a time, using EpicCare
Inpatient, following the testing procedures outlined in section 3.4. After each task, the participants were asked
to complete an Ease of Task Completion rating for that task. The participants’ interactions with the screen,
facial expressions, and verbalizations were recorded electronically during the test and later analyzed to collect
time measurements and evaluate performance. The moderators did not assist the participants in completing
the tasks. All participant data was de-identified.
In accordance with the examples in the NIST 7742 Customized Common Industry Format Template for Electronic
Health Record Usability Testing, various recommended metrics were used to evaluate the usability of the
software. The following quantitative metrics were collected for each eligible participant:
Task completion
Time to complete each task
Number and type of unnecessary steps
Number and type of extra steps
Participant’s Ease of Task Completion ratings
System Usability Scale2 score
The System Usability Scale measures the subjective satisfaction with the system, based on the usability tasks
performed. The SUS scores for this study are 85.94 for inpatient physicians and 87.41 for inpatient nurses,
which represent an above average satisfaction rating (see footnote for score interpretation information). 3
In addition to the performance data, the following qualitative observations were made:
Post-test debrief comments
Major findings
Areas for improvement
1 ONC Health IT Certification (for Meaningful Use) information including pricing and limitations is available here: http://www.epic.com/Docs/MUCertification.pdf. 2 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). 3 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Generally, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 5
2 Introduction The usability study was conducted on the Epic 2017 version of the EpicCare Inpatient EHR Suite, which
facilitates inpatient care workflows and presents healthcare providers in an inpatient setting with integrated
medical information documented in a single patient chart. This usability study included realistic scenarios
typically encountered by clinicians who use the EpicCare Inpatient EHR Suite. Scenarios were constructed in
collaboration with clinicians to ensure clinical accuracy.
The purpose of the usability study was to validate and provide quantitative evidence of the usability of the
EpicCare Inpatient EHR Suite. Accordingly, the testing data measured efficiency, effectiveness, and user
satisfaction through a collection of metrics including time spent on each task, extra and unnecessary steps
taken per task, and Ease of Task Completion ratings from participants.
For the purpose of reporting findings from this usability study, commonly used terms are defined as follows:
Participant: A clinician who has experience with a prior version of the EpicCare EHR Inpatient Suite,
is eligible for participation in the usability study as determined by the Recruiting Screener (see
Appendix 1), and has completed the usability test
Scenario: A patient synopsis, given to participants to provide clinical context for tasks
Task: A verbal and written clinical workflow that is provided to all participants in the usability study
and has a predefined desired outcome
Subtask: The portion of a task relating to a specific criterion for which data is analyzed
Test: The compilation of tasks specific to studied criteria given in a single sitting to a participant
Path: A series of actions that can be taken in the EpicCare Inpatient EHR Suite to reach an outcome
The study was performed on the Epic 2017 version of the EpicCare Inpatient EHR Suite, which includes the
following products:
EpicCare Inpatient
ASAP
MyChart
Care Everywhere
Reporting tools such as Reporting Workbench and Radar
E-Prescribing Interfaces (Outgoing Medication Orders to Retail Pharmacies, Incoming Refill Requests
from Retail Pharmacies, Outgoing Medication Dispense History Query)
Vaccination Interfaces (Outgoing Vaccination Administration, Outgoing Vaccination History Query)
Incoming QRDA Documents Interface
FHIR Services
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 6
3 Method
3.1 Participants
A total of 36 inpatient physicians and 36 inpatient nurses participated in the study. Physicians were asked to
complete 15 tasks and nurses had 10 tasks. Participants were recruited by leaders at their organizations and
were not compensated by Epic for their time. The EpicCare Inpatient EHR Suite is designed to accommodate
physician and nurse specialists, therapists, and other specialized care providers, in addition to general
medicine providers, in an inpatient setting. The participants were actual users of a previous version of
EpicCare Inpatient. Participants were not directly connected to the development of EpicCare Inpatient, nor
were they employed by Epic.
Participants completed a recruiting screener that was used to gather demographic data and to verify eligibility
for participation in the study (see Appendix 1 for the Recruiting Screener used for this study). Recruited
participants represented a mix of backgrounds and demographic characteristics. Participants were assigned
an alphanumeric participant identifier at the time of the usability test to de-identify results.
Of the 46 physicians who were given the Recruiting Screener, 36 qualified to participate in the usability study.
Of the 58 nurses who were given the Recruiting Screener, 36 qualified to participate in the usability study. The
most common reason for ineligibility was that the participant did not provide patient care (see Appendix 1 for
a full list of eligibility criteria). Participants were scheduled for individual 30-minute testing sessions.
Demographic data4 is listed in Tables 1-2 and summarized in Appendix 3.
Table 1: Participant Demographics - Inpatient Physicians
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
1 A19 M Physician Doctorate(MD, DO,
PhD, DNP)
3 Surgery 50-59 Yes 26 20
2 A21 F Physician Doctorate (MD, DO,
PhD, DNP)
10 Obste trics and
Gynecology
30-39 Yes 5 20
3 A23 M Physician Doctorate (MD, DO,
PhD, DNP)
4 Pediatrics 30-39 Yes 4 15
4 A117 M Physician Doctorate (MD, DO,
PhD, DNP)
0.5 Internal Medicine 30-39 Yes 5 10
5 B05 M CEO,
CMIO, CIO,
e tc.
Doctorate (MD, DO,
PhD, DNP)
3.5 Pediatric Cardiac
Intensive Care
40-49 Yes 12 30
4 Note for readers referencing the “open data” CHPL: Experience in Field and Computer Use were originally gathered as ranges, w ith specific values
obtained later once “open data” CHPL specifications were re leased. Average values were applied to these ranges for participants with whom Epic was
not able to establish a second contact.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 7
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
6 B06 M Physician,
Director
Doctorate (MD, DO,
PhD, DNP)
4.5 Psychiatry 50-59 Yes 19 20
7 B10 M CMIO Doctorate (MD, DO,
PhD, DNP)
3 Family Medicine 40-49 Yes 10 25
8 B12 M Physician;
Associate
CMIO;
Information
Technology;
SVP, AVP, VP, e tc.
Doctorate (MD, DO,
PhD, DNP)
15 Internal Medicine 50-59 No 22 26
9 B101 F Physician Doctorate (MD, DO,
PhD, DNP)
6 Geriatrics 50-59 Yes 18 12
10 C05 M Physician Doctorate (MD, DO,
PhD, DNP)
9 Pulmonary, Critical
Care
30-39 Yes 6 30
11 C06 M Physician Doctorate (MD, DO,
PhD, DNP)
8 Pediatric Intensive Care 40-49 Yes 10 20
12 E04 F Physician Doctorate (MD, DO,
PhD, DNP)
5 Pediatrics 30-39 Yes 7 20
13 E10 M Physician Doctorate (MD, DO,
PhD, DNP)
8 Colorectal Surgery 30-39 Yes 8 18
14 E14 M Physician Doctorate (MD, DO,
PhD, DNP)
10 Family Medicine 40-49 Yes 13 30
15 E16 F Physician Doctorate (MD, DO,
PhD, DNP)
9 Infectious Diseases 30-39 Yes 3 18
16 E301 M Physician Doctorate (MD, DO,
PhD, DNP)
9 Physical Medicine and
Rehabilitation
30-39 Yes 5 20
17 E305 M Physician Doctorate (MD, DO,
PhD, DNP)
13 Hospitalist 50-59 Yes 12 25
18 E306 M Physician Doctorate (MD, DO,
PhD, DNP)
11 Internal Medicine,
Pediatrics
40-49 Yes 8 35
19 E307 F Physician Doctorate (MD, DO,
PhD, DNP)
8 Hospitalist 30-39 Yes 4 20
20 E317 M Physician Doctorate (MD, DO,
PhD, DNP)
14 Internal Medicine,
Palliative Care
50-59 No 27 25
21 E407 M Physician Doctorate (MD, DO,
PhD, DNP)
6 Internal Medicine 30-39 Yes 4 25
22 E501 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Family Physician,
Hospitalist
30-39 Yes 9 30
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 8
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
23 E504 M Physician Doctorate (MD, DO,
PhD, DNP)
9 Cardiology 30-39 Yes 5 30
24 F107 M Physician Doctorate (MD, DO,
PhD, DNP)
6 ICU 50-59 Yes 20 20
25 F110 M Physician Doctorate (MD, DO,
PhD, DNP)
1.5 Family Medicine,
Hospitalist
60-69 Yes 30 39
26 F111 M Physician Doctorate (MD, DO,
PhD, DNP)
5 Hospitalist 30-39 Yes 8 10
27 F114 F Physician Doctorate (MD, DO,
PhD, DNP)
8 Pediatric Internal
Medicine
30-39 Yes 8 16
28 G02 F Physician Doctorate (MD, DO,
PhD, DNP)
7 Hematology, Oncology 40-49 Yes 11 20
29 G04 F Physician Doctorate (MD, DO,
PhD, DNP)
6 Pediatric Hospitalist 30-39 Yes 5 20
30 G06 F Physician Doctorate (MD, DO,
PhD, DNP)
4.5 Pediatric Hospitalist 30-39 Yes 7 20
31 G10 M Physician Doctorate (MD, DO,
PhD, DNP)
4.5 Pediatrics 30-39 Yes 4.5 20
32 G11 M Physician Doctorate (MD, DO,
PhD, DNP)
5 Pediatric Physical
Medicine and
Rehabilitation
30-39 Yes 8 20
33 G12 M Physician Doctorate (MD, DO,
PhD, DNP)
2.5 Infectious Diseases 40-49 No 6 16
34 G111 M Director,
Information
Technology,
Physician
Doctorate (MD, DO,
PhD, DNP)
10 Internal Medicine 40-49 Yes 7 25
35 G112 F Physician Doctorate (MD, DO,
PhD, DNP)
9 Obste trics and
Gynecology
30-39 Yes 9 20
36 G117 M Physician Doctorate (MD, DO,
PhD, DNP)
5 Geriatrics 50-59 Yes 5 25
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 9
Table 2: Participant Demographics - Inpatient Nurses
ID Gender Role Education Leve l Epic User
Experience
(Years)
Specialty (as reported
by user)
Age First
Language
English?
Experience
in Fie ld
(Years)
Computer
Use
(Years)
1 A05 F Nurse Bache lor's Degree 4 Adult Acute and
Intermediate
Medicine
30-39 Yes 4 18
2 A33 M Nurse ,
Director
Bache lor's Degree 2 Psychiatric Nursing 60-69 Yes 11 41
3 A39 F Nurse Bache lor's Degree 1.5 OR 40-49 No 2 15
4 A47 F Nurse Associate Degree 1 Direct Observation
Unit
40-49 No 6 15
5 A108 F Nurse Doctorate (MD,
DO, PhD, DNP)
3 Vascular, Plastics,
Orthopedics, Trauma
60-69 Yes 30 30
6 A109 F Nurse Bache lor's Degree 4 ICU 40-49 Yes 20 25
7 A112 F Nurse Bache lor's Degree 6 OR 50-59 Yes 36 25
8 A115 F Nurse Bache lor's Degree 4 Neurology 60-69 Yes 14 20
9 A116 F Nurse Master's degree
(MSN, MS)
4.5 Diabetes 50-59 Yes 18 35
10 A118 M Nurse Bache lor's Degree 1 Critical Care 40-49 Yes 3 22
11 B09 F Nurse Associate Degree 4 PACU 50-59 Yes 24 20
12 B11 F Nurse Bache lor's Degree 4 PACU 60-69 Yes 39 20
13 B13 F Nurse Bache lor's Degree 8 Cardiology, ICU 50-59 Yes 30 24
14 B105 F Nurse Bache lor's Degree 5 Surgical, Urology 30-39 Yes 11 12
15 B106 F Nurse Associate Degree 4 Med Surg 30-39 Yes 4 20
16 B109 M Nurse Associate Degree 5 Gastrointestinal 20-29 No 1.2 5
17 B110 F Nurse Associate Degree 2 Med Surg 20-29 Yes 5 15
18 B111 F Nurse Associate Degree 4 OB, Labor and
Delivery
50-59 Yes 29 10
19 B112 F Nurse Associate Degree 5 Med Surg, Renal 40-49 Yes 10 20
20 B113 M Nurse Associate Degree 4 Critical Care 20-29 Yes 2.5 10
21 B114 M Clinical
Nurse
Manager
Bache lor's Degree 5 Critical Care 20-29 Yes 6 20
22 E303 F Nurse Bache lor's Degree 6 Cardiology 20-29 Yes 6 20
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 10
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported
by user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
23 E304 F Nurse Bache lor's Degree 10 PACU 40-49 Yes 23 10
24 E308 F Nurse Bache lor's Degree 8 Trauma 40-49 Yes 8 20
25 E309 F Nurse Bache lor's Degree 15 Post-Partum,
Newborn Nursery
50-59 Yes 35 30
26 E310 F Nurse Bache lor's Degree 10 Labor and Delivery 30-39 Yes 16 20
27 E311 F Nurse Bache lor's Degree 12 Surgery, Trauma 30-39 Yes 12 20
28 E312 M Nurse Bache lor's Degree 15 Not reported 60-69 Yes 28 15
29 E314 F Nurse Bache lor's Degree 5 Med Surg 50-59 No 14 10
30 E315 F Nurse Bache lor's Degree 10 Med Surg 40-49 No 20 30
31 E316 F Nurse Bache lor's Degree 2 MICU 30-39 Yes 2 20
32 E318 F Nurse Associate Degree 10 Labor and Delivery 40-49 Yes 15 20
33 E404 F Nurse Master's degree
(MSN, MS)
5 Not reported 30-39 Yes Not
reported
Not
reported
34 E507 F Nurse Bache lor's Degree 4 ICU 40-49 Yes 12 20
35 F101 F Nurse Bache lor's Degree 7 PACU 50-59 Yes 39 30
36 F105 F Nurse Bache lor's Degree 5 Rehab 20-29 Yes 2 20
3.2 Study Design
The objective of this study was to demonstrate areas where the application suite performed well – that is,
effectively, efficiently, and satisfactorily – and identify areas where improvements can be made.
Participants interacted with the Epic 2017 version of the EpicCare Inpatient EHR Suite. Each participant used
the system in a designated location, usually a conference or training room at the site where the participant is
employed. All participants were provided with the same instructions by the test moderator. The system was
evaluated for effectiveness, efficiency, and satisfaction as defined by metrics collected and analyzed for each
participant:
Task completion
Time to complete each task
Number and type of unnecessary steps
Number and type of extra steps
Participant’s Ease of Task Completion ratings
System Usability Scale score
For additional information on usability metrics, see section 3.9.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 11
3.3 Tasks
3.3.1 Task List
Tasks were constructed to be realistic and representative of typical activities a user would complete using the
EpicCare Inpatient EHR Suite. The tasks were prioritized and selected through a combination of the risk
analysis framework outlined in section 3.3.2 and the priorities outlined in NISTIR 7804-015.
NISTIR 7804-01 is an industry standard that provides scenarios and guidelines for usability testing of
Electronic Health Records. Tasks for the usability study were constructed to emphasize priorities articulated
in NISTIR 7804-1 (e.g. identification of information, consistency of information, and integrity of information),
and incorporated the NIST test scenarios when applicable.
Tasks are split into subtasks that are measurable components related to criteria supplied by the ONC. See
Appendix 4 for full task wording.
Physician Tasks/Subtasks
Scenario 1: Gertrude is a 55-year-old female who has been admitted to your unit from the ED after
experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition, abrasions,
and a possible concussion.
Task 1: Reconcile a problem from a primary care provider at an outside organization .
§170.315(b)(2) Clinical information reconciliation and incorporation
Task 2: Add a problem to the problem list for a patient admitted from the ED.
§170.315(a)(6) Problem list
Task 3: Add a medication allergy to a patient’s allergy list and assess any interactions.
§170.315(a)(4) Drug-drug, drug-allergy interaction checks
§170.315(a)(8) Medication allergy list
Task 4: Modify an order for a diagnostic imaging procedure.
§170.315(a)(3) Computerized provider order entry – Diagnostic imaging
Scenario 2: Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently
admitted to your unit.
Task 5: Review the patient’s problem list.
§170.315(a)(6) Problem list
Task 6: Modify the details of a patient-controlled analgesic medication order.
§170.315(a)(1) Computerized provider order entry – Medications
Task 7: Modify the details of an inpatient laboratory order.
5 NISTIR 7804-1 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability Guidelines for
Standardization
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 12
§170.315(a)(2) Computerized provider order entry – Laboratory
Task 8: Respond to the system’s clinical decision support and place a vaccine order based on a quality
improvement initiative.
§170.315(a)(8) Clinical decision support
Task 9: Order an outpatient lab to monitor a discharge medication.
§170.315(a)(2) Computerized provider order entry – Laboratory
Task 10: Electronically prescribe a discharge medication and assess any interactions.
§170.315(a)(1) Computerized provider order entry – Medications
§170.315(b)(3) Electronic prescribing
§170.315(a)(4) Drug-drug, drug-allergy interaction checks
Scenario 3: Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and
bradycardia. He came to the ED complaining of weakness in his right side and was admitted to the ICU for a
CVA.
Task 11: Modify a medication allergy on a patient’s allergy list.
§170.315(a)(8) Medication allergy list
Task 12: Place a medication order suggested by the system’s clinical decision support and assess any
interactions.
§170.315(a)(9) Clinical decision support
§170.315(a)(4) Drug-drug, drug-allergy interaction checks
§170.315(a)(1) Computerized provider order entry – Medications
Task 13: Order an ultrasound to diagnose symptoms.
§170.315(a)(3) Computerized provider order entry – Diagnostic imaging
Task 14: Resolve a problem on the patient’s problem list.
§170.315(a)(6) Problem list
Task 15: Electronically prescribe a discharge medication.
§170.315(a)(1) Computerized provider order entry – Medications
§170.315(b)(3) Electronic prescribing
Nurse Tasks/Subtasks
Scenario 1: Walter is a 79-year-old male with a complex medical history that includes CHF, osteoporosis,
dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a wound infection.
Task 1: Document the patient’s preferred language.
§170.315(a)(5) Demographics
Task 2: Reconcile allergy information from an outside urgent care facility.
§170.315(b)(9) Clinical information reconciliation and incorporation
Task 3: Review a complex medication list based on information provided by the patient .
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 13
§170.315(a)(6) Medication list
Task 4: Add an implantable device to the patient’s chart using information provided by the patient.
§170.315(a)(14) Implantable device list
Task 5: Determine if the patient has any active devices on the implantable device list that may be contributing
to symptoms.
§170.315(a)(14) Implantable device list
Task 6: Indicate that the patient has declined the intervention suggested by the system’s clinical decision
support.
§170.315(a)(8) Clinical decision support
Scenario 2: Robin is a 50-year-old who has been admitted for diabetic ketoacidosis.
Task 7: Update gender identity as reported by the patient.
§170.315(a)(5) Demographics
Task 8: Reconcile allergy information from an outside facility.
§170.315(b)(9) Clinical information reconciliation and incorporation
Task 9: Add a patient-reported medication to the medication list.
§170.314(a)(7) Medication list
Task 10: Update the information in a patient’s implantable device list.
§170.315(a)(14) Implantable device list
Task selection was based on criticality of function and the risk analysis described in section s 3.3.2 and 3.3.3
below.
3.3.2 Risk Analysis Framework
Risk assessment for each task involves assessing the Likelihood of Risk Occurrence and the Significance of
Risk Materialization (referred to as Likelihood and Significance, respectively). Overall risk for each task is then
categorized as High, Moderate, Low, or Negligible.
Likelihood is determined by a combination of two factors: Frequency of Workflow and Possibility of
Alternative Outcome. Frequency of Workflow reflects the general prevalence of a specific or closely analogous
workflow. Possibility of Alternative Outcome is an assessment of the likelihood that a variation might occur
during a specific or closely analogous workflow. The way in which Frequency of Workflow and Possibility of
Alternative Outcome contribute to Likelihood is outlined in Appendix 7.
Significance is the measurement of the impact of possible outcomes that result from a variation from an
expected task workflow. Impact of possible outcomes of each task was determined in consult with clinicians
and other subject matter experts.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 14
Figure 1 lists representative factors used to evaluate Likelihood and Significance. Human and system factors
that affect the evaluation of each component were discerned from conceptual and historical analysis and
include, but are not limited to, those that appear in Figure 1. 6
Fig. 1: Factors which Influence Risk
Likelihood Significance
↑ Factors increasing Frequency of Workflow
Task involves a common sequence of events
↑ Factors increasing Possibility of Alternative Outcome
Clinician must make a decision that requires an increased cognitive load (i.e. the clinician must find information in multiple locations and must synthesize or remember high volumes of information to complete the workflow)
Clinician is limited in ability to recover from an issue in documentation or action
Workflow completes an event or otherwise closes an instance of care to any further documentation
High intrinsic complexity of information or information management
Clinician is likely to experience an interruption during the workflow
↑ Factors increasing Significance
Clinician is likely to take inappropriate action after the issue occurs
Issue affects mostly high-acuity patients, high-risk medication, or urgent/critical workflows
Affected data directly informs significant clinical decisions
↓ Factors decreasing Frequency of Workflow
Task involves an unusual sequence of events
↓ Factors decreasing Possibility of Alternative Outcome
Clinician does not make decisions in the course of the workflow or has robust decision-making support within the workflow
Clinician finds information in one location, or the workflow requires low degree of information synthesis or memory
Clinician has ample opportunity to recover from an issue in documentation or action
Task or workflow occurs in the midst of an event; documentation can easily be edited or added
Low intrinsic complexity of information or information management
Clinician is unlikely to experience an interruption during the workflow
↓ Factors decreasing Significance
Clinician is likely to take appropriate action regardless of the issue
Issue affects mostly low-acuity patients, low-risk medications, or non-urgent/non-critical workflows
Affected data does not inform significant clinical decisions
6 See Beasley, J. W., Wetterneck, R. B., Temte, J., Lapin, J., Smith, P., Rivera-Rodriguez, J., & Karsh, B. (2011). Information Chaos in Primary Care: Implications for Physician Performance and Patient Safety. J Am Board Fam Med. , 24(6), 745-751. doi:10.3122/jabfm.2011.06.100255; Carayon, P. Sociotechnical systems approach to healthcare quality and patient safety. Work, 4(1). doi:10.3233/WOR-2012-0091-38Carayon; Holden, R. J., Carayon, P., Gurses, A. P., Hoonakker, P., Hundt, A. S., Ozok, A., & Rivera-Rodriquez, A. (2013). SEIPS 2.0: A human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 53(11). doi:10.1080/00140139.2013.838643; Meeks, D. W., Takian, A., Sittig, D. F., Singh, H., & Barber, N. (2014). Exploring the sociotechnical intersection of patient safety and electronic health record implementation. Journal of the American Medical Informatics Association, 21, 2834. doi:10.1136/amiajnl-2013-001762. Epub 2013 Sep 19; O'Hara, R.,
& Et al. (2014). A qualitative study of decision-making and safety in ambulance service transitions. Health Services and Delivery Research, 2(56). doi:10.3310/hsdr02560; Vincent, C., Taylor-Adams, S., & Stanhope, N. (1995). Framework for Analysing Risk and Safety in Clinical Medicine. BMJ: British Medical Journal, 316(7138), 1154-1157; Wogalter, M. S., & Laughery, K. R. (1996). WARNING! Sign and Label Effectiveness. Current Directions in Psychological Science. doi:10.1111_1467-8721.ep10772712
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 15
Figure 2 illustrates how Likelihood and Significance contribute to overall risk. Significance is weighted more
heavily than Likelihood when completing the overall risk assessment for each task. The relationship between
Significance and Likelihood was created in consult with literature on risk evaluation.7
Fig. 2: Overall Risk Assessment
Significance
Negligible Low Moderate High
Like
lih
oo
d
High Negligible Risk Moderate Risk Moderate Risk High Risk
Moderate Negligible Risk Moderate Risk Moderate Risk High Risk
Low Negligible Risk Low Risk Moderate Risk Moderate Risk
Negligible Negligible Risk Low Risk Moderate Risk Moderate Risk
3.3.3 Risk-Based Task Selection
Moderate to high-risk workflows were selected for physician and nurse tasks based on the framework
presented above. Risk analysis of inpatient physician and nurse tasks is summarized in Tables 3 and 4,
respectively.
Table 3: Risk Analysis of Physician Tasks
Physician Task
Criteria Likelihood Significance Risk
1 Reconcile a problem from a primary care provider at an outside organization.
Clinical information reconciliation and incorporation High Moderate Moderate
2 Add a problem to the problem list for a patient admitted from the ED.
Problem List Moderate Moderate Moderate
3 Add a medication allergy to a patient’s allergy list and assess any interactions.
Drug-drug, drug-allergy interaction checks; Medication allergy list
Moderate High High
4 Modify an order for an inpatient diagnostic imaging procedure.
Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate
7 NASA. (1994). Systems Engineering (EIA/IS-632). Electronic Industries Association (EIA); Ben-Asher, J. Z. (2004). Systems engineering aspects in theatre missile defense? Design principles and a case study. Systems Engineering. doi:10.1002/sys.10058; Ben-Asher, J. Z. (2008). Development Program Risk Assessment Based on Utility Theory. Risk Management, 10(4), 285-299. doi:10.1057/rm.2008.9; Ben-
Asher, J. Z., Zack, J., & Prinz, M. (2000). Development Program Risk Management. AIAA Progress in Aeronautics and Astronautics, 192, 341-351.; Blanchard, B. S., & Fabrycky, W. J. (1981). Systems engineering and analysis. Englewood Cliffs, NJ: Prentice-Hall.; Robertson, T. C. (Ed.). (2000). Systems Engineering Handbook (2000 ed.). INCOSE; Tummala, V. M., & Mak, C. L. (2001). A risk management model for improving operation and maintenance activities in electricity transmission networks. Journal of The Operational Research Society, 52, 125-134. doi:10.1057/palgrave.jors.2601044.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 16
Physician Task
Criteria Likelihood Significance Risk
5 Review the patient’s problem list.
Problem list Moderate Moderate Moderate
6 Modify the details of a patient-controlled analgesic medication order.
Computerized provider order entry - Medications Moderate High High
7 Modify the details of an inpatient laboratory order.
Computerized provider order entry - Laboratory Moderate Low Moderate
8 Respond to the system’s clinical decision support and place a vaccine order based on a quality improvement initiative.
Clinical decision support Moderate Low Moderate
9 Order an outpatient lab to monitor a discharge medication.
Computerized provider order entry – Laboratory Moderate High High
10 Electronically prescribe a discharge medication and assess any interactions.
Computerized provider order entry - Medications; Drug-drug, drug-allergy interaction checks; Electronic prescribing
Moderate High High
11 Modify a medication allergy on a patient’s allergy list.
Medication allergy list Moderate Moderate Moderate
12 Place a medication order suggested by the system’s clinical decision support and assess any interactions.
Clinical decision support; Computerized provider order entry – Medications; Drug-drug, drug-allergy interaction checks
Low Moderate Moderate
13 Order an ultrasound to diagnose symptoms.
Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate
14 Resolve a problem on the patient’s problem list.
Problem list Moderate Low Moderate
15 Electronically prescribe a discharge medication.
Computerized provider order entry - Medications; Electronic prescribing
Moderate High High
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 17
Table 4: Risk Analysis of Nurse Tasks
Nurse Task
Criteria Likelihood Significance Risk
1 Document the patient’s preferred language.
Demographics Moderate Low Moderate
2 Reconcile allergy information from an outside urgent care facility.
Clinical information reconciliation Moderate Moderate Moderate
3 Review a complex medication list based on information provided by the patient.
Medication list Moderate Moderate Moderate
4 Add an implantable device to the patient’s chart using information provided by the patient.
Implantable device l ist Moderate Moderate Moderate
5 Determine if the patient has any active devices on the implantable device list that may be contributing to symptoms.
Implantable device l ist Low Low Low
6 Indicate that the patient has declined the intervention suggested by the system’s clinical decision support.
Clinical decision support Moderate Low Moderate
7 Update gender identity as reported by the patient.
Demographics Moderate Moderate Moderate
8 Reconcile allergy information from an outside facility.
Clinical information reconciliation Moderate Moderate Moderate
9 Add a patient reported medication to the medication list.
Medication list Moderate Moderate Moderate
10 Update the information in a patient’s implantable device list.
Implantable device l ist Moderate Low Moderate
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 18
3.4 Procedures
Moderators were Epic employees with experience in usability testing who underwent training specific to the
type of usability testing being conducted.
Participants arrived at their organization’s testing location where they were greeted by a moderator and
oriented to the testing computer and equipment. Participants were then assigned an alphanumeric participant
identifier in order to de-identify results. Each participant was asked for verbal consent to record the testing
session.
To prepare the participant for testing, the moderator outlined the format of the test and gave general
instructions. The moderator then began recording the session using screen capture, a microphone, and a
portable camera. Before starting the tasks, the moderator showed the participant a brief instructional recording
representative of the training typically given to users before the implementation of new functionality. Pieces
of functionality were included in the instructional recordings when research indicated that not all users were
familiar with the testing setup used, due to organizational variations in configuration. (Epic provides
organizations with a high degree of latitude in setting up the software to meet specific organizational needs.)
Physicians viewed an instructional recording on updated screen layout and clinical information reconciliation.
Nurses viewed an instructional recording on updated screen layout, clinical information reconciliation, and
implantable device list. At this point, the participant was given an opportunity to ask any questions or express
any concerns. The moderator continued to administer general instruction and tasks during the session.
Participants were instructed to perform tasks:
At their normal pace
Without assistance; moderators were allowed to give immaterial guidance and clarification on tasks,
but not instructions on use of the software
Withholding comments until the test was completed
Before each task, the moderator gave participants a written copy of the task and oral instruction. Task timing
began and ended when the participant reached particular predetermined points in the task.
After the participants finished the tasks, they participated in individual debrief sessions. During these sessions,
the moderators solicited feedback from participants on any areas where the participants had ex tra steps,
unnecessary steps, or incomplete tasks or subtasks. The purpose of this session was to gain additional
information about the possible causes of the unnecessary steps, extra steps, or incomplete tasks or subtasks.
3.5 Test Location
Testing was conducted at healthcare organizations where participants were employed. Participants’
organizations provided testing rooms, typically small conference or training rooms. To ensure that the
surroundings were comfortable for participants, noise levels were kept to a minimum and the ambient
temperature kept within a normal range. See Appendix 2 for a table summarizing the testing dates and
locations.
3.6 Test Environment
The EpicCare Inpatient EHR Suite is typically used in an inpatient facility. The testing was conducted at a
variety of participating organizations, in rooms made available for this purpose. Testing workstations were
either a Lenovo T430, Intel Core i5-3320M processor (2.60GHz) with 8 GB RAM or a Lenovo T440P, Intel Core
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 19
i5-4300M processor (2.60GHz) with 16 GB RAM. Workstations of both types used Windows 8 Enterprise x64,
Microsoft Internet Explorer 11 and a 23-inch flat panel display in landscape orientation, 1600 by 900 pixel
resolution, and set to thousands of colors.
The application was locally installed and used an InterSystems Caché 2013.1 database server on a loopback
connection.
The participants used a mouse and keyboard when interacting with the EpicCare Inpatient EHR Suite. The
overall system performance was comparable to what users would experience in a field implementation.
3.7 Test Forms and Tools
During the usability test, the following documents were used:
Recruiting Screener
Moderator Guide
Participant Packet
The participant’s interaction with the EpicCare Inpatient EHR Suite was captured and recorded digitally with
screen capture software running on the test workstation. Each participant’s facial expressions were recorded,
along with onscreen actions and verbal comments. Recordings were saved and used for further analysis.
3.8 Participant Instructions
The moderator read general introductory statements and instructions aloud to the participant before
administering the test. See Appendix 5 for the Sample Participant Orientation script.
The participant was then asked to complete a number of tasks that were read aloud by the moderator and
provided on paper to the participant for reference.
3.9 Usability Metrics
According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records ,
EHRs should support a process that provides a high level of usability for all users. Th e goal is for users to
interact with the system effectively, efficiently, and with high satisfaction. As such, metrics for these measures
were captured during the usability testing.
The goals of the test were to assess:
1. Efficiency of the EpicCare Inpatient EHR Suite by measuring the average task time and extra steps
2. Effectiveness of the EpicCare Inpatient EHR Suite by measuring task completion rates and unnecessary
steps
3. Satisfaction with the EpicCare Inpatient EHR Suite by measuring Ease of Task Completion ratings
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 20
Data Scoring
Table 5 details how metrics measuring efficiency, effectiveness, and satisfaction were scored.
Table 5: Usability Scoring Metrics
Measures Rationale and Scoring
Efficiency:
Average Task Time
Each task is timed from a predetermined starting point until the participant reaches the predetermined end point for the task or verbalizes completion.
Only task times for tasks that were successfully completed are included in the average task time analysis and standard deviation (reported in parentheses in the table below).
Efficiency:
Average Extra Steps per Task
The number of steps in a participant’s path through the application is recorded and compared to the number of steps in the closest acceptable path. An extra step is recorded if the participant performs a step that is not included in the defined path but is also not counterproductive to completing the task.
The total number of steps taken by a participant is counted and the difference between the steps in their path and the closest defined path is calculated. The average of the differences for the participants is calculated. Only extra steps for tasks that were successfully completed are included in the average extra steps per task analysis and standard deviation.
Effectiveness:
Binary Task Completion Rate
A task is considered a success if the participant achieves the defined task outcome without assistance.
The total number of successes was calculated for each task and then divided by the total number of times that the task was attempted. The results are presented as a percentage.
The task failure percentage can be calculated by subtracting the binary task completion rate from 100.
Effectiveness:
Partial Task Completion Rate
A task is considered 100 percent completed if the participant achieves the defined task outcome without assistance. For participants unable to successfully complete a task, the number of steps completed are counted and divided by the number of steps in the closest defined path to calculate the percentage of the task the participant completed. These results are, in turn, added together and divided by the number of participants who attempted the task to obtain the average partial task completion rate.
Effectiveness:
Average Unnecessary Steps per Task
Unnecessary steps are recorded each time a participant performs an action in the system that is not his intended action. Examples of unnecessary steps include typing mistakes and errant clicks that do not contribute to the completion of the task. The total number of unnecessary steps is calculated and divided by the number of participants to obtain the average number of unnecessary steps committed per participant.
Only unnecessary steps for tasks that were successfully completed are included in the average unnecessary steps per task analysis and standard deviation.
Satisfaction:
Ease of Task Completion Rating
The participant’s subjective impression of the ease of use of the application is recorded for each task. After each task was completed, the moderator asked the participant to rate the task on a 5-point Likert scale: 1 (Very Difficult), 2 (Somewhat Difficult), 3 (Neither Difficult nor Easy), 4 (Somewhat Easy), 5 (Very Easy).
These values are averaged across participants for each task with the calculated standard deviation reported in parentheses in the table in the Data Analysis and Reporting section for each criterion.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 21
4 Results The usability testing results for the EpicCare Inpatient EHR Suite are detailed below (see Tables 6-17). Results
are organized and analyzed by criteria. The results were calculated according to the methods specified in section
3.9 Usability Metrics.
4.1 §170.315(a)(9) Clinical Decision Support
4.1.1 Data Analysis and Reporting
Table 6: Clinical Decision Support Physician and Nurse Task Results
N=Number of participants
4.1.2 Discussion of the Findings
Clinical decision support testing with inpatient physicians covered two tasks:
Placing a vaccine order based on a quality improvement initiative (Task 8)
Placing a medication order based on an initiative and assessing any interactions (Task 12)
Testing with inpatient nurses covered one task:
Indicating that the patient declines intervention (Task 6)
Efficiency
Nurses completed Task 6 in 8.26 seconds and did so without taking any extra steps. Physicians completed the
tasks in an average of 5.66 seconds for Task 8 and 23.63 seconds for Task 12.
Clinical Decision Support
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Placing a vaccine order based on a
quality improvement initiative
35 5.66 (3.16) 0.20 (0.45) 100 (0) 100 (0) 0 (0) 4.86 (0.43)
Placing a medication order based on
an initiative and assessing any interactions
35 23.63
(16.38)
0.34 (0.59) 100 (0) 100 (0) 0 (0) 4.09 (1.07)
Nurse Subtasks
Indicating that the patient declines intervention
35 8.26 (3.62) 0 (0) 100 (0) 100 (0) 0 (0) 4.51 (0.56)
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 22
Effectiveness
The binary task completion rate for physicians is 100 percent for both Tasks 8 and 12. The binary task
completion rate for nurses is 100 percent for Task 6. Both doctors and nurses completed the tasks with no
unnecessary steps.
Satisfaction
The average satisfaction ratings for all tasks are between Somewhat Easy an d Very Easy. The average task ease
ratings for physician Tasks 8 and 12 are 4.86 and 4.09, respectively. The average task ease ratings for nurse
Task 6 is 4.51.
Major Findings
Physicians and nurses completed clinical decision support tasks consistently across various scenarios and
reported that the tasks were between Somewhat Easy and Very Easy. Metrics between physicians and nurses
are similar, suggesting a consistent experience across user groups.
In Tasks 8 and 12, the extra step physicians most frequently took was attempting to select an option that was
already selected by default. This action had no impact on patient care, but increased the average task time
slightly and accounted for the majority of extra steps taken in clinical decision support tasks.
Areas for Improvement
Overall, both physicians and nurses were able to successfully assess the information provided by the system's
clinical decision support and appropriately respond in scenarios where the system suggested a course of
treatment. Based on observation and analysis, the majority of extra steps for clinical decision support tasks
occurred when physicians were uncertain whether a choice was selected by default or needed to be made
manually. Though the outcome of these extra steps taken had no clinical impact, making default selections
more clear by improving the appearance of selected options could further improve clinician efficiency.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 23
4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation
4.2.1 Data Analysis and Reporting
Table 7: Clinical Information Reconciliation and Incorporation Physician and Nurse Task Results
4.2.2 Discussion of the Findings
Clinical information reconciliation and incorporation testing with inpatient physicians covered one task:
Reconciling a problem from a primary care provider at an outside organization (Task 1)
Testing with inpatient nurses covered two tasks:
Reconciling allergy information from an outside urgent care facility (Task 2)
Reconciling allergy information from a primary care provider at an outside facility (Task 8)
Efficiency
Thirty-five out of 36 nurses who completed Task 2 did so without any extra steps. Nurses completed Task 8
with a negligible number of extra steps. (Negligible is hereafter defined as within one confidence interval of
zero.) Physicians completed Task 1 with a negligible number of extra steps.
Effectiveness
The binary task completion rate for all clinical information reconciliation tasks is 100 percent. Both nurses
and physicians completed the clinical information reconciliation tasks without taking any unnecessary steps.
Satisfaction
The majority of participants rated the clinical information reconciliation tasks as Very Easy. For physicians,
the average satisfaction score for Task 1 is 4.80. For nurses, the average satisfaction scores for Tasks 2 and 8
are 4.75 and 4.86, respectively.
Clinical Information Reconciliation and
Incorporation
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Reconciling a problem from an outside organization
35 13.26 (14.83)
0.06 (0.24) 100 (0) 100 (0) 0 (0) 4.80 (0.41)
Nurse Subtasks
Reconciling an allergy from an outside urgent care facility
36 14.33 (17.85)
0.19 (0.44) 100 (0) 100 (0) 0 (0) 4.75 (0.55)
Reconciling an allergy from an outside facility
36 9.83 (5.18) 0.06 (0.24) 100 (0) 100 (0) 0 (0) 4.86 (0.42)
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 24
Major Findings
Between the two user groups, 100 percent of participants successfully completed the full clinical information
reconciliation workflow with an average satisfaction rating of Very Easy. Thirty-three out of 35 physicians
and 33 out of 36 nurses did not take any extra steps. Extra steps taken were to look in other areas of the
patient chart but all participants were able to complete the workflow.
Areas for Improvement
A few participants looked through other areas of the chart to reconcile outside information before pursuing
the task workflow, which accounts for the longer task times. Even with these extra steps, all users were able
to complete the full reconciliation workflow with high task satisfaction ratings. This observation highlights
opportunities to improve the integration and visibility of the clinical information reconciliation functionality:
Optimize references to reconciling outside information alongside internally documented information
Explore the use of inline links to incorporate outside information to provide more direct paths to task
completion
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 25
4.3 §170.315(a)(1) Computerized Provider Order Entry – Medications
4.3.1 Data Analysis and Reporting
Table 8: Computerized Provider Order Entry – Medications Physician Task Results
Computerized Provider Order Entry - Medications
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean
(seconds) (SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)
5 = very easy
Physician Subtasks
Modifying the details of a PCA medication order
36 18.47 (17.61)
0 (0) 100 (0) 100 (0) 0 (0) 4.61 (0.73)
Ordering a discharge medication and assessing any interactions
34 36.47 (24.84)
0.26 (0.51) 100 (0) 100 (0) 0.06 (.24) 4.03 (1.00)
Placing a medication order based on an initiative and assessing any
interactions
35 23.63 (16.38)
0.14 (0.38) 100 (0) 100 (0) 0.03 (0.16) 4.09 (1.07)
Ordering a discharge medication 35 27.37 (13.92)
0.14 (0.38) 100 (0) 100 (0) 0 (0) 4.43 (0.95)
4.3.2 Discussion of the Findings
Computerized provider order entry of medications testing with inpatient physicians covered four tasks:
Modifying the details of a PCA medication order (Task 6)
Ordering a discharge medication and assessing any interactions (Task 10: Subtask A)
Placing a medication order based on an initiative and assessing any interactions (Task 12: Subtask B)
Ordering a discharge medication (Task 15: Subtask A)
Efficiency
For discharge medications, the average time taken to complete Task 10—which had a drug interaction for the
participant to assess—was 36.47 seconds. The average time for Task 15—which did not have a drug
interaction—was 27.37 seconds. The average times for Tasks 6 and 12, which involved placing or modifying
an inpatient order, are more consistent. The average time for these tasks ranged from 18.47 seconds to 23.63
seconds.
Effectiveness
All medication ordering tasks had a binary completion rate of 100 percent. Participants took no unnecessary
steps for Tasks 6 and 12, which had participants modify or place an inpatient order. Participants who
completed Tasks 10 and 15 by placing discharge medication orders did not take any unnecessary steps for
Task 15 and took a negligible number of unnecessary steps to complete Task 10.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 26
Satisfaction
The average satisfaction ratings for all tasks are between Somewhat Easy and Very Easy. The average task ease
ratings for ordering a medication and assessing interactions in Tasks 10 and 12 are 4.03 and 4.09, respectively.
The average task ease ratings while ordering a medication without any interactions, Tasks 6 and 15, are higher
at 4.61 and 4.43, respectively.
Major Findings
All participants were able to successfully complete the medication ordering tasks, suggesting a high degree of
effectiveness in these workflows. The differences in other metrics between tasks with and without drug
interaction components suggest that interactions can affect provider effectiveness and perceived ease of use in
ordering tasks. For further analysis of the drug interaction tasks, see section 4.7 §170.315(a)(4).
Areas for Improvement
The medication ordering task presented participants with user interface components updated in Epic 2017.
Although all participants successfully completed all tasks, some extra steps were observed that may be related
to adjusting to the new UI components, suggesting that changes could be made to facilitate this transition to
the updated interface. This may be an appropriate area for further study, particularly:
Exploring potential options to orient users to new UI
Improving continuity of provider navigation in response to alternative courses of clinical decision -
making
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 27
4.4 §170.315(a)(2) Computerized Provider Order Entry – Laboratory
4.4.1 Data Analysis and Reporting
Table 9: Computerized Provider Order Entry – Laboratory Physician Task Results
Computerized Provider Order Entry - Laboratory
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean
(seconds) (SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)
5 = very easy
Physician Subtasks
Modifying the details of an inpatient lab order
36 10.14 (3.59)
0.03 (0.17) 100 (0) 100 (0) 0.03 (0.17) 4.83 (0.45)
Ordering an outpatient lab to monitor a discharge medication
36 9.28 (6.01) 0.11 (0.33) 100 (0) 100 (0) 0.03 (0.17) 4.64 (0.64)
4.4.2 Discussion of the Findings
Computerized provider order entry of laboratory tests with inpatient physicians covered two tasks:
Modifying the details of an inpatient lab order (Task 7)
Ordering an outpatient lab to monitor a discharge medication (Task 9)
Efficiency
The average task times for Tasks 7 and 9 are within one second of each other, suggesting a consistent
experience between inpatient and discharge lab ordering.
Effectiveness
The binary task completion rate for participants is 100 percent for Tasks 7 and 9. Participants took a negligible
number of unnecessary steps for both Tasks 7 and 9.
Satisfaction
The majority of participants rated both tasks Very Easy, with average satisfaction scores for Tasks 7 and 9 of
4.83 and 4.64, respectively.
Major Findings
One hundred percent of physicians completed the tasks, and usability metrics suggest that they did so with a
high degree of efficiency, effectiveness, and satisfaction.
Areas for Improvement
User feedback indicates little difficulty with the entry of laboratory orders, which can in part be attributed to
the consistency with other inpatient ordering workflows. As industry best practices for placing laboratory
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 28
orders, assessing relevant information, and communicating additional information to laboratories evolve, we
will continue to monitor this area for future enhancement opportunities, with specific focus on the following
areas:
The display of relevant clinical information at the point of ordering
The communication of additional details to laboratory staff
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 29
4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging
4.5.1 Data Analysis and Reporting
Table 10: Computerized Provider Order Entry – Diagnostic Imaging Physician Task Results
Computerized Provider Order Entry – Diagnostic Imaging
Efficiency Effectiveness Satisfaction
N Average
Task Time
Average Extra
Steps per Task
Binary Task
Completion Rate
Partial Task
Completion Rate
Average
Unnecessary Steps per Task
Task Ease
Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Modifying a diagnostic imaging
procedure
36 17.03
(5.63)
0.03 (0.17) 100 (0) 100 (0) 0.03 (0.17) 4.83 (0.38)
Ordering a diagnostic imaging
procedure
36 13.36
(10.96)
0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.72 (0.57)
4.5.2 Discussion of the Findings
Computerized provider order entry testing for diagnostic imaging tests with inpatient physicians covered two
tasks:
Modifying a diagnostic imaging procedure to include a comment (Task 4)
Ordering a diagnostic imaging procedure (Task 13)
Effectiveness
Participants completed Task 4 in an average of 17.03 seconds with negligible extra steps. Task 13 was
completed in an average of 13.36 seconds with negligible extra steps.
Effectiveness
The binary completion rate for Tasks 4 and 13 is 100 percent. Participants completed Task 4 with a negligible
number of unnecessary steps. All participants completed Task 13 with no unnecessary steps.
Satisfaction
The majority of participants rated the tasks as Very Easy, with average scores of 4.83 and 4.72 for Tasks 4 and
13, respectively.
Major Findings
All of the imaging order tasks were completed with high levels of effectiveness, efficiency, and satisfaction,
suggesting that physicians are comfortable using this functionality.
Areas for Improvement
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 30
Physicians successfully completed all imaging orders tasks. Diagnostic image ordering continues to be an
important topic with implications for patient care and the financial health of organizations. Although the test
results did not indicate specific areas for improvement, as new industry practices arise, future testing should
focus on the following areas:
Decision support for Appropriate Use Criteria
Display of radiation exposure information
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 31
4.6 §170.315(a)(5) Demographics
4.6.1 Data Analysis and Reporting
Table 11: Demographics Nurse Task Results
Demographics
Efficiency Effectiveness Satisfaction
N Average
Task Time
Average Extra
Steps per Task
Binary Task
Completion Rate
Partial Task
Completion Rate
Average
Unnecessary Steps per Task
Task Ease
Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD)
Mean (SD) Mean (SD) 5 = very
easy
Nurse Subtasks
Documenting preferred language 32 11.69 (9.75)
0.06 (0.25) 100 (0) 100 (0) 0.06 (0.25) 4.72 (0.58)
Updating sexual orientation and gender identity
36 10.26 (12.21)
0.11 (0.34) 97 (0.03) 97 (0.03) 0.03 (0.17) 4.61 (0.55)
4.6.2 Discussion of the Findings
Demographics testing with inpatient nurses covered two tasks:
Documenting the patient’s preferred language (Task 1)
Updating sexual orientation and gender identity as reported by the patient (Task 7)
Efficiency
Participants that completed Tasks 1 and 7 did so with an average time between 10 and 12 seconds and took a
negligible number of extra steps to complete Task 1. This suggests a consistently efficient experience when
documenting demographic information, such as sexual orientation, gender identity, and preferred language.
Effectiveness
Participants performed Tasks 1 and 7 with a negligible number of unnecessary steps and binary task
completion rates of 100 percent and 97 percent, respectively. Participants demonstrated they were able to
effectively document demographics information using the software.
Satisfaction
Participants on average rated Tasks 1 and 7 as between Somewhat Easy and Very Easy. Task 1, which involved
preferred language, had higher satisfaction metrics than Task 7, which involved gender identity.
Major Findings
Task 1, which involved preferred language, had slightly higher effectiveness and satisfaction metrics than Task
7, which involved gender identity. Based on debrief session feedback, some participants struggled with
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 32
discussing gender identity in a medical context, which led to uncertainty when completing documentation for
Task 7.
Areas for Improvement
Documenting gender identity is a relatively new workflow for most clinicians. Some nurses commented that
although they were comfortable with the demographics documentation tools in the system, they were
unfamiliar with the concepts and terminology for gender identity that they were asked to document. This
suggests the importance of training staff on gender identity prior to implementing the workflow for capturing
the information. In addition, given the ongoing evolution of standard terminology in the areas of sexual
orientation and gender identity, "just-in-time" training regarding the definition of unfamiliar terms could be
added to help facilitate user adoption.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 33
4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks
4.7.1 Data Analysis and Reporting
Table 12: Drug-Drug and Drug-Allergy Interaction Checks Physician Task Results
Drug-Drug, Drug-Allergy Interaction Checks
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Adding a medication allergy and assessing any interactions
35 10.49 (7.74)
0 (0) 100 (0) 100 (0) 0 (0) 4.51 (0.74)
Ordering a discharge medication and assessing any interactions
34 12.44 (10.18)
0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.03 (1.00)
Ordering an inpatient medication and assessing any interactions
35 23.63 (16.38)
0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.09 (0.31)
4.7.2 Discussion of the Findings
Drug-drug and drug-allergy interaction checks testing with inpatient physicians covered three tasks:
Adding a medication allergy and assessing any interactions (Task 3: Subtask B)
Ordering a discharge medication and assessing any interactions (Task 10: Subtask C)
Ordering an inpatient medication and assessing any interactions (Task 12: Subtask C)
Efficiency
Participants took no extra steps while completing Task 3 and a negligible number of extra steps while
completing Tasks 10 and 12.
Effectiveness
All three tasks had a binary completion rate of 100 percent, and participants took no unnecessary steps while
completing the tasks.
Satisfaction
The average satisfaction ratings for all tasks are between Somewhat Easy to Very Easy. Task 3 had higher
satisfaction ratings than Tasks 10 and 12, which may be attributed to the lower complexity of that task —
assessing an interaction after adding an allergy rather than assessing an interaction during an ordering
workflow.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 34
Major Findings
Physicians did not perform any unnecessary steps while completing the tasks and all three tasks had perfect
completion scores. Overall, physicians completed the tasks efficiently and effectively. The ease of task
completion ratings highlight an opportunity for improvement regarding assessing interactions during
ordering.
Areas for Improvement
Overall, physicians completed drug-allergy and drug-drug interaction tasks with high efficiency and
effectiveness. The ease of task completion ratings and debrief comments suggest that participants were
responding to the perceived frequency of alerts, not the functionality of the interaction alerts themselves. Alert
fatigue is an important topic in the healthcare industry which EHR vendors and organizations using EHR
software must address often. Organizations using Epic have a high degree of flexibility in configuring which
interaction warnings appear and to whom, which can be leveraged to address the concern of alert fatigue.
User satisfaction could be further addressed by investigating ways to display more visual cues of interactions
within the ordering display. However, sometimes more intrusive warnings are necessary to ensure they are
not overlooked, and any investigation will adhere to the principle that patient safety should always be
paramount, even at the expense of user satisfaction, as seen in section 4.3 §170.315(a)(1).
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 35
4.8 §170.315(b)(3) Electronic Prescribing
4.8.1 Data Analysis and Reporting
Table 13: Electronic Prescribing Physician Task Results
4.8.2 Discussion of the Findings
Electronic prescribing (e-prescribing) testing with inpatient physicians covered two tasks:
E-prescribing a discharge medication and assessing any interactions (Task 10: Subtask B)
E-prescribing a discharge medication (Task 15: Subtask B)
Efficiency
Participants took no extra steps while completing Tasks 10 and 15, suggesting a consistently efficient
experience for e-prescribing tasks.
Effectiveness
The binary task completion rate for both tasks is 100 percent. No participants performed unnecessary steps
while completing these tasks.
Satisfaction
The average satisfaction ratings for both tasks is between Somewhat Easy and Very Easy with a rating of 4.03
for Task 10 and 4.43 for Task 15. Task 15 had higher satisfaction ratings than Task 10, which may be attributed
to the lower complexity of that task which does not include assessing an interaction.
Major Findings
Physicians did not perform any extra or unnecessary steps while completing the tasks, and both of the tasks
had perfect completion scores. Overall, physicians completed the tasks efficiently and effectively, with ease of
task completion highlighting an opportunity for improvement. Participant debrief comments suggest that the
satisfaction scores for these two tasks might be due to the intrinsic complexity of the discharge order
reconciliation process, requiring both the review of existing information and placing new orders.
Electronic Prescribing
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
E-prescribing a discharge medication and assessing any interactions
34 36.47 (24.84)
0 (0) 100 (0) 100 (0) 0 (0) 4.03 (1.00)
E-prescribing a discharge medication 35 27.37 (13.92)
0 (0) 100 (0) 100 (0) 0 (0) 4.43 (0.95)
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 36
Areas for Improvement
Physicians successfully completed tasks to e-prescribe discharge medication orders with perfect efficiency and
effectiveness metrics. Ease of task completion scores indicate satisfaction as a possible area for improvement ,
such as through the following:
Based on observed actions and participant debrief comments, the workflow could be further
streamlined by reducing the amount of navigation required
Task 10 involves interaction checking, and the ease score for this task is likely influenced by the
interaction checking portion of the task. Improvements for interaction checks are discussed in section
4.7 §170.315(a)(4)Drug-Drug, Drug-Allergy Interaction Checks
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 37
4.9 §170.315(a)(14) Implantable Device List
4.9.1 Data Analysis and Reporting
Table 14: Implantable Device List Nurse Task Results
Implantable Device List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Nurse Subtasks
Adding a historically implanted device 36 61.23 (27.77)
0.14 (0.38) 97 (0.03) 99 (0.02) 0.17 (0.41) 4.39 (0.69)
Verifying the information for a historically implanted device
34 13.53 (13.76)
0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.74 (0.51)
Updating the information for a historically implanted device
35 8.74 (4.74)
0 (0) 100 (0) 100 (0) 0 (0) 4.94 (0.24)
4.9.2 Discussion of the Findings
Implantable device list testing with inpatient nurses covered three tasks:
Adding a historically implanted device based on information provided by the patient (Task 4)
Verifying the information for a historical entry on the implantable device list (Task 5)
Updating the information for a historical entry on the implantable device list based on information
provided by the patient (Task 10)
Efficiency
Participants who completed tasks verifying and updating historical implant information did so with zero or
negligible extra steps. As anticipated, Task 4, which involved adding a new implant to the chart, has a longer
task time than the other two tasks.
Effectiveness
The binary task completion rate for Tasks 5 and 10 is 100 percent, and 35 out of 36 participants completed Task
4. Participants that completed Tasks 5 and 10 did so without taking any unnecessary steps.
Satisfaction
The majority of participants rated the implants tasks as Somewhat Easy or Very Easy. The ease of task
completion rating is 4.39 for Task 4, 4.74 for Task 5, and 4.94 for Task 10.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 38
Major Findings
Nurses successfully verified and updated information for a historically implanted device, demonstrating a
strong effectiveness and satisfaction with the system even if they did not have prior experience with the
implantable devices functionality. The nature of adding a new historically implanted device required more
manual entry, which accounts for the longer average task time.
Areas for Improvement
The complexity and volume of data that needs to be entered into the system in a given workflow inversely
correlates with high satisfaction and effectiveness metrics. Further development can aim to bolster the efficiency
of the implantable device list by:
Optimizing the organization of data in the implantable device list
Improving visual cues for specific input fields
Similarly, there are techniques that have increased efficiency in other workflows by providing expedited ways
to document common responses. Further study can determine the best way to further implement these
techniques in the area of implantable device documentation in order to minimize high-volume data entry tasks
and optimize the efficiency of the workflow. Areas to explore include:
Auto-completion of implantable device information where clinically appropriate
More advanced default settings that leverage information such as context of use and implant type
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 39
4.10 §170.315(a)(8) Medication Allergy List
4.10.1 Data Analysis and Reporting
Table 15: Medication Allergy List Physician Task Results
4.10.2 Discussion of the Findings
Medication allergy list testing with inpatient physicians covered two tasks:
Adding a medication allergy and assessing any interactions (Task 3: Subtask A)
Entering new patient-reported information for an existing allergy (Task 11)
Efficiency
The average time taken to complete Task 3, which involved adding a new allergy, was 20.68 seconds, while the
average time for Task 11, which involved adding a reaction to an existing allergy, was 9.37 seconds.
Effectiveness
For Task 3, 34 out of 35 participants completed the task and for Task 11, 35 out of 36 participants completed the
task. No unnecessary steps were taken for either task.
Satisfaction
The majority of participants rated both allergy tasks as Very Easy. The ease of task completion rating is 4.51 for
Task 3 and 4.69 for Task 11.
Major Findings
The usability metrics suggest a consistent level of effectiveness and efficiency for adding a medication allergy
to a patient's chart and adding information to an existing allergy. The similar results for entering a new allergy
in Task 3 and for modifying an existing allergy in Task 11 suggest the consistency of the workflow for both
methods of updating the patient's medication allergy list.
Medication Allergy List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very easy
Physician Subtasks
Adding a medication allergy and assessing any interactions
35 20.68 (16.53)
0.09 (0.30) 97 (0.03) 99 (0.02) 0 (0) 4.51 (0.74)
Entering new information for an existing allergy
36 9.37 (7.54) 0.06 (0.24) 97 (0.03) 97 (0.03) 0 (0) 4.69 (0.75)
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 40
Areas for Improvement
Comments from participants and other research indicate that modifying existing allergies is an infrequent task
for most inpatient physicians, but results suggest that existing guidance provided by the system allows users
to successfully complete this task, despite a lower degree of familiarity.
Overall, physicians successfully added medication allergies to patients' charts. The few extra and unnecessary
steps relate to variations in allergy terminology usage, such as the distinction between reaction and reaction
type. Clearer industry definitions of reaction and reaction type may improve efficiency and effectiveness in
accurately documenting medication allergies.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 41
4.11 §170.315(a)(7) Medication List
4.11.1 Data Analysis and Reporting
Table 16: Medication List Nurse Task Results
4.11.2 Discussion of the Findings
Medication list testing with inpatient nurses covered two tasks:
Reviewing a complex medication list based on information provided by the patient (Task 3)
Adding a patient-reported medication (Task 9)
Efficiency
Participants that completed Tasks 3 and 9 did so at a high level of efficiency, with no extra steps taken while
completing the tasks.
Effectiveness
Thirty-five out of 36 participants successfully reviewed a complex medication list in Task 3 and 35 out of 36
added a patient-reported medication in Task 9. No unnecessary steps were taken to complete Task 3 and the
number of unnecessary steps is negligible for Task 9.
Satisfaction
A majority of users rated the medication list tasks as Very Easy. The two tasks garnered task ease ratings of
4.81 for Task 3 and 4.69 for Task 9, suggesting participants had a consistent level of satisfaction with both
tasks.
Major Findings
As indicated by the usability metrics, nurses completed Task 3 efficiently and effectively, while giving the task
a high satisfaction rating. In Task 9, 35 of 36 users successfully added the new medication to the patient's chart.
Overall, these results demonstrate high proficiency among nurses using the medication list.
Medication List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Nurse Subtasks
Reviewing a complex medication list 36 27.64 (19.19)
0 (0) 97 (0.03) 99 (0.01) 0 (0) 4.81 (0.47)
Adding a patient-reported medication 36 31.12
(13.76)
0 (0) 94 (0.04) 98 (0.02) 0.06 (0.24) 4.69 (0.58)
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 42
Areas for Improvement
Overall, efficiency and satisfaction scores were high for both tasks, with no extra steps taken and an average
satisfaction rating of Very Easy. Effectiveness metrics for Task 9 suggest improvements could be made to how
the medication list responds to new information. Effectiveness and ease of use could be further improved by
system updates to the medication list functionality to more explicitly indicate when the “last taken” date was
last updated and the prioritization of how recently added information is presented on the screen.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 43
4.12 §170.315(a)(6) Problem List
4.12.1 Data Analysis and Reporting
Table 17: Problem List Physician Task Results
Problem List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Adding a problem to a patient’s problem list
35 22.53 (10.98)
0.09 (0.30) 97 (0.03) 97 (0.03) 0 (0) 4.63 (0.49)
Reviewing a patient’s problem list 36 2.39 (1.81) 0 (0) 100 (0) 100 (0) 0 (0) 4.92 (0.37)
Resolving a problem on a patient’s problem list
35 3.14 (0.88) 0 (0) 100 (0) 100 (0) 0 (0) 4.83 (0.57)
4.12.2 Discussion of the Findings
Problem list testing with inpatient physicians covered three tasks:
Adding a problem to a patient’s problem list (Task 2)
Reviewing a patient’s problem list (Task 5)
Resolving a problem on a patient’s problem list (Task 14)
Efficiency
All participants completed Tasks 5 and 14 without any extra steps. Thirty-one out of 34 participants who
completed Task 2 did so without any extra steps.
Effectiveness
All participants completing the tasks did so without any unnecessary steps. The binary task completion rate
for both Tasks 5 and 14 is 100 percent, indicating a high level of effectiveness. The binary task completion rate
for Task 2 is 97 percent.
Satisfaction
The majority of participants rated the tasks as Very Easy. The average task ease rating for Task 2 was 4.63, for
Task 5 was 4.92, and for Task 14 was 4.83.
Major Findings
In Task 5 and 14 participants reviewed and resolved a problem on a patient's problem list efficiently and
effectively, while giving the tasks high ease of completion ratings. The higher ease of task completion rate
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 44
average for Tasks 5 and 14 compared to Task 2 is likely due to the lower inherent complexity of reviewing or
resolving a problem from a patient's problem list compared to adding a problem to a patient's problem list.
Areas for Improvement
In general, participants performed all problem list tasks with high effectiveness and satisfaction. Although the
results were positive and did not indicate specific areas for improvement, future study should focus on the
following areas as new industry practices arise:
• The evolution of problems over time to facilitate documentation of a longitudinal patient story
• Patients with a complex medical history and a wide variety of interrelated medical conditions
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 45
5 Results Conclusion Physicians and nurses performed tasks that were selected to address areas of Moderate to High risk (see section
3.3.3 for details). The overall average number of extra steps is 0.07 per task, average binary task completion is
99 percent, and average unnecessary steps is 0.02 per task. Common areas for improvement include:
Consolidating navigation to support a variety of clinical decision-making approaches
Due to differences in background and clinical education, clinicians can manage clinical decisions
in a wide range of ways. As discussed in sections 4.3, 4.8, and 4.9, participants were sometimes
observed using navigation and screen elements in a non-linear order. Further research could be
done to ensure that navigation supports both the most common workflows and significant
variations.
Enhancing visual cues to increase visibility of system status
Visibility of system status is key to guiding users to the appropriate next steps in a workflow and
provides immediate visual feedback of the effects of their choices. While elements of these design
principles were already observed to positively affect usability in our testing, we also identified
further opportunities for their use, such as in sections 4.1, 4.7, and 4.11.
In addition to Ease of Task Completion Scores recorded per task, each participant completed the System
Usability Scale (SUS) at the end of each testing session. The SUS is a reliable industry standard for measuring
user satisfaction.
Inpatient physicians gave an average SUS score of 85.94. Inpatient nurses gave an average SUS score of 87.41.
According to usability research, both the physician and nurse scores correlate with an “excellent” user
experience,8 indicating that Epic users find that the software is easy to use overall.
8 See Bangor, A., Kortum, P. T., & Miller, J.T. (2009). Determining What Individual SUS Scores Mean; Adding an Adjective Rating Scale. Journal of Usability Studies, 4(3), 114-123.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 46
Appendices
Appendix 1
Recruiting Screener
Note: Italicized text indicates information used for internal determination of eligibility and was not included
on the copy given to the participant.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 47
Recruiting Screener Demographic Information
1. Name:
2. Credentials:
3. Highest Level of Education:
a. High school graduate/GED
b. Some college
c. College graduate
d. Postgraduate
e. Other
4. Organization:
5. Primary Work Location:
6. Contact method (please provide one of the following):
a. Work phone:
b. Cell phone:
c. Email:
7. What is your gender?
a. Male
b. Female
c. Other (please specify):__________________
8. Which of these best describes your current age?
a. <20
b. 20-29
c. 30-39
d. 40-49
e. 50-59
f. 60-69
g. 70-79
h. ≥80
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 48
Additional Information
9. Is English your first language?
a. Yes
b. No
10. Are you a fluent English speaker? [if No, disqualify]
a. Yes
b. No
11. Due to logistical restraints and the parameters of this study, we cannot provide assistive technologies
during the testing session. Do you require any assistive technologies to use a computer? [if Yes,
disqualify]
a. Yes
b. No
12. Do you, or does anyone in your household, have a commercial interest in an electronic health record
software or consulting company? [if Yes, disqualify]
a. Yes
b. No
13. How many years of experience do you have using computers for personal and professional activities
(such as email, shopping, record keeping, etc.)?
a. <5 years
b. 5-10 years
c. 10-20 years
d. >20 years
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 49
14. What is your current role? [if not Nurse or Physician, disqualify]
a. Analyst
b. Application Coordinator
c. Certified Nursing Assistant (CNA)
d. CEO, CMIO, CIO, etc.
e. Consultant
f. Director
g. Information Technology
h. Licensed Practical Nurse (LPN)
i. Marketing/Communications
j. Medical Assistant (MA)
k. Nurse
l. Nurse Practitioner (NP)
m. Office Manager
n. Pharmacist
o. Physician
p. Physician Assistant (PA)
q. Project Manager
r. SVP, AVP, VP, etc.
s. Trainer
t. Other (please specify)
15. Do you currently provide direct patient care? [if No, disqualify]
a. Yes
b. No
16. In which setting do you primarily work? [if Ambulatory or Emergency Department, disqualify]
a. Inpatient
b. Emergency Department
c. Ambulatory
17. What is your specialty? [if role is Physician and specialty is Radiology, Ophthalmology or Pathology,
disqualify]
18. How many years have you been working in your field?
a. <5 years
b. 5-10 years
c. 10-20 years
d. >20 years
19. Have you participated in Epic usability testing previously?
a. Yes
b. No
If yes, please describe.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 50
20. How do you capture patient data in your organization? [if Primarily on paper, disqualify]
a. Primarily on paper
b. Primarily electronically
21. Is Epic the EHR you use most often in your organization? [if No, disqualify]
a. Yes
b. No
22. How long have you been using Epic? [if <3 months, disqualify]
23. How frequently do you use Epic? (daily, weekly, monthly)
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 51
Appendix 2
Testing Dates and Locations
Test Dates Locations
1 January 9, 2017 Ann Arbor, MI
2 January 9, 2017 Loma Linda, CA
3 January 10, 2017 Dearborn, MI
4 January 10, 2017 Royal Oak, MI
5 January 11, 2017 Los Angeles, CA
6 January 11, 2017 Tacoma, WA
7 January 12, 2017 Anaheim, CA
8 January 17, 2017 St. Louis Park, MN
9 January 17, 2017 St. Paul, MN
10 January 18, 2017 Minneapolis, MN
11 January 18, 2017 Kettering, OH
12 January 19, 2017 Maplewood, MN
13 January 19, 2017 Akron, OH
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 52
Appendix 3
Participant Demographics
Following is a high-level summary of participants in this study.
Gender Men 32 Women 40 Total (participants) 72
Occupation/Role RN/BSN 36 Physician 36 Total (participants) 72
Years of Experience Years of experience with Epic (average)
6.26
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 53
Appendix 4
IP Physician Scenario 1
Your first patient is Gertrude. Gertrude is a 55-year-old female who has been admitted to your unit from the
ED after experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition,
abrasions, and a possible concussion.
Task 1:
Gertrude mentions that she saw her PCP at an outside organization, River Hills Medical System. She
remembers that he diagnosed her with something new. Reconcile Gertrude’s problem list by adding and
accepting the information from River Hills.
Task 2:
You discuss Gertrude’s current symptoms with her, which include stiffness in both hip joints which makes
it difficult to walk or bend. You examine Gertrude and conclude she is experiencing osteoarthritis in both
hips. As there is no prior documentation of this diagnosis, add this problem to her chart.
Task 3:
Gertrude mentions that sulfa antibiotics give her a rash. Add sulfa antibiotics to her allergy list with a
reaction of rash. You know that any drug-allergy interaction due to her furosemide is unlikely. Use your
clinical judgment to respond to any warnings you receive.
Task 4:
Gertrude complains of pain in her right wrist and you see that it is bruised and swollen. You suspect that
she may have fractured it when she fell. The ED physician already ordered a right wrist x-ray. Update the
order with a comment to the radiologist to look for a possible scaphoid fracture. Sign the order when
complete.
IP Physician Scenario 2
Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently admitted to
your unit.
Task 5:
In addition to osteoarthritis of the left knee, Sheryl mentions that her current conditions include high
cholesterol and constipation. Document that you have reviewed her problem list .
Task 6:
As you speak with Sheryl, you learn that she is still experiencing significant pain, despite constant use of her
PCA. Increase the dose of her existing HYDROmorphone (Dilaudid) PCA order slightly by decreasing the
lockout interval to 5 minutes and increasing the basal rate to 0.2 mg/hour. After modifying, sign the order.
Task 7:
There were concerns with how much blood Sheryl lost in surgery, so a CBC was ordered for every 8 hours.
Her hemoglobin levels are looking good, so the frequency of the CBC order can be decreased. Modify the
existing CBC order to have a frequency of daily. After modifying, sign the order.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 54
Task 8:
Your hospital has a quality improvement initiative that all patients should be up-to-date with a pneumonia
immunization before discharge. Sheryl is not up-to-date with her pneumonia vaccine. Using the information
provided by the system, place and sign the suggested order.
Task 9:
You’ve already completed most of Sheryl’s discharge order reconciliation. Since she will continue her current
warfarin dose, you would like to continue monitoring Sheryl’s INR. Enter an order for Protime-INR. You’ll
also be prescribing a medication, so WAIT to sign this order.
Task 10:
You want to prescribe Sheryl warfarin for post-surgical prophylaxis. Your system automatically sends
prescriptions electronically to the patient’s preferred pharmacy, which has already been specified as Epic
Apothecary. Place and sign an order for warfarin. Sheryl is stabilized on her current warfarin dose. Sign all
orders when complete. Use your clinical judgement to respond to any warnings you receive.
IP Physician Scenario 3
Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and bradycardia.
He came to the ED complaining of weakness in his right side and was admitted to the ICU for a CVA.
Task 11:
While reviewing Arthur’s chart, you notice that there is no reaction for his aspirin allergy. He tells you that
aspirin gave him hives. Update Arthur’s aspirin allergy with a reaction of hives.
Task 12:
Your organization has an initiative to increase antiplatelet utilization for stroke and CVA patients. Using the
suggestion provided by the system and your clinical judgement, select and sign an order for an appropriate
antiplatelet therapy.
Task 13:
You decide to order a carotid ultrasound to help diagnose Arthur’s condition. Place and sign an order for a
bilateral carotid ultrasound.
Task 14:
Several days have passed and Arthur is ready to be discharged. Shortly after admission Arthur was
diagnosed with hypokalemia (low potassium). He responded well to the prescribed potassium drip and
now his levels are much better. Resolve hypokalemia in the problem list.
Task 15:
You’ve already completed most of Arthur’s discharge order reconciliation. You would like Arthur to
continue taking Plavix for antiplatelet therapy. Your system automatically sends prescriptions
electronically to the patient’s preferred pharmacy, which has already been specified as Epic Apothecary.
Place and sign an order for Plavix.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 55
IP Nurse Scenario 1
Your first patient is Walter. Walter is a 79-year-old male with a complex medical history that includes CHF,
osteoporosis, dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a
wound infection.
Task 1:
Walter’s preferred language is Spanish and his PCP speaks Spanish, so he has never needed an interpreter
until now. You have already documented that Walter needs interpreter services. Add Spanish as Walter’s
primary language.
Task 2:
Walter tells you that he went to an outside organization’s urgent care clinic and they determined he has a
new allergy, but he can't remember what the allergy is. You discuss the allergy with Walter and his wife and
confirm what is displayed as accurate. Reconcile Walter’s allergy list by adding and accepting the
information from River Hills.
Task 3:
Walter’s wife gives you a list of Walter’s at-home medications and states that he took them all yesterday.
Review his medications, adding or changing as needed. Indicate that Walter last took doses for all his
medications yesterday. Document that you have reviewed his medication list.
Task 4:
You’re taking Walter’s vitals when he mentions he has a pacemaker. You see that the pacemaker is not on
his Implants List. Walter does not have his pacemaker card with him but his wife gives you some details.
Add Walter’s pacemaker to the Implants List.
Task 5:
While discussing implants, Walter also mentions he had his hip replaced 6 years ago. You remember that
Depuy Orthopaedics recalled several of their hip replacement systems due to the implants shedding metal
shards. You want to determine if Walter could be affected. Review Walter’s hip implant and state verbally if
the manufacturer is Depuy Orthopaedics.
Task 6:
When giving Walter a snack, you notice he has difficulty chewing. After documenting this, you see that a
nutritional consult is recommended by the system. You discuss this with Walter, but he refuses to see a
nutritionist. Document that Walter has difficulty chewing and move to the next section. Review the
advisory given by the system and indicate that Walter refused the nutritional consult.
IP Nurse Scenario 2
Your next patient is Robin. Robin is a 50-year-old who prefers male gender pronouns. He has been admitted
for diabetic ketoacidosis.
Task 7:
Your organization has an initiative to more accurately document gender identity. There is already
documentation present for Robin and you discuss this with him. Robin informs you that while his sex
assigned at birth was female, he identifies as a man. Update this information in the chart.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 56
Task 8:
Robin recently saw his PCP at an outside organization, River Hills Medical System, for an allergic reaction.
His PCP documented a new allergy which you discuss with Robin and confirm is accurate. Reconcile Robin’s
allergy list by adding and accepting the information from River Hills.
Task 9:
Robin mentions that he is taking Lasix (furosemide) 20 mg at home and last took it yesterday. You also
confirm he is still taking his other medications. Add Lasix to the medication list and indicate that he also
took his other medications yesterday.
Task 10:
You notice Robin touching his right knee as he mentions some tenderness. He confirms that he had a right
knee replacement last year. You review the implant and notice there is no laterality listed for it. Add the
appropriate laterality to Robin’s chart.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 57
Appendix 5
Sample Participant Orientation
1) We are not testing you or your ability to use the system. We are measuring the usability of the system
itself.
2) You will be taken to the appropriate starting point for each task.
3) You will have a written copy of the task to read.
4) Work at your normal speed and only do what you are specifically asked to do in the system.
5) Because we are testing specific pieces of functionality, you may not complete the entirety of your
normal clinical workflow with a patient.
6) There may be multiple ways to complete a task. You can complete the task in whichever way is
apparent to you or easiest for you.
7) Verbalize that you are done upon completion of each task.
8) Fill out the Ease of Task Completion rating after each task.
9) You will complete a survey about your experience after all tasks are complete.
10) At the end of the test, we may discuss your thought process during specific tasks.
11) Save your comments until all tasks are completed. The facilitator will not offer help or answer any
questions during the test.
12) All of the information you provide will be kept confidential and your name will not be associated with
the results of this session.
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 58
Appendix 6
System Usability Scale Questionnaire
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 59
Rate the Functionality
This scale measures your overall impression of the functionality you tested today.
1 - Strongly Disagree 2 – Disagree 3 – Neutral 4 – Agree 5 – Strongly Agree
I think that I would enjoy using this system if I had to use it frequently
I found the system unnecessarily complex
I thought the system was easy to use
I think I would need the support of a technical person to be able to use the system
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I would imagine that most people would learn to use this system very quickly
I found the system very cumbersome to use
I felt very confident using the system
I need to learn a lot about this system before I could effectively use it
System Usability Scale
________________________________________________________________________________________________
© 2017 Epic Systems Corporation 60
Appendix 7
Frequency of Workflow and Possibility of Alternative Outcome
Possibility of Alternative Outcome
Negligible Low Moderate High
Fre
qu
en
cy o
f W
ork
flo
w
High Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood
Moderate Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood
Low Negligible Likelihood Low Likelihood Moderate Likelihood Moderate Likelihood
Negligible Negligible Likelihood Negligible Likelihood Low Likelihood Low Likelihood