EHR Usability Test Report Cx360 Version 7 · E. System Usability Scale ... gender identity, birth sex, preferred language – A5 14. Access and Display Demographics - A5 15. ... Cx360

Cx360 V7.0 EHR Usability Test Report

C o r e S o l u t i o n s I n c P a g e 1 | 77

EHR Usability Test Report

Cx360 Version 7.0



Cx360 Version 7.0

Date of Usability Test: 10/2/2017 – 10/6/2017

Date of Report: 10/25/2017

Report Prepared By: Cx360 Product Team

Attested by:

Balakumar Balasubramanian

Product Director



Table of Contents Executive Summary ....................................................................................................................................... 4

Major findings ......................................................................................................................................... 13

Areas for improvement ........................................................................................................................... 13

Introduction ................................................................................................................................................ 14

Method ....................................................................................................................................................... 14

PARTICIPANTS ......................................................................................................................................... 14

STUDY DESIGN ........................................................................................................................................ 16

TASKS ...................................................................................................................................................... 17

PROCEDURES ........................................................................................................................................... 19

TEST LOCATION ....................................................................................................................................... 20

TEST ENVIRONMENT ............................................................................................................................... 20

TEST FORMS AND TOOLS ........................................................................................................................ 20

PARTICIPANT INSTRUCTIONS .................................................................................................................. 21

USABILITY METRICS ................................................................................................................................. 23

DATA SCORING ........................................................................................................................................ 24

RESULTS ...................................................................................................................................................... 26

DATA ANALYSIS AND REPORTING ........................................................................................................... 26

DISCUSSIONS OF THE FINDINGS ............................................................................................................. 33

EFFECTIVENESS ....................................................................................................................................... 33

EFFICIENCY .............................................................................................................................................. 33

SATISFACTION ......................................................................................................................................... 33

MAJOR FINDINGS .................................................................................................................................... 34

AREAS FOR IMPROVEMENT .................................................................................................................... 34

APPENDICES ............................................................................................................................................ 35

A. Recruitment Phone Call Script ........................................................................................................ 35

B. Participants Demographics ............................................................................................................. 37

C. NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ............................................... 40

D. Moderators Guide ........................................................................................................................... 42

E. System Usability Scale ..................................................................................................................... 77



Executive Summary

A usability test of Cx360 Version 7.0 was conducted between 10/2/2017 – 10/6/2017 at King of

Prussia, Pennsylvania by Core Solutions Inc. The purpose of this test was to test and validate the

usability of the current user interface, and provide evidence of usability in the EHR Under Test

(Cx360). During the usability test, 10 healthcare providers matching the target demographic

criteria served as participants and used the Cx360 in simulated, but representative tasks.

This study collected performance data on the following tasks typically conducted on an EHR:

1. Create Medication Order – A1

2. Access and Display Medication Order - A1

3. Change Medication Order – A1

4. Create Lab Order – A2

5. Access Lab Order – A2

6. Change Lab Order – A2

7. Create Radiology / Imaging Order - A3

8. Access Radiology / Imaging Order – A3

9. Change Radiology / Imaging Order – A3

10. Verify drug-drug and drug-allergy interventions prior to Medication Order completion by

Triggering a drug- drug interaction – A4

11. Verify drug-allergy interventions prior to Medication Order completion by Triggering a

drug- allergy interaction - A4

12. Adjust Severity Levels of a displayed drug- drug interaction – A4

13. Record Demographics including Date of Birth, Race, Ethnicity , Sexual orientation,

gender identity, birth sex, preferred language – A5

14. Access and Display Demographics - A5

15. Change Demographic including Date of Birth, Race, Ethnicity , Sexual orientation,




16. Record Problems – A6

17. Access and Display historical Problems including Active Problems – A6

18. Change Problems – A6

19. Record Medication List – A7

20. Change Medication List – A7

21. Access Medication List and display active and historical medication list - A7

22. Record Medication Allergies – A8

23. Access Allergies and display active and historical medication list - A8

24. Change Allergies List – A8

25. Configure User Roles for CDS and Reference Information -A9

• Activate Demographics Clinical Decision Support Intervention -A9

• Activate Problem List Clinical Decision Support Intervention -A9

• Activate Allergy Clinical Decision Support Intervention -A9

• Activate Medication List Clinical Decision Support Intervention -A9

• Activate Vital Sign Clinical Decision Support Intervention -A9

• Activate Lab Test Clinical Decision Support Intervention -A9

• Activate combination rule ( Demographics and Problem List) -A9

• Activate drug-drug and drug-allergy Clinical Decision Support Intervention (Dr.

First/Rcopia) -A9

26. Trigger CDS interventions added using the applicable data elements from each of the

required elements – A9

27. Access CDS resource using the info button standard

28. Trigger CDS based on problem, medication, Allergy and demographics from a transition

of care referral summary – A9

29. Access CDS reference attributes including bibliographic citation, developer, funding

source, release/revision date – A9

30. Record Implantable Devices UDI – A14

31. Change Implantable Devices UDI Status – A14

32. Access UDI device description, identifiers, and attributes – A14



During the 60-minute test, each participant was greeted by the administrator and asked to provide

verbal consent on participation and they were instructed that they could withdraw at any time.

Some of the participants did not have prior experience with the EHR and Core Solutions provided

training on basic application navigation the functionalities under test. The administrator introduced

the test, and instructed participants to complete a series of tasks (given one at a time) using Cx360.

During the testing, the administrator timed the test and, along with the data logger(s) recorded user

performance data on paper and electronically. The administrator did not give the participant

assistance in how to complete the task.

Participant screens and audio were recorded for subsequent analysis. The following types of data

were collected for each participant:

• Number of tasks successfully completed within the allotted time without assistance

• Time to complete the tasks

• Number and types of errors

• Path deviations

• Participant’s verbalizations

• Participant’s satisfaction ratings of the system

All participant data was de-identified – no correspondence could be made from the identity of the

participant to the data collected. Following the conclusion of the testing, participants were asked

to complete a post-test questionnaire. Various recommended metrics, in accordance with the

examples set forth in the IST Guide to the Processes Approach for Improving the Usability of

Electronic Health Records, were used to evaluate the usability of the Cx360. Following is a

summary of the performance and rating data collected on Cx360.



Task Identifier Task Description

Task Success - Mean (%)

Task Success - Std Dev (%)

Task Path Deviation - Observed #

Task Path Deviation - Optimal #

Task Time - Mean (seconds)

Task Time - Standard Deviation (seconds)

Task Time Deviation - Mean Observed Seconds

Task Time Deviation - Mean Optimal Seconds

Task Errors Mean (%)

Task Errors - Std Dev %)

Task Rating - Scale Type

Task Rating

Task Rating - Standard Deviation

A1.1

Login to Cx360, Search for a client and record a medication using Medication Order electronic form 100 0 0 0 105 12.6 15 13 0 0

System Usability Scale

4.8 .7

A1.2

Edit the medication and change to a different medication 100 0 0 0 91 6.5 11 9 0 0

System Usability Scale 5 0

A1.3

Search for the client and access the medication list to view the updated medication 100 0 0 0 68 9.0 10 8 0 0


A2.1

Login to Cx360, Search for a client and record a Lab Order using Lab Order electronic form 100 0 0 0 115 13.7 16 14 0 0


A2.2

Edit the lab order and change to a different lab order 100 0 0 0 101 7.9 12 10 0 0


A2.3

Search for the client and access the lab order to view the updated lab order 100 0 0 0 70 11 13 9 0 0




A3.1

Login to Cx360, Search for a client and record a Radiology / Imaging Order using Radiology / Imaging Order electronic form 100 0 0 0 109 10.1 13 11 0 0


A3.2

Edit the Radiology / Imaging Order and change to a different Radiology / Imaging Order 100 0 0 0 95 7.1 9 8 0 0


A3.3

Search for the client and access the Radiology / Imaging Order to view the updated Radiology / Imaging Order 100 0 0 0 72 6.7 8 6 0 0


A4.1

Verify drug-drug

and drug-allergy

interventions prior

to Medication

Order completion

by Triggering a drug- drug

interaction 100 0 0 0 65 8.9 12 9 0 0


A4.2

Verify drug-allergy interventions prior to Medication Order completion by Triggering a drug- allergy interaction 100 0 0 0 45 7.3 12 10 0 0




A4.3

Adjust Severity Levels of a displayed drug- drug interaction 100 0 0 0 105 13.7 17 10 0 0


A5.1

1. Record Demographics

including Date of

Birth, Race,

Ethnicity, Sexual

orientation, gender

identity, birth sex, preferred language

– A5 100 0 0 0 120 14.3 13 8 0 0


A5.2

2. Access and Display

Demographics -

A5 100 0 0 0 40 4.8 8 6 0 0


A5.3

3. Change

Demographic including Date of

Birth, Race,

Ethnicity, Sexual orientation, gender

identity, birth sex,

preferred language – A5 100 0 0 0 75 8.7 7 5 0 0


A6.1

Login to Cx360, Search for a client and record a problem using Problem List electronic form 100 0 0 0 75 11.5 14 13 0 0


A6.2

Edit the Problem and change to a different Problem 100 0 0 0 55 4.5 8 6 0 0




A6.3

Search for the client and access the Problem list to view the updated Problem. Access and Display Historical Problems including Active Problems 100 0 0 0 60 8.5 9 7 0 0


A7.1

Login to Cx360, Search for a client and record a Medication using Medication List electronic form 100 0 0 0 95 11.3 12 11 0 0


4.6 .8

A7.2

Edit the Medication and change to a different Medication 100 0 0 0 75 7.3 13 10 0 0


A7.3

Search for the client and access the Medication list to view the updated Medications. Access and Display Historical Medications including Active Medications 100 0 0 0 60 8.5 9 7 0 0


A8.1

Login to Cx360, Search for a client and record a Medication Allergy using Medication List electronic form 100 0 0 0 103 12.6 15 13 0 0




A8.2

Edit the Medication Allergy and change to a different Medication 100 0 0 0 90 6.7 9 7 0 0


A8.3

Search for the client and access the Medication Allergies list to view the updated Medications. Access and Display Historical Medications including Active Medications 100 0 0 0 65 8.3 10 8 0 0


A9.1

Configure User Roles for CDS and Reference Information for the following Activate Demographics Clinical Decision Support Intervention. Activate Problem List Clinical Decision Support Intervention. Activate Allergy Clinical Decision Support Intervention. Activate Medication List Clinical Decision Support Intervention. Activate Vital Sign Clinical Decision Support Intervention. Activate Lab Test Clinical Decision Support Intervention. Activate combination rule (Demographics and Problem List). Activate drug-drug and drug-allergy Clinical Decision Support Intervention (Dr. First/Rcopia) 93 8.1 5 3 650 67.8 55 50 0 0

System Usability Scale 4 .8



A9.2

Trigger CDS

interventions

added using the applicable data

elements from

each of the required elements 100 0 0 0 890 87 45 40 0 0


4.8 .4

A9.3

Access CDS resources using info button 100 0 0 0 120 19 25 20 0 0


4.4 .6

A9.4

Trigger CDS based on problem, medication, Allergy and demographics from a transition of care referral summary 96 8.5 7 5 139 23 30 25 0 0


4.9 .5

A9.5

Access CDS reference attributes including bibliographic citation, developer, funding source, release/revision date 100 0 0 0 98 14 15 10 0 0


4.6 .4

A14.1

Record

Implantable

Devices UDI 100 0 0 0 78 11.3 12 10 0 0


4.8 .7

A14.2

Change Implantable

Devices UDI

Status 100 0 0 0 45 8.5 10 9 0 0


A14.3

Access UDI device

description,

identifiers, and attributes 100 0 0 0 65 5.0 10.5 8.5 0 0




The results from the System Usability Scale scored the subjective satisfaction with the system

based on performance with these tasks to be: 86.5

Major findings

Overall participants response on user experience was very positive. Several participants

emphasized on the ease of use of the system from a navigation and usability stand point. Every

participant was able to complete his or her assigned task with very little difficulty. Participants

were appreciative of the fully integrated medication management module. Single sign on feature

and the real-time sync feature of the electronic prescription modules were well received.

Participants also highlighted the layout of Cx360 client chart and the easy accessibility to client

data including allergies list, radiology orders, and lab and medication orders. Participants in

administrative roles appreciated the drug-drug and drug-allergies severity level setting option

including the clear and prominent alerts displayed in Cx360.

Areas for improvement

Overall participants were satisfied with the ease of use of the system. Some participants

highlighted that an option to add pharmacies not present in the system would be a useful feature

in the EHR. Participants also indicated the definitions of “Prescribe” vs “Manage Medications”

could be better highlighted in the application. Participants took additional time in figuring out the

difference between a signed and unsigned orders. Participants also pointed out a

few text areas that could be bigger form a UI stand point. Participant planning the administrative

role pointed out that the screen in which the “Drug-Drug” and “Drug-Allergy” interactions are

controlled had too many options and it was not easy to find the appropriate setting.



Introduction

The EHR tested for this study was Cx360 Version 7.0. Cx360 is a comprehensive electronic health

record platform services provider in ambulatory and inpatient setting. Cx360 integrated behavioral

Health EHR Platform enables the transformation of the behavioral health experience by improving

the provider to client relationship. Cx360 capabilities help to strengthen the trust-based

relationship between a provider and client by: simplifying the end-to-end behavioral health

experience, delivering integrated care coordination, improving client engagement and streamlining

accurate provider reimbursement.

The usability testing attempted to represent realistic exercises and conditions.

The purpose of this study was to test and validate the usability of the current user interface, and

provide evidence of usability in the EHR Under Test (Cx360). To this end, measures of

effectiveness, efficiency and user satisfaction, such as time on task, deviations etc. were captured

during the usability testing.

Method

PARTICIPANTS

A total of 10 participants were tested on the Cx360’s. Participants in the test were Physicians,

Doctors, Nurses and System Administrators. Participants had no direct connection to the

development of or organization producing Cx360. Participants were not from the testing or

supplier organization. Participants were given the opportunity to have the same orientation and

level of training as the actual end users would have received.



Recruited participants had a mix of backgrounds and demographic characteristics. The following

is a table of participants by characteristics, including demographics, professional experience and

computing experience. Participant names were replaced with Participant IDs so that an individual’s

data cannot be tied back to individual identities.

Participant Identifier

Participant Gender

Participant Age

Participant Education

Participant Occupation/Role

Participant Professional Experience

Participant Computer Experience

Participant Product Experience

Participant Assistive Technology Needs

A Male 40-49

Master's Degree Counselor 120 200 13 No

B Male 50-59

Master's Degree

Case Manager 150 225 20 No

C Female 40-49

Bachelor's Degree

Clinical Coordinator 60 200 12 No

D Male 40-49

Master's Degree


E Male 50-59

Bachelor's Degree

Clinical Assistant 140 250 14 No

F Female 30-39

Associate degree


G Female 30-39

Bachelor's Degree


H Female 20-29

Associate degree


I Female 50-59

Doctorate degree (e.g., MD, DNP, DMD, PhD) RN 150 220 20 No



J Female 60-69

Doctorate degree (e.g., MD, DNP, DMD, PhD) MD 200 250 20 No

Twelve participants were recruited and Ten participated in the usability test. Two participants

failed to show for the study. Participants were scheduled for 60 minutes sessions. A spreadsheet

was used to keep track of the participants, and included each participant’s demographic

characteristics.

STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – that

is, effectively, efficiently, and with satisfaction – and areas where the application failed to meet

the needs of the participants. The data from this test may serve as a baseline for future tests with

an updated version of the same EHR and/or comparison with other EHRs provided the same

tasks are used. In short, this testing serves as both a means to record or benchmark current usability,

but also to identify areas where improvements must be made.

During the usability test, participants interacted with Cx360. Each participant used the system in

the same location, and was provided with the same instructions. The system was evaluated for

effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each

participant:

• Number of tasks successfully completed within the allotted time without assistance

• Time to complete the tasks



• Number and types of errors

• Path deviations

• Participant’s verbalizations (comments)

• Participant’s satisfaction ratings of the system

TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of

activities a user might do with Cx360, including:








































First/Rcopia) -A9











1.



PROCEDURES

Upon arrival, participants were greeted; their identity was verified and matched with a name on

the participant schedule. Participants were then assigned a participant ID. To ensure that the test

ran smoothly, two staff members participated in this test, the usability administrator and one staff

member to log data and take notes. The usability testing staff conducting the test was experienced

usability practitioners with more than five years’ experience in the EHR industry.

The administrator moderated the session including administering instructions and tasks. The

administrator also monitored task times, obtained post-task rating data, and took notes on

participant comments. The other member served as the data logger and took notes on task success,

path deviations, number and type of errors, and comments.

Participants were instructed to perform the tasks (see specific instructions below):

➢ As quickly as possible making as few errors and deviations as possible.

➢ Without assistance; administrators were allowed to give immaterial guidance and

clarification on tasks, but not instructions on use.

➢ Without using a think aloud technique.

For each task, the participants were given a written copy of the task. Task timing began once the

administrator finished reading the question. The task time was stopped once the participant

indicated they had successfully completed the task.

Following the session, the administrator gave the participant the post-test system usability test and

thanked each individual for their participation.

Participants' demographic information, task success rate, time on task, errors, deviations, verbal

responses, and post-test questionnaire were recorded into a spreadsheet.



TEST LOCATION

The participants sessions were held remotely using Go-To meeting web technology. Participants

used their own computer hardware and facilities. To ensure that the environment was comfortable

for users, noise levels were kept to a minimum with the ambient temperature within a normal

range.

TEST ENVIRONMENT

Cx360 would be typically be used in a healthcare office or facility. In this instance, the testing was

conducted remotely via Go-To meeting from the participants clinical facilities. For testing, the

computer used a desktop running windows operation system with IE10. The participants used key

board and mouse when interacting with Cx360.

Cx360 which is based on Microsoft technology is a web based application and hosted in Core

Solutions managed web servers using a test data base. The participants accessed Cx360 via their

web browsers. Technically, the system performance (i.e., response time) was representative to

what actual users would experience in a field implementation. Additionally, participants were

instructed not to change any of the default system settings.

TEST FORMS AND TOOLS

During the usability test, various documents and instruments were used, including:

1. Moderator’s Guide

2. Post-test Questionnaire

The Moderator’s Guide was devised so as to be able to capture required data. The participant’s

interaction with Cx360 was captured using Go-To Meeting technology. Administrator and the

loggers witnessed the tests using Go-To Meeting.



PARTICIPANT INSTRUCTIONS

The administrator reads the following instructions aloud to each participant

Thank you for participating in this study. Your input is very important. Our session today will last

about 60 minutes. During that time, you will use an instance of Cx360. I will ask you to complete

a few tasks using this system and answer some questions. You should complete the tasks as quickly

as possible making as few errors as possible. Please try to complete the tasks on your own

following the instructions very closely. Please note that we are not testing you we are testing the

system, therefore if you have difficulty all this means is that something needs to be improved in the

system. I will be here in case you need specific help, but I am not able to instruct you or provide

help in how to use the application.

Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be

useful to you, and how we could improve it. I did not have any involvement in its creation, so

please be honest with your opinions. All of the information that you provide will be kept

confidential and your name will not be associated with your comments at any time. Should you

feel it necessary you are able to withdraw at any time during the testing.

Following the procedural instructions, participants were shown the EHR and as their first task,

were given time five minutes to explore the system and make comments. Once this task was

complete, the administrator gave the following instructions:

For each task, I will read the description to you and say “Begin.” At that point, please perform

the task and say “Done” once you believe you have successfully completed the task. I would like

to request that you not talk aloud or verbalize while you are doing the tasks.

Participants were then given the following tasks to complete.










































First/Rcopia) -A9











USABILITY METRICS

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic

Health Records, EHRs should support a process that provides a high level of usability for all users.

The goal is for users to interact with the system effectively, efficiently, and with an acceptable

level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were

captured during the usability testing. The goals of the test were to assess:

1. Effectiveness of Cx360 by measuring participant success rates and errors

2. Efficiency of Cx360 by measuring the average task time and path deviations



3. Satisfaction with Cx360 by measuring ease of use ratings

DATA SCORING

The following table details how tasks were scored, errors evaluated, and the time data analyzed.

Measures

Rationale and Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant

was able to achieve the correct outcome, without

assistance, within the time allotted on a per task basis.

The total number of successes were calculated for each

task and then divided by the total number of times that

task was attempted. The results are provided as a

percentage.

Task times were recorded for successes. Observed task

times divided by the optimal time for each task is a

measure of optimal efficiency.

Optimal task performance time, as benchmarked by

expert performance under realistic conditions, is

recorded when constructing tasks. Target task times

used for task times in the Moderator’s Guide must be

operationally defined by taking multiple measures of

optimal performance and multiplying by some factor

[e.g., 1.25] that allows some time buffer because the

participants are presumably not trained to expert

performance. Thus, if expert, optimal performance on

a task was [x] seconds then allotted task time

performance was [x * 1.25] seconds. This ratio should

be aggregated across tasks and reported with mean

and variance scores.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the

correct answer or performed it incorrectly, or reached

the end of the allotted time before successful



completion, the task was counted as an “Failures.” No

task times were taken for errors.

The total number of errors was calculated for each task

and then divided by the total number of times that task

was attempted. Not all deviations would be counted as

errors.11

On a qualitative level, an enumeration of errors and

error types should be collected.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the

application was recorded. Deviations occur if the

participant, for example, went to a wrong screen,

clicked on an incorrect menu item, followed an

incorrect link, or interacted incorrectly with an on-

screen control. This path was compared to the optimal

path. The number of steps in the observed path is

divided by the number of optimal steps to provide a

ratio of path deviation.

It is strongly recommended that task deviations be

reported. Optimal paths (i.e., procedural steps) should

be recorded when constructing tasks.

Efficiency:

Task Time

Each task was timed from when the administrator said

“Begin” until the participant said, “Done.” If he or she

failed to say “Done,” the time was stopped when the

participant stopped performing the task. Only task

times for tasks that were successfully completed were

included in the average task time analysis. Average

time per task was calculated for each task. Variance

measures (standard deviation and standard error) were

also calculated.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of

the application was measured by administering both a



simple post-task question as well as a post-session

questionnaire. After each task, the participant was

asked to rate “Overall, this task was:” on a scale of 1

(Very Difficult) to 5 (Very Easy). These data are

averaged across participants. 12

Common convention is that average ratings for systems

judged easy to use should be 3.3 or above.

To measure participants’ confidence in and likeability

of the CX360 overall, the testing team administered

the System Usability Scale (SUS) post-test

questionnaire. Questions included, “I think I would like

to use this system frequently,” “I thought the system

was easy to use,” and “I would imagine that most

people would learn to use this system very quickly.”

RESULTS

DATA ANALYSIS AND REPORTING

The results of the usability test were calculated according to the methods specified in the Usability

Metrics section above. Participants who failed to follow session and task instructions had their data

excluded from the analyses. The usability testing results for the Cx360 are detailed below. The

results should be seen in light of the objectives and goals outlined in Section Study Design.

Efforts were made to use descriptions and terminology which would be familiar to users. As part

of testing, users were presented with entirely new data entry screens. Users initially had some

hesitation in understanding the terminology , but later were comfortable in understanding the

systems terminology.



Task Identifier Task Description

Task Success - Mean (%)

Task Success - Std Dev (%)

Task Path Deviation - Observed #

Task Path Deviation - Optimal #

Task Time - Mean (seconds)

Task Time - Standard Deviation (seconds)

Task Time Deviation - Mean Observed Seconds

Task Time Deviation - Mean Optimal Seconds

Task Errors Mean (%)

Task Errors - Std Dev %)

Task Rating - Scale Type

Task Rating

Task Rating - Standard Deviation

A1.1

Login to Cx360, Search for a client and record a medication using Medication Order electronic form 100 0 0 0 105 12.6 15 13 0 0


4.8 .7

A1.2

Edit the medication and change to a different medication 100 0 0 0 91 6.5 11 9 0 0


A1.3

Search for the client and access the medication list to view the updated medication 100 0 0 0 68 9.0 10 8 0 0


A2.1

Login to Cx360, Search for a client and record a Lab Order using Lab Order electronic form 100 0 0 0 115 13.7 16 14 0 0


A2.2

Edit the lab order and change to a different lab order 100 0 0 0 101 7.9 12 10 0 0


A2.3

Search for the client and access the lab order to view the updated lab order 100 0 0 0 70 11 13 9 0 0




A3.1

Login to Cx360, Search for a client and record a Radiology / Imaging Order using Radiology / Imaging Order electronic form 100 0 0 0 109 10.1 13 11 0 0


A3.2

Edit the Radiology / Imaging Order and change to a different Radiology / Imaging Order 100 0 0 0 95 7.1 9 8 0 0


A3.3

Search for the client and access the Radiology / Imaging Order to view the updated Radiology / Imaging Order 100 0 0 0 72 6.7 8 6 0 0


A4.1

Verify drug-drug

and drug-allergy

interventions prior

to Medication

Order completion

by Triggering a drug- drug

interaction 100 0 0 0 65 8.9 12 9 0 0


A4.2

Verify drug-allergy interventions prior to Medication Order completion by Triggering a drug- allergy interaction 100 0 0 0 45 7.3 12 10 0 0




A4.3

Adjust Severity Levels of a displayed drug- drug interaction 100 0 0 0 105 13.7 17 10 0 0


A5.1

1. Record Demographics

including Date of

Birth, Race,

Ethnicity, Sexual

orientation, gender


– A5 100 0 0 0 120 14.3 13 8 0 0


A5.2

2. Access and Display

Demographics -

A5 100 0 0 0 40 4.8 8 6 0 0


A5.3

3. Change

Demographic including Date of

Birth, Race,

Ethnicity, Sexual orientation, gender

identity, birth sex,

preferred language – A5 100 0 0 0 75 8.7 7 5 0 0


A6.1

Login to Cx360, Search for a client and record a problem using Problem List electronic form 100 0 0 0 75 11.5 14 13 0 0


A6.2

Edit the Problem and change to a different Problem 100 0 0 0 55 4.5 8 6 0 0




A6.3

Search for the client and access the Problem list to view the updated Problem. Access and Display Historical Problems including Active Problems 100 0 0 0 60 8.5 9 7 0 0


A7.1

Login to Cx360, Search for a client and record a Medication using Medication List electronic form 100 0 0 0 95 11.3 12 11 0 0


4.6 .8

A7.2

Edit the Medication and change to a different Medication 100 0 0 0 75 7.3 13 10 0 0


A7.3

Search for the client and access the Medication list to view the updated Medications. Access and Display Historical Medications including Active Medications 100 0 0 0 60 8.5 9 7 0 0


A8.1

Login to Cx360, Search for a client and record a Medication Allergy using Medication List electronic form 100 0 0 0 103 12.6 15 13 0 0




A8.2

Edit the Medication Allergy and change to a different Medication 100 0 0 0 90 6.7 9 7 0 0


A8.3

Search for the client and access the Medication Allergy list to view the updated Medications. Access and Display Historical Medications including Active Medications 100 0 0 0 65 8.3 10 8 0 0


A9.1

Configure User Roles for CDS and Reference Information for the following Activate Demographics Clinical Decision Support Intervention. Activate Problem List Clinical Decision Support Intervention. Activate Allergy Clinical Decision Support Intervention. Activate Medication List Clinical Decision Support Intervention. Activate Vital Sign Clinical Decision Support Intervention. Activate Lab Test Clinical Decision Support Intervention. Activate combination rule (Demographics and Problem List). Activate drug-drug and drug-allergy Clinical Decision Support Intervention (Dr. First/Rcopia) 93 8.1 5 3 650 67.8 55 50 0 0

System Usability Scale 4 .8



A9.2

Trigger CDS

interventions

added using the applicable data

elements from

each of the required elements 100 0 0 0 890 87 45 40 0 0


4.8 .4

A9.3

Access CDS resources using info button 100 0 0 0 120 19 25 20 0 0


4.4 .6

A9.4

Trigger CDS based on problem, medication, Allergy and demographics from a transition of care referral summary 96 8.5 7 5 139 23 30 25 0 0


4.9 .5

A9.5

Access CDS reference attributes including bibliographic citation, developer, funding source, release/revision date 100 0 0 0 98 14 15 10 0 0


4.6 .4

A14.1

Record

Implantable

Devices UDI 100 0 0 0 78 11.3 12 10 0 0


4.8 .7

A14.2

Change Implantable

Devices UDI

Status 100 0 0 0 45 8.5 10 9 0 0


A14.3

Access UDI device

description,

identifiers, and attributes 100 0 0 0 65 5.0 10.5 8.5 0 0




DISCUSSIONS OF THE FINDINGS

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the

system based on performance with these tasks to be 86.5

EFFECTIVENESS

Participants statements reflected that the system was effective and user friendly. All the

participants were able to successfully complete the tasks. Most of the tasks were rated as very easy

to accomplish in the system. Participant’s feedback on the systems ease of use were predominantly

positive. Overall the SUS scores and test results underscores the system’s effectiveness.

Administrative level participants were appreciating of the fact that CDS rules can be controlled by

user roles resulting in alerts being displayed by user role. Administrators indicated that this would

result in improved user efficiency as users not having to view the alerts not pertaining to their role

and having ready access to reference materials.

EFFICIENCY

Participants were able to efficiently navigate the system and complete all the assigned tasks. Path

deviations were minimal and the participants time to complete tasks were very close to the optimal

task completion time. Director level participants were excited about the increase in productivity

that the system could offer for their organization. Nurses were appreciative of the fact that their

time could be efficiently used towards patient care. Some of the elevated task completion time

were attributed to participants eagerness to check out additional features of the application rather

than the any inefficiencies in the system.

SATISFACTION

Study participants were asked to fill out and submit a survey based on Subjective Usability Scale

(SUS). The results from the SUS scored the subjective satisfaction with the system based on

performance with the tasks to be 86.5 which indicates very high level of satisfaction among the



participants. Participants indicated their high level of satisfaction with respect to system’s ease of

use and ready access to client information.

MAJOR FINDINGS

Overall participants response on user experience was very positive. Several participants

emphasized on the ease of use of the system from a navigation and usability stand point. Every

participant was able to complete his or her assigned task with very little difficulty. Participants

were appreciative of the fully integrated electronic prescription and medication management

module. Single sign on feature and the near real-time sync of electronic feature of the electronic

prescription modules were well received. Participants also highlighted the layout of Cx360 client

chart and the easy accessibility to client data including allergies list, radiology orders, lab and

medication orders. Participants in administrative roles appreciated the drug-drug and drug-

allergies severity level setting option including the clear and prominent alerts displayed in Cx360.

Participants were appreciative of the ease of use with respect to setting up the CDS rules and

activating them as part of client workflow. Participants in administrative roles appreciated the

configuration features in Cx360 that allowed the CDS rules and reference information to be

controlled effortlessly. Users also appreciated the clear and prominent alerts displayed in Core

Cx360 along with link to the associated reference material.

AREAS FOR IMPROVEMENT

Overall participants were satisfied with the ease of use of the system. Some participants

highlighted that an option to add pharmacies not present in the system would be a useful feature

in the EHR. Participants also indicated the definitions of “Prescribe” vs “Manage Medications”

could be better highlighted in the application. Participants took additional time in figuring out the

difference between a signed and unsigned orders. Participants also pointed out a few text areas

that could be bigger form a UI stand point. Participant planning the administrative role pointed

out that the screen in which the “Drug-Drug” and “Drug-Allergy” interactions are controlled had

too many options and it was not easy to find the appropriate setting. Participants reported the

scroll bar within the prescription module was a little confusing as some of the buttons were not



readily visible. Overall participants were satisfied with the ease of use of the Core Cx360. Some

participants indicated that an end user configurable CDS engine would be a useful feature in the

EHR could be better highlighted in the application. Participants took additional time in figuring

out the association of rules with corresponding referral resources. Participants conveyed that a

few additional labels with directions on what the fields meant in CDS engine would help the end

user. Several questions were raised about the systems capability to place prescription as

telephone orders and suggestions were given to have a capability to capture telephone orders.

APPENDICES

A. Recruiting Screener Template

B. Participant Demographics

C. Non-Disclosure Agreement (NDA) and Informed Consent Form

D. Moderator’s Guide

E. System Usability Scale Questionnaire

A. Recruitment Phone Call Script

Hello,

Recruiting Script for Recruiting Firm Hello, my name is <>, calling from Core Solutions Inc. We are

recruiting individuals to participate in a usability study for an electronic health record. We would like to

ask you a few questions to see if you qualify and if would like to participate. This should only take a few

minutes of your time. This is strictly for research purposes. If you are interested and qualify for the

study, you will be paid to participate. Can I ask you a few questions?

1. Have you participated in a focus group or usability test in the past 12 months? [If yes, Terminate]

2. Do you, or does anyone in your home, work in marketing research, usability research, web design

[…etc.]? [If yes, Terminate]

3. Do you, or does anyone in your home, have a commercial or research interest in an electronic health

record software or consulting company? [If yes, Terminate]



4. Which of the following best describes your age? [23 to 39; 40 to 59; 60 - to 74; 75 and older]

5. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian,

Black/African-American, Latino/a or Hispanic, etc.]

6. Do you require any assistive technologies to use a computer? [if so, please describe]

7. What is your current position and title? (Must be healthcare provider)

8. How long have you held this position?

9. Describe your work location (or affiliation) and environment? (Recruit according to the intended

users of the application) [e.g., private practice, health system, government clinic, etc.]

10. Which of the following describes your highest level of education? [e.g., high school graduate/GED,

some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)]

Computer Expertise Customize this to reflect what you know about your EHR’s audience

11. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR,

research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no

computer uses at all, Terminate]

12. About how many hours per week do you spend on the computer? [Recruit according to the

demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]

13. What computer platform do you usually use? [e.g., Mac, Windows, etc.]

14. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]

15. In the last month, how often have you used an electronic health record?

16. How many years have you used an electronic health record?

17. How many EHRs do you use or are you familiar with?

18. How does your work environment patient records? [Recruit according to the demographics of the

intended users]

Contact Information If the person matches your qualifications, ask Those are all the questions I have for

you. Your background matches the people we're looking for.

Would you be able to participate on any day between October 2nd and October 6th? [If so collect contact

information]

May I get your contact information?

Name of participant:



Address:

City, State, Zip:

Daytime phone number:

Evening phone number:

Alternate [cell] phone number:

Email address:

B. Participants Demographics

Following is a high-level overview of the participants in this study

Gender

Men - 4

Women – 6

Total (participants) – 10

Occupation/Role

RN/BSN - 1

Physician/MD – 1

Admin Staff/Clinical Assistant/Coordinator – 4

Case Manager – 3

Counselor – 1

Total (participants) - 10

Years of Experience

Facility Use of EHR All paper - 1

Some paper, some electronic – 5

All electronic - 4




Assistive Technology Needs

Yes – 0

No – 10


Participant Identifier

Participant Gender

Participant Age

Participant Education

Participant Occupation/Role

Participant Professional Experience

Participant Computer Experience

Participant Product Experience

Participant Assistive Technology Needs

A Male 40-49

Master's Degree Counselor 120 200 13 No

B Male 50-59

Master's Degree


C Female 40-49

Bachelor's Degree

Clinical Coordinator 60 200 12 No

D Male 40-49

Master's Degree


E Male 50-59

Bachelor's Degree


F Female 30-39

Associate degree


G Female 30-39

Bachelor's Degree


H Female 20-29

Associate degree




I Female 50-59

Doctorate degree (e.g., MD, DNP, DMD, PhD) RN 150 220 20 No

J Female 60-69

Doctorate degree (e.g., MD, DNP, DMD, PhD) MD 200 250 20 No



C. NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM

Non-Disclosure Agreement

THIS AGREEMENT is entered into as of _ _, 2017, between _________________________ (“the

Participant”) and the testing organization Core Solutions Inc. located at 620 Freedom Business Center

King of Prussia PA-19468. The Participant acknowledges his or her voluntary participation in today’s

usability study may bring the Participant into possession of Confidential Information. The term

"Confidential Information" means all technical and commercial information of a proprietary or

confidential nature which is disclosed by Core Solutions Inc, or otherwise acquired by the Participant, in

the course of today’s study.

By way of illustration, but not limitation, Confidential Information includes trade secrets, processes,

formulae, data, know-how, products, designs, drawings, computer aided design files and other

computer files, computer software, ideas, improvements, inventions, training methods and materials,

marketing techniques, plans, strategies, budgets, financial information, or forecasts.

Any information the Participant acquires relating to this product during this study is confidential and

proprietary to Core Solutions Inc and is being disclosed solely for the purposes of the Participant’s

participation in today’s usability study. By signing this form, the Participant acknowledges that s/he will

receive monetary compensation for feedback and will not disclose this confidential information

obtained today to anyone else or any other organizations.

Participant’s printed name: ___________________________________________

Signature: _____________________________________ Date: ____________________



Informed Consent

Core Solutions Inc would like to thank you for participating in this study. The purpose of this study is to

evaluate an electronic health records system. If you decide to participate, you will be asked to perform

several tasks using the prototype and give your feedback. The study will last about 60 minutes. At the

conclusion of the test, you will be compensated for your time.

Agreement

I understand and agree that as a voluntary participant in the present study conducted by Core Solutions

Inc. I am free to withdraw consent or discontinue participation at any time. I understand and agree to

participate in the study conducted and recorded by the Core Solutions Inc.

I understand and consent to the use and release of the recording by Core Solutions Inc. I understand

that the information and recording is for research purposes only and that my name and image will not

be used for any purpose other than research. I relinquish any rights to the recording and understand the

may be copied and used by Core Solutions Inc without further permission.

I understand and agree that the purpose of this study is to make software applications more useful and

usable in the future. I understand and agree that the data collected from this study may be shared with

outside of Core Solutions Inc and Core Solutions Inc’s client. I understand and agree that data

confidentiality is assured, because only deidentified data – i.e., identification numbers not names – will

be used in analysis and reporting of the results.

I agree to immediately raise any concerns or areas of discomfort with the study administrator. I

understand that I can leave at any time.

Please check one of the following:

[] YES, I have read the above statement and agree to be a participant.

[] NO, I choose not to participate in this study.

Signature: _____________________________________ Date: ____________________



D. Moderators Guide

Administrator ________________________

Data Logger ________________________

Date _____________________________ Time _________

Participant # ________

Location ____________________________

Prior to testing

Confirm schedule with Participants

Ensure Cx360 lab environment is running properly

Ensure lab and data recording equipment is running properly Prior to each participant:

Reset application

Start Go to Meeting session recordings Prior to each task:

Reset application to starting point for next task After each participant:

End Go To Meeting session recordings

Orientation (5 minutes)

Thank you for participating in this study. Our session today will last 60 minutes. During that time, you

will take a look at an electronic health record system. I will ask you to complete a few tasks using this

system and answer some questions. We are interested in how easy (or how difficult) this system is to

use, what in it would be useful to you, and how we could improve it. You will be asked to complete

these tasks on your own trying to do them as quickly as possible with the fewest possible errors or

deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help

you with anything to do with the system itself. Please save your detailed comments until the end of a

task or the end of the session as a whole when we can discuss freely. I did not have any involvement in

its creation, so please be honest with your opinions. Some of the data may not make sense as it is

placeholder data. We are recording the audio and screenshots of our session today. All of the

information that you provide will be kept confidential and your name will not be associated with your

comments at any time. Do you have any questions or concerns?



Task 1 - Login to Cx360, Search for a client and record a medication using Medication Order electronic

form

Take the participant to the starting point for the task which is the login page of Cx360

Success:

[] Easily Completed

[] Completed with difficulty or help: Describe below

[] Not completed

Comments:

Task Time: ________ Seconds

Optimal Path: Enter Login Details in login page Enter client name John Doe in search box Click on

Medication Order Select Medication (Xanax) and fill the rest of the fields

[] Easily Completed

[] Minor Deviations: Describe below

[] Major Deviations: Describe below

Comments:

Observed Errors and Verbalizations:

Comments:

Rating: Overall, this task was: ______



Task 2 - Edit the medication and change to a different medication

Take the participant to the starting point for the task which Client Chart of client John Doe

Success:

[] Easily Completed

[] Completed with difficulty or help: Describe below

[] Not completed

Comments:


Optimal Path: Click on Medication Order Select Medication (Xanax) and click on edit Change the

medication to (Prozac) and fill out the rest of the fields and click on “OK”

[ ] Easily Completed

[ ] Minor Deviations:: Describe below

[ ] Major Deviations :: Describe below

Comments:


Comments:




Task 3 - Search for the client and access the medication list to view the updated medication


Success:


[ ] Completed with difficulty or help :: Describe below

[ ] Not completed

Comments:


Optimal Path: Click on Medication Order Look for medication (Prozac) Select Medication (Prozac )

and click on edit click on “OK”


[ ] Minor Deviations :: Describe below


Comments:


Comments:




Task 4 - Login to Cx360, Search for a client and record a Lab Order using Lab Order electronic form


Success:



[ ] Not completed

Comments:



Lab Order Select Lab Order (HBA1C) and fill the rest of the fields




Comments:


Comments:




Task 5 - Edit the lab order and change to a different lab order


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Lab Order Select Lab Order (HBA1C ) and click on edit Change the Lab order

to (HDL-C) and fill out the rest of the fields and click on “OK”




Comments:


Comments:




Task 6 - Search for the client and access the Lab Order list to view the updated Lab Order


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Lab Order Look for Lab (HDL-C) Select Medication (HDL-C ) and click on

edit click on “OK”




Comments:


Comments:




Task 7 - Login to Cx360, Search for a client and record a Radiology / Imaging Order using Radiology / Imaging Order electronic form


Success:



[ ] Not completed

Comments:



Radiology Order Select Radiology Order (X-Ray Left Foot) and fill the rest of the fields




Comments:


Comments:




Task 8 - Edit the Radiology / Imaging Order and change to a different Radiology / Imaging Order


Success:



[ ] Not completed

Comments:


Optimal Path: Click Radiology Order Select Radiology Order (X-Ray Left Foot) and click on edit

Change the Radiology order to (X-Ray Right Foot) and fill out the rest of the fields and click on “OK”




Comments:


Comments:




Task 9 - Search for the client and access the Radiology / Imaging Order to view the updated

Radiology / Imaging Order


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Radiology Order Look for Radiology (X-Ray Left Foot) Select Radiology Order

(X-Ray Left Foot)) and click on edit click on “OK”




Comments:




Task 10 - Verify drug-drug and drug-allergy interventions prior to Medication Order completion by

Triggering a drug- drug interaction


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Medication Oder Add a Medication record for (Prozac) Oder medication

for (Xanax) Verify Drug – Drug interaction alert




Comments:




Task 11 - Verify drug-allergy interventions prior to Medication Order completion by Triggering a drug- allergy

interaction


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Medication Oder Add a Allergy record for Xanax Oder medication for

(Xanax) Verify Drug – Allergy interaction alert




Comments:




Task 12 - Adjust Severity Levels of a displayed drug- drug interaction


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Medication Oder Access Setting Lower the Severity Level Add a Allergy

record for Xanax Oder medication for (Xanax) Verify Drug – Allergy interaction alert based on the

adjusted severity level




Comments:




Task 13 - Record Demographics including Date of Birth, Race, Ethnicity , Sexual orientation, gender


Take the participant to the starting point for the task which Client Chart Page

Success:



[ ] Not completed

Comments:


Optimal Path: Click on Profile Enter the following data including Date of Birth, Race, Ethnicity ,

Sexual orientation, gender identity, birth sex, preferred language




Comments:




Task 14 - Access and Display Demographics


Success:



[ ] Not completed

Comments:


Optimal Path: Search for Client John Doe Open Profile and view demographics data




Comments:




Task 15 - Change Demographic including Date of Birth, Race, Ethnicity , Sexual orientation,

gender identity, birth sex, preferred language


Success:



[ ] Not completed

Comments:


Optimal Path: Search for Client John Doe Open Profile and view demographics data Change

Demographic including Date of Birth, Race, Ethnicity , Sexual orientation, gender identity, birth

sex, preferred language




Comments:




Task 16 - Login to Cx360, Search for a client and record a problem using Problem List electronic form

Take the participant to the starting point for the task which Client Chart Page of Client John Doe

Success:



[ ] Not completed

Comments:


Optimal Path: Click on Problems Add a problem for the client Hypertension (ICD 10 code I10) and set

the problem as Active




Comments:




Task 17 - Edit the Problem and change to a different Problem


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Problems Access Hypertension (ICD 10 code I10) Edit Problem and Change

to ICD 10 Code E11.9 (Type 2 diabetes mellitus)




Comments:




Task 18 - Search for the client and access the Problem list to view the udpated Problem . Access and

Display historical Problems including Active Problems


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Problems Access ICD 10 Code E11.9 (Type 2 diabetes mellitus)) View the

changed information




Comments:




Task 19 - Login to Cx360, Search for a client and record a medication using Medication List electronic

form


Success:



[ ] Not completed

Comments:



Medication List Select Medication (Xanax) and fill the rest of the fields




Comments:


Comments:




Task 20 - Edit the medication list and change to a different medication


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Medication List Select Medication (Xanax ) and click on edit Change the

medication to (Prozac) and fill out the rest of the fields and click on “OK”




Comments:


Comments:




Task 21 - Search for the client and access the medication list to view the updated medication


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Medication List Look for medication (Prozac) Select Medication (Prozac )

and click on edit click on “OK”




Comments:


Comments:




Task 22 - Login to Cx360, Search for a client and record a medication allergy using Medication Allergy

List electronic form


Success:



[ ] Not completed

Comments:



Allergy List Select Medication Allergy (Xanax) and fill the rest of the fields




Comments:


Comments:




Task 23 - Edit the medication allergy list and change to a different Medication Allergy


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Medication Allergy List Select Medication Allergy (Xanax ) and click on edit

Change the Medication Allergy to (Prozac) and fill out the rest of the fields and click on “OK”




Comments:


Comments:




Task 24 - Search for the client and access the medication allergy list to view the updated

medication allergy


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Medication Allergy List Look for medication allergy (Prozac) Select

Medication Allergy (Prozac ) and click on edit click on “OK”




Comments:


Comments:




Task 25 - Configure User Roles for CDS and Reference Information for the following Activate

Demographics Clinical Decision Support Intervention. Activate Problem List Clinical Decision Support

Intervention. Activate Allergy Clinical Decision Support Intervention. Activate Medication List Clinical

Decision Support Intervention. Activate Vital Sign Clinical Decision Support Intervention. Activate Lab

Test Clinical Decision Support Intervention. Activate combination rule ( Demographics and Problem List).

Activate drug-drug and drug-allergy Clinical Decision Support Intervention (Dr. First/Rcopia)

Take the participant to the starting point for the task which Configuration Page

Success:



[ ] Not completed

Comments:


Optimal Path: Click on Configuration Page Click on Configure CDS Activate the following CDS

Problem list - Control the High Blood Pressure

Medication list - Order or Prescribe “Anti-Platelet Therapy”

Medication allergy list - Patient’s drug allergies are not documented or "No known drug allergies" has

not been selected.

Demographics - Age Based Immunization Schedule

Laboratory test and values/results - Order “HbA1C” lab test

Vital Signs - BMI is outside the acceptable parameters, create a follow-up Plan

Combination (Demographics & Problem list) - Assess Client for Suicide Risk

Incorporated Lab Results - Specimen Rejected - Initiate New Lab Order




Comments:


Comments:




Task 26 - Trigger CDS interventions added using the applicable data elements from each of the required

elements

Take the participant to the starting point for the task which is Client Chart John Doe

Success:



[ ] Not completed

Comments:


Optimal Path: Perform the following tasks to trigger CDS


In this rule, the application checks for client’s age between 18 to 85 years with a Vital Sign recorded

(Systolic > 140 or Diastolic > 90 in vital sign) and a Problem of Hypertension. If a Problem of

Hypertension is recorded for a Client, then notification will be created and displayed indicating that

the Diastolic and Systolic pressure should be brought to control for the client


In this rule, the application checks for the client’s with active problems, if “Coronary Artery Disease” is

an active problem, and then the application checks for the medications. If there is no medication

“Ticlid” or “Plavix” as an active medication, then a notification will be created and displayed indicating

that medication associated with “Anti-Platelet Therapy” should be ordered or prescribed for the client

Medication allergy list - Patient’s drug allergies are not documented or "No known drug

allergies" has not been selected.

In this rule, the application checks for the client’s Medication allergies when the Medication list form is

saved. If the client did not have any Medication allergies or a minimum of "No known drug allergies”

recorded, then the notifications will be created and displayed indicating Medication allergies are not

verified.

Demographics - Order “HbA1C” lab test

In this rule, the application checks for the Client’s between 18 to 75 years of age with active problems,

if “Diabetes” is an active problem, then the application checks for the Lab Results. If there is no



“HbA1C” lab test in the Lab Results then a notification will be created and displayed indicating that

“HbA1C” lab test should be ordered for the client

Laboratory test and values/results - Diabetes not under control


if “Diabetes” is an active problem, then the application checks for the Lab Results. If there is “HbA1C”

lab test in the Lab Results in the last six months with a Result value >9.0 then a notification will be

created and displayed indicating that Diabetes is not under control for the client.


In this rule, the application checks for the Client’s Vital Signs (Height and Weight) and calculates the

BMI, if the BMI is outside the acceptable parameters, then notification will be created and displayed

indicating that the BMI is high and a follow-up Plan should be created. Client’s of Age 65 years and older BMI => 23 and < 30 Client’s of Age 18 – 64 years BMI => 18.5 and < 25


In this rule, the application checks for the client’s age between 6 to 17 years and having a Problem of

Major Depressive Disorder. If the Suicide Risk Assessment is not recorded for those clients, then a

notification is created indicating to assess the client for Suicide risk.


In this rule, when an incorporated Lab results has the information of Specimen rejected reason, a

notification is created to intimate that a new Lab order should be initiated.




Comments:


Comments:




Task 27 - Access CDS resources using info button


Success:



[ ] Not completed

Comments:


Optimal Path: Click on info button for each of the CDS below and review the information




not been selected.









Comments:


Comments:




Task 28 - Trigger CDS based on problem, medication, Allergy and demographics from a transition of care

referral summary

Take the participant to the starting point for the task which is Client Chart John Doe

Success:



[ ] Not completed

Comments:


Optimal Path: Perform the following tasks to trigger CDS based on incorporated data based on

transition of care referral summary


In this rule, the application checks for client’s age between 18 to 85 years with a Vital Sign recorded

(Systolic > 140 or Diastolic > 90 in vital sign) and a Problem of Hypertension. If a Problem of

Hypertension is recorded for a Client, then notification will be created and displayed indicating that

the Diastolic and Systolic pressure should be brought to control for the client


In this rule, the application checks for the client’s with active problems, if “Coronary Artery Disease” is

an active problem, and then the application checks for the medications. If there is no medication

“Ticlid” or “Plavix” as an active medication, then a notification will be created and displayed indicating

that medication associated with “Anti-Platelet Therapy” should be ordered or prescribed for the client

Medication allergy list - Patient’s drug allergies are not documented or "No known drug

allergies" has not been selected.

In this rule, the application checks for the client’s Medication allergies when the Medication list form is

saved. If the client did not have any Medication allergies or a minimum of "No known drug allergies”

recorded, then the notifications will be created and displayed indicating Medication allergies are not

verified.

Demographics - Order “HbA1C” lab test


if “Diabetes” is an active problem, then the application checks for the Lab Results. If there is no



“HbA1C” lab test in the Lab Results then a notification will be created and displayed indicating that

“HbA1C” lab test should be ordered for the client




Comments:


Comments:




Task 29 - Access CDS reference attributes including bibliographic citation, developer, funding source,

release/revision date


Success:



[ ] Not completed

Comments:


Optimal Path: Click on the CDS reference attributes in configuration including bibliographic citation,

developer, funding source, release/revision date for the following CDS




not been selected.









Comments:


Comments:




Task 30 - Record Implantable Devices UDI


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Implantable Devices Form Record a UDI Fill out rest of the data and click on

“OK”




Comments:


Comments:




Task 31 - Change Implantable Devices UDI Status


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Implantable Devices Form Search for an UDI Edit and change the UDI and

click on “OK”




Comments:


Comments:




Task 32 - Access UDI device description, identifiers, and attributes


Success:



[ ] Not completed

Comments:


Optimal Path: Click on Implantable Devices Form Search for an UDI Review UDI device

description, identifiers, and attributes




Comments:


Comments:




E. System Usability Scale

Source - http://www.usabilitynet.org/trump/documents/Suschapt.doc Strongly Strongly disagree agree 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

Documents

EHR Usability Test Report Cx360 Version 7 · E. System Usability Scale ... gender identity, birth sex, preferred language – A5 14. Access and Display Demographics - A5 15. ... Cx360