Usability Sample

Client Name Product name

BLOOD ANALYZER FINDINGS November 2011


The goal of this project was to evaluate the usability and desirability of a new piece of lab equipment for emergency use. This presentation was given to upper management to focus future development of this product.

2


PROJECT OVERVIEW KEY FINDINGS DETAILED FINDINGS SCANING PACKAGING TRANSFER/INNOCULATION INSERTION & REMOVAL PRINTING RESULTS & FEEDBACK

3


PROJECT OVERVIEW PROJECT GOALS PARTICIPANT OVERVIEW SESSION OVERVIEW

4

Client Name Product name 5

Evaluate responses and interactions of targeted Point Of Care users in association with the current PRODUCT; attention will be focused on (but not limited to): •Ease of use •Ergonomics •Fit with current workflow •Evaluation of system workflow •System size

Evaluate viability of the current system and system interaction scheme. Specifically packaging, card handling, inoculation, interface between card and instrument, and removal of the card.

While it was not in the original scope of the project, we noted issues regarding the general flow of the user interface within the context of this study. This study was not intended to provide feedback on the UI so findings regarding UI will be kept at a very high-level. However, user feedback suggests that the UI is not finalized and will need further review.

PROJECT OVERVIEW-GOALS


Participant overview infographic

PROJECT OVERVIEW- PARTICIPANT PROFILE

*Full profiles of each participant can be found in the appendix

Institution Had more than 250 beds and were considered acute care hospitals with emergency departments

Nurses Worked in an Emergency Department in the last two years. Half had experience performing cardiac marker testing themselves, while the other half did not Nurses had a mix of experience level

Lab Techs Had at least 3 years of experience working in a core or stat lab

POC coordinators Managed ED nurses or stat lab techs Half work in hospitals with cardiac marker testing in the ED

All Participants Had a mix of age, gender, glove size, & hand dominance


Introduction Participants discussed the types of tests that they are familiar with at point of care.

7

PROJECT OVERVIEW- SESSION OVERVIEW

Training Participants were shown a 3 minute training video which covered basic workflow, inoculation techniques, and system features. This training insured that all participants had the same base line knowledge of the process.

Initial Impressions Participants then discussed their initial impressions of the concept based on the video.

Trials Participants completed a number of different trials to gain familiarity with the equipment. These trials helped participants articulate what worked and didn’t work with the system. After the Chicago sessions the trials were changed slightly to reflect product improvements made between cities.


Workflow Participants evaluated the proposed workflow then created their own ideal workflow using stimuli cards.

8

PROJECT OVERVIEW- SESSION OVERVIEW

Evaluation Participants evaluated each part of the process on a 7 point Lickertt scale.

Preference Participants were asked their preference with regards to the features that have multiple go-to-market options like transfer device, scanner, etc.

Interview Participants “summed up” each part of the system and discussed on the changes they would make.

Feature Sort Participants sorted a list of features from most important to least important.


HIGH-LEVEL FINDINGS KEY TAKE AWAYS INITIAL IMPRESSIONS ERRORS USER RATINGS WORKFLOW CHALLENGES VALUE PROPOSITION

9


We thought:

HIGH LEVEL FINDINGS- KEY FINDINGS Now we know:

Participants would be comfortable with opening a vacutainer and using a transfer pipette

Participants may be unhappy with the insertion and removal process

The PRODUCT would be accepted as a centralized system

Inoculation- specifically opening a vacutainer to use a transfer pipette is inherently “scary” to nurses, however the Closed Tube Sampler (CTS) was well received.

Insertion and removal is acceptable to users.

Nurses are currently comfortable conducting some point of care tests bedside, and might expect to use the PRODUCT in the patient’s room.

Participants who were not currently running tests at bedside saw no problem with the size of the analyzer, and mentioned that the would find space for it.

The size of the analyzer might be a concern for users.


Overall the PRODUCT was well received and easy to use by all participants. This graphic shows the frequency of positive concepts mentions when participants were asked their initial impressions of the product. The size of the words correlate to the number of times each concept was mentioned positively.

HIGH LEVEL FINDINGS- POSITIVE INITIAL IMPRESSIONS

Concept Mentions easy to use 12

3 bays 11 time 11 barcode scanning 8

QC 8 CTS 7 durable 3

accuracy 2 Data management 2

whole blood 2 big screen 1 familiar (similar to istat) 1

flexibility in transfer options 1 high quality 1 print with out docking 1

uses cartridges 1


HIGH LEVEL FINDINGS- NEGATIVE INITIAL IMPRESSIONS This graphic shows the frequency of negative concepts mentions when participants were asked their initial impressions of the product. The size of the words correlate to the number of times each concept was mentioned negatively.

Concept Mentions not bedside 8

transfer method 7 size 4 overfill 3

time 3 availability 2 cost 1

might not work in a helicopter or ambulance 1 waste 1

• Initial impressions were generally more positive than negative • Negative mentions of “Size” and “Not Bedside” were more popular with current I-stat users

who were performing cardiac testing at bedside [ATLANTA] • Transfer method (Transfer pipette and CTS) was the most frequently mentioned concept. • Time was mentioned both positively and negatively. Currently I-Stat tests take 10 minutes but

lab results may take up to 90 minutes.


In total participants completed 180 trials, of those 40 encountered at least one error. This number is relatively low and shows us that participants were capable of easily learning the system. Errors included: Scan in wrong order Slight under fill or overfill Threw away pouch before scanning Insert into wrong bay orientation issues during insertion Touched lenses Forgot to hit go Open exterior box from top

HIGH LEVEL FINDINGS- USER ERRORS Percentage of Errors for user trials

Errors occurring in trials 1-3

19%

Trials with no errors

76%

Errors occurring in trials 4-6*

5%

*Most of these errors were- participant threw away package before scanning


This chart shows a breakdown of the 40 errors.

HIGH LEVEL FINDINGS- USER ERRORS

8% 5% 5% 5% 2% 1% 1% 1% 0%

10%20%30%40%50%60%70%80%90%

100%

Percentage of Errors for User Trials

*error only applicable to Chicago trials. Barcode was added to the cartridge for Atlanta

**error only applicable to Atlanta trials. Interface was improved to direct users to specific bays for Atlanta


This chart shows a breakdown of the 40 errors.


8% 5% 5% 5% 2% 1% 1% 1% 0%

10%20%30%40%50%60%70%80%90%

100%




Happened infrequently and would be alleviated with proper training

Only happened with transfer pipette

Addressed by adding a barcode to the cartridge


Of these errors, “scan in the wrong order” and “insert into the wrong bay” are the most severe.




8% 5% 5% 5% 2% 1% 1% 1% 0%

10%20%30%40%50%60%70%80%90%

100%Did not happen on any of the first trials, meaning that confidence in using the system lead to this error

This is a critical error that can be overcome with the proper affordances.



Ver

y Ea

sy

Ver

y D

iffi

cult

When participants were asked to rate the difficulty of each part of the process, opening the exterior box and pouch rated the lowest. These parts of the process are not severe and would not cause safety issues, but greatly impact user experience.

HIGH LEVEL FINDINGS- USER RATINGS

• USER ERRORS and factors affecting USER EXPERIENCE are both critical to understand when making design changes.


We also asked participants to what extent they agree or disagree with the following statements. This graphic shows and averaged rank of the results.

HIGH LEVEL FINDINGS- USER RATINGS

• Focus on the design and look of the product, the user-interface, and the overall footprint of the system.

Strongly Disagree Strongly Agree


HIGH LEVEL FINDINGS- WORKFLOW CHALLENGES

SCANNING ORDER BEDSIDE VS.

CENTRALIZED MULTIPLE CARTRIDGE

1 2 3


HIGH LEVEL FINDINGS- CURRENT WORKFLOW Below is the current product workflow: Take patient blood Press new test Scan user ID Scan patient/specimen ID Scan cartridge or cartridge pouch Open pouch Inoculate Insert cartridge into the analyzer


Bedside vs. Centralized Nurses who are familiar with bedside testing would like to have most, or all of, the workflow happen at bedside. The benefit of a bedside workflow: • The nurse would not have to leave the patient during critical times • The nurse could not possibly use the wrong patient’s blood to run a test • The nurse could teach the patient about the test, what the results

mean, and show that the hospital cares because they have the most advanced equipment

HIGH LEVEL FINDINGS- WORKFLOW CHALLENGES Some nurses mentioned that they would want analyzers in every room. While putting the current system on a cart may be a short term solution, there are a lot of core challenges with equipment on carts. Nurses seemed receptive to doing scanning and inoculation in the patient room and moving to a centralized location to insert the cartridge, but they would really prefer not to leave the room.

•Consider finding ways to make a bedside workflow possible


Scanning Order The biggest challenge with the current workflow is scanning all of the items in the proper order. While all of the user groups expected to scan 1. a user identifier, 2. a patient identifier, and 3. the cartridge/pouch, when they started moving fast they would scan in the wrong order or skip a scan.


•Consider creating a system that is order agnostic

VS

1

2

3


Multiple Tests We asked participants to take us through the workflow they would expect for running multiple tests on the same patient and while the details varied there were two key themes:


•Offer hospitals the flexibility to dictate how a multi-cartridge workflow will work for them

Safety- some participants wanted to go through the entire process including all scanning for each test. They saw this as the safest way to run multiple tests.

Efficiency- some participants wanted to do this process as efficiently as possible. Scanning their id and the patient identifier one time and opening, inoculating, scanning, and inserting all three cartridges together.


HIGH LEVEL FINDINGS- VALUE PROPOSITION

Quality & Reliability of

the instruments & reagents

Advanced, highly

automated analyzers

with lab-like results

Amount of time each test takes

Breadth of automated

assays

Technical Support and

Service

Cost of the reagents

Cost of the instrument

Supplier experience

with point of care testing

Most Important Least Important

Participants were asked to sort this list of attributes. This order is an averaged score of all users. The order was consistent between user groups and geographies.


POC Coordinators

Hospital Board

Nurses

Easy to Manage and Maintain To sell to this group talk about: •Quality control •Inventory control •Speed •Easy of use = lack of errors •Wave certification (if applicable)

Easy to Use To sell to this group talk about: •Better/faster patient results •Easy to use = more control over your patient’s care with out the hassle of dealing with the lab

Strong ROI To sell to this group talk about: •Improved results lead to better patient outcomes and lower risk •Faster turn around time leads to higher patient satisfaction •A modular system makes expansions and replacements faster and less expensive.

HIGH LEVEL FINDINGS- VALUE PROPOSITION Participants talked candidly about how a product like this would be introduced to their hospital

• In order to meet the needs of this diverse group, flexibility is key


DETAILED FINDINGS SCANING PACKAGING TRANSFER/INNOCULATION INSERTION & REMOVAL PRINTING RESULTS & FEEDBACK USER INTERFACE

26


“It’s sturdier in the machine but I liked the handheld thing because it picked up quick. My nurses are the best in the world but they are hard on things” -Victoria

“I like scanning the cartridge before I add the blood to it so scanning with the integrated scanner works for me. Scanning with the handheld scanner would not be my preference but to me it was very easy. It seemed that one time I had to move everything around. “ – Wayne

SCANING- FEEDBACK

27

In general using the scanner was…

Very difficult Very easy

“The scanner I would prefer is…”


SCANING- TAKE-AWAYS

28

• Participants were comfortable using a scanner. Scanning is a familiar interaction, however there were a number of errors caused by participants scanning items in the incorrect order.

• The current integrated scanner is difficult to use because it does not offer perceptual feedback

• The green light is hard to see when using the integrated scanner

• Some participants worried about touching the external scanner for fear that it was dirty.

• The cartridge pouch was difficult to scan using the integrated scanner, especially after the cartridge had been removed.

• Some participants disposed of the cartridge pouch before scanning during the inoculate first trial.

• Nurses like the idea of moving part of the workflow to the patient’s bedside •This is the current practice with i-stat •Scanning the patient wristband (instead of the

specimen) reduces the possibility of identification error

• Consider making the scanning workflow more flexible

• Some hospitals will want the option to manually enter patient ID for John Doe situations.


SCANING- HIGH LEVEL RECOMMENDATIONS

29

Short Term Mid Term Long Term

• Offer an external wired scanner with a cradle as an add on option.

• Explore ways to make the scanning UI more flexible.

• Adjust the internal scanner so users can gain visual feedback.

•Develop a bedside workflow using wireless technology


"I could see nurses stocking the cartridges in their IV tray" -Brittany

“I think it could be designed a little bit differently. It wasn't hard, but it wasn't as simple as it could be“ – Susan

EXTERIOR BOX- FEEDBACK

30

Opening the Exterior Box was…


“The exterior box I would prefer is…”

"I am not a big fan of the size of the opening“ – Erin


EXTERIOR BOX- TAKE-AWAYS

31

• Participants understood how to open the box, but mentioned that they might want more flexibility.

• Opening the box was clear. • Some participants wanted a visual indicator of how

many pouches remain in each box • Some participants would still want to write the

expiration date on the box, even if they are using inventory management.

• Some participants had a hard time ripping the perforations

• Removing a cartridge pouch required 2 hands • The current box does not facilitate a multiple

cartridge “Grab”

• Flexibility is key • Inventory management is a game changing feature

that is best supported by the T-slot box, however the T slot has the perception of being “too fussy” for the grab and dump ED

• The product will not always be used with inventory management or in high volumes making the t-slot a hindrance for removing multiple cartridges at the same time

• The wire rack is a good add on option


EXTERIOR BOX- HIGH LEVEL RECOMMENDATIONS

32


• Create a clearer/easier to pull tab for the T-slot.

• Use tighter perforation • Create a graphic area for users to

write the use-by date

• *Offer an improved version of the wire rack system as an optional feature/add-on

•Create a multi-open box

• Create a visual indicator for the number of cartridges remaining in the package


“The first one I went to open I didn’t hit the little notch as well. You have to yank it harder. It would be nice if there was a little zip thing. I could be simpler. It would be nice to have a little arrow showing where the notch is. “ – Wayne

REMOVE & OPEN POUCH-FEEDBACK

33

“It’s a little tough.” -Dana

"once you get it open you still have to wrestle it“ – Sandra

"It could have a little bit more of a tear. If there was room for it, just to kind of get you going a little bit better. It was evident that you pull here where the black dot is, and it pulled pretty easy because I am not that strong. “- Crystal


REMOVE & OPEN POUCH-FEEDBACK

34

Removing the pouch was… Very difficult Very easy

Opening the pouch was…

Removing the cartridge from the pouch was… Very difficult Very easy


REMOVE & OPEN POUCH-TAKE-AWAYS

35

• There were no cognitive issues with removing or opening the pouch.

• Removing the pouch is clear • There could be clearer indicators for where to open

the pouch • Participants may still hand write the date on the

pouch and would like an area to do that.

• A slightly larger T-slot may make removing pouches easier

• Tearing the pouch open was incredibly difficult • Participants used the pouch to handle the cartridge • Some participants expressed worries about the

desiccant pack.

• Participants were all able to discern where and how to open the pouch, they struggled with the physical force required to open

• Worried that opening the pouch too soon would reduce the accuracy of the test.

• The prototype packages had 2 barcodes which was confusing to some participants

• Some participants mentioned that the package seemed a little bit bigger than it needs to be.


REMOVE & OPEN POUCH- HIGH LEVEL RECOMMENDATIONS

36


• Slightly larger slot on external box

• More obvious graphic cues for where to open pouch

• Graphic area for marking expiration date.

• Easier to tear pouch with deeper notches and a more forgiving liner material.

• Automatic visual indicator on pouch when time at room temperature has been exceeded.


CARTRIDGE-FEEDBACK

37

"I wonder if there is an easier way, like a glucose test where you just prick the finger.” – Brittany

"I like that you have the option to write in patient ID, but you don't really have to" -Sonia


CARTRIDGE-TAKE-AWAYS

38

• Cartridge orientation was challenging for some participants. • Touch/no-touch areas were intuitive

• Graphics on the cartridge did not help participants determine how the cartridge should be oriented during transfer and insertion

• The sight port was well received as a concept but challenging for participants to actually see because the user’s hand blocks the line of sight.

• Participants liked the area to write patient information.

• All of the participants used the handle correctly • Participants mentioned that having the barcode on

the cartridge itself was a great “fail-safe” in case they threw the pouch away.

• Almost all of the participants held the cartridge using their thumb when inserting

• Some noted that the bottom of the cartridge was uncomfortable.

• Participants looked at the cartridge after the test for some indication that the test had been completed.


CARTRIDGE-HIGH LEVEL RECOMMENDATIONS

39


• Add arrows to assist with orientation

• Add 3 blood drop graphic

• Consider ways to make the cartridge visually different after it has been used.

• Consider moving the sight port slightly to make it more visible during inoculation.


I really like the fact that you don’t have to pull back and push. I like the fact that you just put it up against and push down. But the other way I just see biohazard everywhere and infection Insert- very easier, in fact easier that i-stat -Victoria

It is challenging until you get training. After I did it once it was easy. There is something about the word inoculate. I didn’t like it up there. I knew what you meant, but there has to be a better way to say it. – Wayne

TRANSFER-FEEDBACK

40

In general transferring blood was…


“The transfer method I would prefer is…”


TRANSFER-TAKE-AWAYS

41

• The word ”inoculate“ was confusing or inappropriate for users • There was a general fear surrounding opening

vacutainers or handling used CTS devices, especially among nurses.

• Again the sight port was hard to see for some participants because their hands or the vacutainer were in the way

• One participant suggested adding an icon to the cartridge to show 3 blood drops (the amount she thought was the appropriate fill).

• In a couple of cases the CTS broke or did not properly release the vacuum

• The blunt end of the CTS made it challenging to insert.

• Overcome the fear of transferring by offering a easy to use and safe closed tube sampler.


TRANSFER-HIGH LEVEL RECOMMENDATIONS

42

Short Term Mid Term

•On the interface change the word inoculate to transfer

•Improve the CTS by making it: • Safer •A tighter fit with the cartridge • Easier to use (requiring less

physical strength)

Transfer


“Initially you couldn't exactly see where it was going, but once you got it, it was simple“ -Susan

“The very first time, I didn’t know if it had to go at an angle because when the door was open it looked like it more go at an angle. Once I knew how to do it, it was extremely simple. But again that goes in training. If they have been shown how to do it, it becomes a 7 for me-” Wayne

INSERT & REMOVE-FEEDBACK

43

Inserting the Cartridge was…


“The shutter I preferred for Insertion was…”

“The shutter I preferred to Close was…”

Removing the Cartridge was…


INSERT & REMOVE-TAKE-AWAYS

44

• There were a few small orientation errors with the first insertion. • Some participants tried to insert the cartridge above

the handle for the shutter. • In a few cases participants tried to insert the

cartridge into a full slot • The biggest cognitive challenge with insertion is

determining which port to use.

• The angle positioning of the ports makes them difficult to see. Most users bent down for their first insertion but did not look at the ports again for subsequent insertions.

• None of the participants fingers touched the sides or shutters when inserting and removing.

• Most users had to bend down to insert the first time but noted that they would be trained and not need to bend down.

• Insertion and removal was physically easier than expected, but cognitively more challenging than expected. The active bay must be clearer.


INSERT & REMOVE-HIGH LEVEL RECOMMENDATIONS

45


• Create affordances on the interface to make correct bay more obvious

• Consider adding soft lighting inside the bays

• Create a bay labeling system • Add design details that protrude from the bay to show users where the cartridge should go.


PRINTING-TAKE-AWAYS

46

• Participants expected to have some means of printing, but did not intend on printing every result. • They would like to see 2 patient identifiers as well as

result ranges on the print out

• Participants did not notice that there was a printer onboard

• They would rather have the print out come out of the side or top of the unit

• Some shorter participants had to work to reach the print out in the current location.

“The printer I would prefer is…”

• Users will need the capability to print, but will probably not print every result.

• Thermal paper is considered temporary and can therefore not be a part of a patients chart, it must be taped and photocopied.

• The current location of the integrated printer is challenging. Consider offering an external printer which “fits” with the system.


INVENTORY MANAGEMENT

47

• Most participants reacted very positively to the idea of inventory management

• However, explaining what inventory management is, and how it works was challenging.

• This could be a game changing feature, but it will require a well thought out sales approach.

• Some participants worried that they would still need a visual indicator on the package showing room temperature expiration.


RESULTS & FEEDBACK

48

Results •Currently the I-stat only lets the user see the

last result, so the ability to go back and see past results is a value add. • This also means that there isn’t really a

convention for dealing with previous results (search by operator, search by patient, etc). •Currently medical record systems in hospitals a

incredibly complicated ranging from paper medical records to multiple OS’s per hospital. • There was some interest in instantly updating

doctors with critical values, but feasibility may be challenging right now.

Feedback •Participants were interested in visual

differentiation of the interface for critical values. •Optional auditory feedback would be well

received for critical values and errors. •Color uses- when possible follow human factors

standard regarding alert colors.


USER INTERFACE-HIGH LEVEL TAKEAWAYS

49

SCANNING NEW TEST BAY ALIGNMENT


USER INTERFACE-NEW TEST

50

• Remembering to press new test was consistently challenging for users. • When creating workflows this was the one step that felt confusing or unnecessary. • Consider adding a “next test” screen for users who are running multiple tests on the same

patient.

Difficult to see

Participants naturally touched the green area to start a test.


USER INTERFACE-SCANNING

51

• Participants were comfortable with the idea of scanning three items, but as they became more confident with the system they were more likely to scan items out of order. There is no clear mental model to help users remember the prescribed order of scanning.


USER INTERFACE- BAY ALIGNMENT

52

• Bay alignment was one of the most challenging parts of the interface. Brainstorm ways to make bay alignment clearer and more connected to the physical analyzer.


SUMMARY

53

Inoculation- specifically opening a vacutainer to use a transfer pipette is inherently “scary” to nurses, however the Closed Tube Sampler (CTS) was well received.

Insertion and removal is acceptable to users.

Nurses are currently comfortable conducting some point of care tests bedside, and might expect to use the PRODUCT in the patient’s room.

Participants who were not currently running tests at bedside saw no problem with the size of the analyzer, and mentioned that the would find space for it.

Documents

Usability Sample