Melissa Baysari - Australian Institute of Health Innovation

Evaluation of computerised decision support in

eMed systems

Melissa Baysari

With Wu Yi Zheng, David Lowenstein, Anmol Sandhu, Ric Day, Johanna Westbrook, Rosemary Burke, Eliza Kenny & Meredith Makeham

Computerised decision support

2

Can mean different things to different people

Computerised alerts

Order sentences

Reference material

Drop down lists

Notes or instructions

Calculators

Etc

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION

FACULTY OF MEDICINE AND HEALTH SCIENCES

Alert effectiveness

3 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION


Literature tells us that alerts can result in substantial changes in

prescribing behaviour

BUT

Most studies evaluate an alert for a specific condition or problem

e.g. alerts designed to reduce the use of contraindicated drugs in

patients with renal failure drop in proportion of patients receiving a

contraindicated medication from 89% to 47%

Less evidence for the effectiveness of basic decision support

alerts within eMM systems

e.g. few studies showing that DDI alerts lead to reductions in DDIs

JAMIA 2005 12:269-74

Alert fatigue



A consequence of too many alerts being presented

Main barrier to prescriber acceptance of computerised alerts

A significant problem for hospitals because it

results in user frustration & annoyance

leads to prescribers learning to ignore all alerts, even those that present

useful & sometimes safety critical information

Alert fatigue affects most doctors in most organisations

most alerts are overridden

Effective alerting



Many strategies have been proposed for reducing number of alerts (and

minimising alert fatigue), such as:

- Customising alerts for clinicians

- Increasing alert specificity

- Presenting only high-level (severe) alerts to clinicians

- Improving alert design

These strategies sound simple, but are very difficult to implement

Effective warning design From the human factors literature (process industries etc)

Alert content



Message

length

Short messages are easier and faster to process.

As a general rule, do not include more than 5-6 bits of

information

Abbreviations Avoid using, unless they are understood even by the most

inexperienced users

Procedures Break procedures up into short, sequential steps, with each

step presented on a separate line

Symbols Should be used wherever possible to avoid the need to read

text.

Only familiar or standard symbols should be used

Wording Affects comprehension and speed of reading.

If possible, use familiar words, active sentences, and positive

statements

Warning signs



These convey information about a potential hazard or risk

Include 4 main components:

• A signal word, larger than the rest of the alert text

• A description of the hazard

• A description of the consequence of the hazard

• A description of the behaviour needed to avoid the hazard

Examples



Messages should be structured in short, concise statements using active

verbs

WARNING

Surface is slippery when wet

May cause you to fall

Please use hand-rail

WARNING This document failed to save

You will lose all changes

To save, click OK

ALERT

Patient is allergic to penicillin

Continuing with the order may

cause anaphylactic shock

Click here to cancel order

Click here to continue order

Layout of the message



People read warnings in an unstructured way – they scan to identify

objects or words of interest

If large amounts of information need to be displayed, group into smaller

units

Short messages could be centred but longer messages (across multiple

lines) should be displayed on numbered lines and left justified

Good vs. bad warnings



This is an emergency. To make an emergency

call, press the red button. Wait for emergency

services to answer and then speak clearly into

the speaker.

EMERGENCY

To make an emergency call:

1. Press the red button

2. Wait for emergency services to answer

3. Speak clearly into speaker

Appearance of text



• If message is more than 3 words in length, use mixed upper and lower

case

• Short messages (ALERT, WARNING) can be written in capital letters

• Minimise use of italics, they are harder to read

• Colour combinations that have good contrast are easier to read

• Use colours that are expected (e.g. red for warning)

Phansalkar’s review and HF tool

Human factors principles for design and implementation of medication

safety alerts

2010 review of the literature



Aim: to summarise human factors research on computer-based alerting

systems from a range of industries

They identified 11 HF principles

Alarm philosophy Colour

False alarms Learnability and confusability

Placement Textual information

Visibility Habituation

Prioritization Mental models

Proximity of task components

How to measure HF compliance?



I-MeDeSA - Instrument for evaluating human factors principles in

medication related decision support alerts

Tool measures compliance to 9 HF principles

Composed of 26 items with binary scoring (0 or 1)

Development and validation (for DDI alerts):

• Items were created for the quantifiable HF principles from the review

• 3 HF experts reviewed, modified and eliminated items

• 3 reviewers tested the items on their EMR systems

• 2 reviewers evaluated the same DDI alerts and IRR was assessed (k=0.76)

• Validity assessed – correlated performance of DDI alerts on I-MeDeSA with

age of the system

Example items



HF principle Item

Placement Is the alert linked with the medication order by

appropriate timing? (i.e. a DDI alert appears as soon as a

drug is chosen)

Visibility Is the area where the alert is located distinguishable from

the rest of the screen?

This might be achieved through the use of a different

background colour, a border, highlighting, bold

characters, occupying the majority of the screen etc

Applications of I-MeDeSA



Used to assess HF compliance of DDI alerts in 14 EHRs in the US

Used to assess DDI alerts in a large hospital in Korea

Not been used in Australia

Our study

Aims



1. To compare DDI alert interfaces in 8 electronic systems currently

used in Australia, in terms of their compliance with HF principles

1. To identify any potential problems with using I-MeDeSA

1. To determine whether alerts which are more compliant with HF

principles are also viewed more favourably by doctors and

pharmacists

Method – Part 1



Systems evaluated:

• To generate DDI alerts for evaluation, major DDIs were prescribed in the

training/test versions of each system

• 3 reviewers independently evaluated each DDI alert, then met to reach a

consensus on a score for each system

• A screen-shot of 1 alert (allopurinal + azathioprine) was taken from each

system for Part 2

Cerner PowerChart TrakCare

MedChart Medical Director

iPharmacy Best Practice

MOSAIQ FRED

Modified scoring



• During piloting, we found that scoring ‘yes’ for a number of I-MeDeSA

items was dependent on scoring ‘yes’ for the preceding item

• Is the prioritization of alerts indicated by appropriate colour?

• Does the alert use prioritization with colours other than red and green

to take into consideration users who may be colour blind?

• We also found that some items only applied to systems with multiple

levels of alerts in place

• Ten items were excluded to create a modified scoring system

Part 2 – user survey (n = 45)



• Which electronic prescribing system or medication dispensing

system are you currently using or have you used in the past?

• How long have you been using electronic prescribing or

electronic medication dispensing systems?

• On average, how many DDI alerts do you experience in a day?

• Please review the 9 DDI alerts below and rank the alert

interfaces from best to worst (1=best, 9=worst) using the table

below. Please also tell us what you like or dislike about the alerts

(e.g. the alert text was short).







Compliant alert



Results Data collection and analysis is ongoing

Main lessons learnt



• HF compliance was poor across the 8 systems

• Large variation in the design of DDI alert interfaces

• Non significant correlation between I-MeDeSA assessment

scores and user preferences

BUT

• Our ‘compliant’ alert was most preferred – this suggests that

the HF principles in I-MeDeSA are sound

• Lots of problems with I-MeDeSA including subjective items,

dependent items, arbitrary scoring (weights) assigned to each

HF principle

Where to now?



• Finalise survey data collection and analysis

• Explore better ways to assess alert design:

• Working with our colleagues in Lille who have developed an

evidence-based framework linking alert usability principles to

usability flaws and usage problems

Thank you

This research was supported by NHMRC Program grant

1054146

Contact: [email protected]

mailto:[email protected]

Healthcare

Melissa Baysari - Australian Institute of Health Innovation