Validation and Re ﬁnementofaPainInformation Model from …...Validation and ReﬁnementofaPainInformation Model from EHR Flowsheet Data Bonnie L. Westra1 Steven G. Johnson1 Samira

Validation and Refinement of a Pain InformationModel from EHR Flowsheet DataBonnie L. Westra1 Steven G. Johnson1 Samira Ali2 Karen M. Bavuso3 Christopher A. Cruz4

Sarah Collins5 Meg Furukawa6 Mary L. Hook7 Anne LaFlamme8 Kay Lytle9 Lisiane Pruinelli1

Tari Rajchel10 Theresa (Tess) Settergren11 Kathryn F. Westman12 Luann Whittenburg13

1School of Nursing, University of Minnesota, Minneapolis,Minnesota, United States

2Adjunct Faculty, School of Nursing, Grand Canyon University,Phoenix, Arizona, United States

3Department of Clinical Informatics, Partners Healthcare System,Wellesley, Massachusetts, United States

4Department of Care Delivery BioClinical Informatics, KaiserPermanente, Oakland, California, United States

5Department of Medicine, Brigham and Women’s Hospital, PartnersHealthcare System, Harvard Medical School, Boston,Massachusetts, United States

6Department of Information Services & Solutions, University ofCalifornia Los Angeles Health, Los Angeles, California, United States

7Center for Nursing Practice and Research, Aurora Health Care,Milwaukee, Wisconsin, United States

8Department of Nursing, Fairview Health Services, Minneapolis,Minnesota, United States

9Department of Nursing & Duke Health Technology Solutions, DukeUniversity Health System, Durham, North Carolina, United States

10Department of Information Technology, North Memorial MedicalCenter, Robbinsdale, Minnesota, United States

11Department of Enterprise Information Services, Cedars-SinaiHealth System, Los Angeles, California, United States

12Department of Nursing, Allina Health, Minneapolis, Minnesota,United States

13 Informatics Consultant, Bumrungrad International, HealthInformatics/Health System Architecture, Bangkok, Thailand

Appl Clin Inform 2018;9:185–198.

Address for correspondence Bonnie L. Westra, PhD, RN, FAAN, FACMI,School of Nursing, University of Minnesota, 308 Harvard Street SE,Minneapolis, MN 55455, United States (e-mail: [email protected]).

Keywords

► pain management► nursing informatics► electronic health

records and systems► knowledge modeling

and representation► secondary use► efficiency

improvement

Abstract Background Secondary use of electronic health record (EHR) data can reduce costs ofresearch and quality reporting. However, EHR data must be consistent within andacross organizations. Flowsheet data provide a rich source of interprofessional data andrepresents a high volume of documentation; however, content is not standardized.Health care organizations design and implement customized content for different careareas creating duplicative data that is noncomparable. In a prior study, 10 informationmodels (IMs) were derived from an EHR that included 2.4 million patients. There was aneed to evaluate the generalizability of the models across organizations. The pain IMwas selected for evaluation and refinement because pain is a commonly occurringproblem associated with high costs for pain management.Objective The purpose of our study was to validate and further refine a pain IM fromEHR flowsheet data that standardizes pain concepts, definitions, and associated valuesets for assessments, goals, interventions, and outcomes.Methods A retrospective observational study was conducted using an iterativeconsensus-based approach to map, analyze, and evaluate data from 10 organizations.Results The aggregated metadata from the EHRs of 8 large health care organizationsand the design build in 2 additional organizations represented flowsheet data from 6.6million patients, 27 million encounters, and 683 million observations. The final pain IM

receivedSeptember 14, 2017accepted after revisionJanuary 15, 2018

Copyright © 2018 Schattauer DOI https://doi.org/10.1055/s-0038-1636508.ISSN 1869-0327.

Research Article 185

Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

mailto:[email protected]

https://doi.org/10.1055/s-0038-1636508

https://doi.org/10.1055/s-0038-1636508

Background and Significance

Thewidespread implementation of electronic health records(EHRs) provides health care organizations the opportunity tocapture, use, and share data for evaluation, benchmarking,quality improvement, and research to improve the effective-ness, efficiency, and outcomes of patient care. Secondary useand sharing, however, requires data to be represented usingrecognized terminologies and descriptors that are consis-tent, understood, and effectively formatted for comparison.These requirements suggest that concepts must be standar-dized, formally modeled, and mapped into the EHR foroptimal use. An “information model” (IM) is an organizedstructure to represent knowledge about a clinical conditionor concept including data elements, their relationships, andthe data standards that are independent of implementationin EHRs.1 IMs can be mapped to EHR data to identifysemantic similarities2 and, more importantly, to enableresearchers to understand and normalize differences whenthey occur to improve data sharing.

The American Recovery and Reinvestment Act of 2009(ARRA) provided incentives for creating a national healthcare technology infrastructure and accelerating the adoptionand meaningful use of enterprise-wide, vendor-based EHRsystems with a key focus on physician-based data capture.Vendors provide generic content and guide organizations inusing consensus-based approaches to configure the bulk oftheir system to meet the clinical requirements of the orga-nization. Much of the documentation, however, is capturedin flowsheet format, using nonstandardized semistructureddata in a matrix format for patient assessments, goals,problems, interventions, and outcomes of care. Limitedresources and rapid deployment timelines provide littletime for organizations to identify and adopt standardizedterminologies and use IMs to design flowsheets for futuredata sharing. Further, informaticians are required to choosefrom multiple terminologies3,4 with limited reference stan-dards to guide flowsheet builds. These conditions alloworganizations to continue to design and implement custo-mized content, creating flowsheet rows (unique identifica-tions [IDs]) for different care areas, e.g., intensive care units,emergency departments, or medical–surgical units,5 withvaried choice options for documentation in flowsheet rowswith the same or similar names. As organizations movebeyond deployment, it is time to reevaluate the extensiveclinical information captured in flowsheets and considerhow to optimize and manage data better in the future.Furthermore, analyzing existing content may inform thedevelopment of standardized terminologies and IMs for

representing the essential nursing and interprofessionalassessments and interventions to achieve best patientoutcomes.

Nursing informatics leaders have successfully utilizedmethods for developing generalizable domain-specific IMsbased on documentation artifacts captured in the EHR.6,7

These investigators used consensus-based, data-drivenmethods for analyzing EHR data elements embedded bylarge multisite health care systems to develop a skin inspec-tion and pressure ulcer IM for standardizing and codingconcepts. Both groups identified that existing EHR systemscontain heterogeneous data with limited interoperability.They recommended ongoing efforts to create common IMsbased on best evidence, clinical expertise, and standardizedterminology beyond skin and pressure ulcer prevention.

Similarly, Westra et al5 utilized EHR data to develop aReference Information Model for the concept of pain. Theseresearchers selected the concept of pain because it is acommonly occurring problem, assessed and managed byall professional nurses and those who specialize in painmanagement.8 About 126 million, or more than half (56%)of adults in the United States, reported some level of painwithin a 3-month period.9 The estimated total nationaleconomic cost (direct and indirect) attributed to pain in2010 ranged from$560 to $635million.10 The concept of painremains an important aspect of hospital-based patient care,with additional regulatory focus on conducting pain assess-ments consistent with age, condition, and ability to under-stand, with an increased focus on patient involvement andthe effective use of nonpharmacological interventions.11 ThePain Reference IMwas developed by extracting themetadatafrom a clinical data repository (CDR) of one large integratedhealth care system representing over 2.4 million patients.The validation of the Pain Reference IM with other healthcare organizations was needed to increase the generaliz-ability of the model.

Objective

The purpose of our study was to validate and refine a PainReference IM fromEHR flowsheet data that standardizes painconcepts, definitions, and associated value sets for assess-ments, goals, interventions, and outcomes..

Methods

This study is a retrospective observational study using aniterative consensus-based approach to map, analyze, andevaluate EHR pain data across several organizations to

has 30 concepts, 4 panels (classes), and 396 value set items. Results are built on LogicalObservation Identifiers Names and Codes (LOINC) pain assessment terms and extendthe need for additional terms to support interoperability.Conclusion The resulting pain IM is a consensus model based on actual EHRdocumentation in the participating health systems. The IM captures the mostimportant concepts related to pain.

Applied Clinical Informatics Vol. 9 No. 1/2018

Pain Information Model Westra et al.186

Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

validate and refine the Pain Reference IM.5 A conveniencesample of nursing informatics researchers who were activein the Nursing Knowledge Big Data Science Initiative12 wereinvited to represent their organization as participants in thestudy. One researcher was a pain management specialist;others consulted pain experts in their organizations or painresources (i.e., pain society guidelines or studies).Theresearchers represented medium to large size multihospitalhealth care systems with the majority of the group using theEpic EHR (see ►Table 1). Of the 10 participating organiza-tions, 8 shared metadata for mapping their EHR to the PainReference IM and 2 additional organizations shared how theybuilt their systems since theywere just going live. The sharedmetadata included all flowsheet data, but only general painconcepts from inpatient and outpatient settings includingthe emergency department were analyzed for this project.Specialized cardiac/chest pain assessments were excluded asthe focus was on general pain.

Organizations were asked to extract metadata about theflowsheet documentation contained in their EHR. The meta-data consisted of a unique identifier for each flowsheet data

row representing assessments, interventions, goals, or out-comes; the internal description and name used to display theflowsheet row; the name of the template (data entry screen)that was used to collect the data (andwhich grouping of painconcepts within the screen); the number of observations,encounters, and patients; and the date of first and last uses.This metadata represented actual documentation by clini-cians at each organization.►Fig. 1 shows an example of howthe pain flowsheet data are documented and the relationshipto the metadata. Within each organization, the EHR datawere transferred to their Clarity relational database. AStructured Query Language (SQL) script was developedthat allowed each of the organizations to extract the meta-data in exactly the same manner. Based on organizations’resources for data extraction, there was variation in the timeframes selected and specific hospitals or practices includedin metadata extractions.

Each organization next mapped their metadata to theconcepts in the Pain Reference IM. This was accomplishedusing software (FloMap) that allowed the metadata to beimported from each organization. FloMap was developed by

Table 1 Data source for validation of the pain information model

Organization Organization type Data source Number of beds Datesrepresented bydata

Allina Health Hospitals, medical centers,clinics, rehabilitation, hospice,homecare, retail pharmacy

13 hospitals, 90þclinics

1,775 2005–2016

Aurora Health Care Private, not-for-profit, integratedhealth care system w/ 16 hospi-tals including behavioral health,rehab, and hospice

1 hospital; qua-ternary medicalcenter

710 CY 2016

Bumrungrad Interna-tional Hospitala

Hospital 1 hospital 580 2013–2016

Cedars Sinai Academic medical center andhealth system

1 hospital, 40clinics

886 2009–2016

Duke University HealthSystem

Health system 3 hospitals, 400clinics

1,512 2012–2016

Fairview Health Services Hospitals, academic health cen-ter, clinics, senior housing, retailpharmacy

7 hospitals,40 þ clinics

2,530 2011–2016

Kaiser Permanente Health system, hospitals, aca-demic hospitals (graduate medi-cal education), clinics,ambulatory care centers, acuterehab, inpatient psychiatry

Northern Califor-nia region only: 21hospitals, 233medical officebuildings, 203ambulatory carecenters

3,922 2005–2016

North Memorial MedicalCenter

Hospitals, specialty and primarycare clinics, home care, medicaltransportation

2 hospitals 355 2016

Partners Healthcarea Integrated health system 9 hospitals, manyclinics

2,825 2016

UCLA Health Health system 4 hospitals 861 2013–2016

Abbreviations: CY, calendar year; EHR, electronic health record; UCLA, University of California, Los Angeles.aOrganizations that provided information about their EHR build only.


Pain Information Model Westra et al. 187

Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

one of the researchers (S.J.) and is not currently publicallyavailable. A researcher from each organization used FloMapto search for pain-related flowsheet rows in their organiza-tion’s metadata and map them to the appropriate concept inthe Pain Reference IM. FloMap allows sophisticated search-ing using Boolean logic to make it easy to find local data thatmatched the pain concepts. Flowsheet rows related to exclu-sion criteria (i.e., cardiac/chest pain) or rows that had lessthan 10 observations were not mapped. ►Fig. 2A demon-strates the mapping process. This example shows howFloMap finds all flowsheet rows that contain “pain” andone of the additional terms. Users can see the value setswhich help to determine if the flowsheet rows represent asimilar concept. They then select the flowsheet rows thatmap to the concept and click on “Add items to concept.”After

these flowsheet rows are added to the concept, they aredisplayed and included in reports for comparison(see ►Fig. 2B).

After the local flowsheet dataweremapped to concepts inthe Pain Reference IM, the group met biweekly to evaluatethe concept mappings across all of the organizations. AFloMap-generated report was used by the group to makedecisions about which concepts to keep, combine, remove, oradd additional concepts. Concepts were retained when allresearchers agreed that the concepts represented essentialquestions for the majority of patients. One researcher was apain management specialist; others consulted pain expertsin their organizations or pain resources (i.e., pain societyguidelines or studies). There was discussion that there aresome differences in use based on the population such as age

Fig. 1 Example of documenting pain on flowsheets. The orange template is a screen view that shows the Adult Assessment which includesmultiple groups of related questions shown in light green on the left. The group called “Pain” shows examples of specific questions/flowsheetmeasures displayed to the clinician. The clinician selects answers from the value sets with actual documentation shown in blue fordocumentation that occurred at specific dates/times.



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

(pediatric vs. adult), type of unit (e.g., intensive care unit vs. amedical–surgical unit), or the patient’s capability (e.g., abil-ity to verbalize pain). The group developed a definition anddiscussed the use of the concepts to help determine decisionsabout the concept and associated value sets. A value setrepresents the list of all possible values (answers) associatedwith a specific concept (question). Value set response countsvaried by concept and ranged from a few (3–4) to many(> 100). For example, the concept of “Body Site” had 507different response choices across the 10 organizations. Someresponse values were not useful (e.g., misspelled or incom-plete words like “a,” “ac,” “acu,” “acut” for a value choice of“acute”) or clearly inappropriate such as “…,” “/,” “ þ þ þ,”etc. After the inappropriate responseswere removed, severalconcepts with multiple diverse value sets remained forevaluation.

To support the group in evaluating diverse responsevalues, a FloMap “survey” feature was developed. The Flo-Map survey aggregated all of the response values for aparticular concept into a single list while retaining detailsabout which organizations used each choice. Researchersfrom each organization received a survey via email with 1 to2 concepts and a list of response values set choices for eachconcept from all of the organizations. The email contained asecure link to the survey. Survey participants were asked toselect values that were considered generalizable acrossorganizations even if their organization did not currentlyinclude that value. The results were then discussed at thebiweekly calls. Value set items that received 50% or more ofthe votes were automatically retained in the Pain ReferenceIM. Those items that received less than 30% were automati-

cally removed from themodel and those between 30 and 50%were discussed by the group. The group decided on thesethresholds to reduce the amount of discussion needed toreach consensus. The results were compared with painconcepts included in Logical Observation Identifiers Namesand Codes (LOINC). Some concepts were then renamed tomatch those in the Nursing Physiologic Assessment Panel inLOINC or other LOINC locations.

Results

The aggregate metadata from 8 large health care organiza-tions that contributed metadata represented flowsheet datafrom 6 million patients, 27 million encounters, and 683million observations. A high level diagram of the resultingpain IM concepts is shown in ►Fig. 3; the red font indicatesnew panels and concepts added. ►Table 2 shows a compar-ison of the original Pain Reference IM and final consensusregarding which concepts were retained with or withoutrevision, removed, or added. The new model consists of 30concepts grouped into 4 panels with 396 value set items. Thein-depth analysis revealed that 24 concepts were retained, 6added, and 59 removed compared with the concepts in theoriginal Pain Reference IM. Since some scales require copy-right permission to use, we retained only the scale score foreach of the pain scales for consistency. The SupplementalDigital Content (SCD) 1 includes a detailed list of the retainedconcepts, definitions, and their value sets. ►Table 3 lists theinformation for each concept that was part of the finalmodel: the minimum (Min) and maximum (Max) numberof flowsheet rows per organization mapped to a concept as

Fig. 2 (A) Example of Boolean searching FloMap for mapping flowsheet rows to “Factors that Aggravate Pain.” (B) Display of flowsheet measuresmapped to the concept of “Factors that Aggravate Pain.”



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

well as the average (Avg) number of flowsheet rows acrossorganizations. On average, organizations mapped 9 flow-sheet rows to a single concept in the model. In fact, oneorganization had 81 unique flowsheet rows for recording“Numeric Pain Rating 0–10 Score.” ►Table 3 also includesstatistics for the percent of organizations using a particularconcept, the number and percent of patients for which aconcept was documented, and the total number of observa-tions documented. Some concepts, such as “Numeric PainRating 0–10 Score” are documented on 100% of patients.Finally, ►Table 3 also includes the number of value setchoices for each concept in the original and final models.The number of items in a value set ranged from 4 items for“Pain Duration” to 91 items for “Body Site.” The concepts thatremained in themodel (not newlyadded) are documented onaverage for 16% of patients. ►Table 4 lists the 13 conceptsfrom the final pain IM that are currently mapped to LOINCand 17 new concepts needed in LOINC. Additionally, therewere pain concepts in LOINC that were not found in organi-zations’ data.

Discussion

The purpose of our study was to develop and refine a pain IMfrom EHR flowsheet data that standardizes pain concepts,definitions, and associated value sets for assessments, goals,interventions, and outcomes. The data-driven consensus

process among 10 organizations resulted in a considerablereduction of concepts, panels (classes), and value set itemscompared with the original Pain Reference IM whichincluded 84 concepts grouped into 14 panels and 599 valueset items.5 The new model consists of 30 concepts groupedinto 4 panelswith 396 value set items. The consensus processhelped eliminate concepts from the original model mainlydue to limited use across organizations and consistency inrepresenting pain assessment scales. However, some infre-quently occurring concepts were retained in the model asthey were used to simplify documentation, such as a one-item question to assess nonverbal pain indicators versus a 5to 9 item observational pain scale. We found that organiza-tions combined some concepts for ease of documentationsuch as body orientation which included value items fromboth body location qualifier and body laterality. The pain IMseparated body orientation into the two concepts to beconsistent with LOINC standards.

One of the strengths of our study was extracting allflowsheet rows related to pain and mapping them to thePain Reference IM. EHRs become unwieldy over time withmultiple people building a system and upgrades occurring.Mapping all semantically comparable flowsheet rows to aconcept demonstrated the redundancy in EHRs. While Harriset al7 used a similar consensus process for developing apressure ulcer model, our study goes beyond their process toinclude the ability to find andmap data throughout the EHR. A

Fig. 3 Concepts retained in the pain information model (IM) through a data-driven consensus process.



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

Table

2Pa

inreferenc

eIM

conc

epts

withvalid

ationde

cision

s

Pain

Reference

IMConc

eptNam

eValidation

Dec

ision

Commen

tDefinition

Pain

Assessm

entPa

nel

New

Thispa

ininclud

esmultipleco

ncep

tsas

wella

spa

nelsforpa

inpa

ttern,

pain

loca

tion

,an

dpa

inassessmen

tscales.

Theus

eof

multipleco

ncep

tsforassessmen

tof

pain

Current

Pain

Retain

Patien

tiscu

rren

tlyex

perien

cing

pain

atthetimehe

/she

isaske

dthequ

estion

Pain

Type

Retain

Categ

orizationof

thetype

ofpa

inthat

helpsgu

idefurthe

rassessmen

tan

dtrea

tmen

tch

oice

s

Con

text

ofPa

inRa

ting

Retain

Thecircum

stan

cesthat

form

thesettingforan

even

tin

which

pain

isassessed

–qu

alifier

forwhe

npa

inisassessed

Pain

Qua

lity(narrative

)Re

tain

Preferredterm

asitisus

edin

LOINC-rem

ove“n

arrative

”as

therewas

ava

lueset

Words

patien

tsus

eto

describe

theirpa

in

Non

verbal

Pain

Indicators

Retain

Merge

dExpression

ofpa

inwiththisco

ncep

tBe

haviorale

xpressions

ofpa

in–Th

eseinclud

eno

ticeab

lech

anges

inbe

haviorsan

dfluc

tuations

inph

ysiologic

sign

s.

Pain

Exacerba

ting

Factors

FactorsTh

atAgg

rava

tePa

inRe

tain

Rena

med

tomatch

LOINC

Postures/m

ovem

ents/age

ntsthat

produ

ceor

intens

ifya

patien

t’spa

inan

dareused

toestablishtheseve

rity,irritab

ility,

andna

ture

oftheco

ndition

Pain

Alle

viatingFactors

FactorsTh

atRe

lieve

/Alle

viatePa

inRe

tain

Rena

med

tomatch

LOINC

Patien

trepo

rtingwha

trelie

vestheirpa

in,an

agen

tthat

alleviates/red

uces/relieve

spa

in

Pain

PatternPa

nel

Add

Thisisthena

meof

apa

nelw

hich

does

notha

veavalueset

Thepa

neln

amethat

includ

esthesp

eedof

onset,freq

uenc

y,du

ration

,an

dco

urse

ofpa

in

Spee

dof

Pain

Ons

etRe

tain

Nam

emod

ified

tomatch

LOINC

Therate

atwhich

pain

occu

rred

,be

fore

thevisit/en

coun

ter

Pain

Duration

Retain

Thisco

ncep

treq

uires2an

swers-a

numbe

rand

ach

oice

listfor

theun

itof

mea

sure

i.e.,2an

dda

ysORLO

INCha

squ

antity

Patien

trepo

rted

descriptionof

how

long

thepa

inoc

curs

during

apa

rticular

pain

episod

e.

Pain

Freq

uenc

yRe

tain

How

oftenpa

inoc

curs

Pain

Cou

rseprogression

(narrative

)Re

tain

Pain

Cou

rseispreferredov

erPa

inProgression

beca

use‘Pro-

gression

’im

pliesso

mething

that

isge

ttingworse

(e.g.,a

disease);‘Cou

rse’

canim

plyso

mething

gettingbe

tter

orworse.LO

INCha

sPa

intempo

ralp

attern

which

isno

minal

Thepa

tternof

pain

orco

urse

canim

plyso

mething

getting

better

orworse

Pain

Body

Loca

tion

Pane

lRe

tain

Thisisthena

meof

apa

nelsyn

onym

ouswithBo

dysite

iden

tification

pane

linLO

INC

Thepa

neln

amethat

includ

esbo

dysite,loc

ationqu

alifier,a

ndlateralityof

pain

BodySite

Loca

tion

Retain

Cha

nged

nameto

match

LOINC

Spec

ifican

diden

tified

anatom

ical

location

BodyLo

cation

Qua

lifier

Orien

tation

Retain

Split

thisco

ncep

tinto

body

loca

tion

qualifier

(orien

tation

)and

laterality.

3911

2–8Bo

dyLo

cationQua

lifier

Ana

tomical

loca

tion

description

BodyLaterality

Add

Thisco

ncep

twas

derive

dfrom

body

orientationba

sedon

LOINClaterality.

2022

8–3Ana

tomic

part

Laterality

Description

ofaside

ofabo

dy

(Con

tinue

d)



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

Table

2(Con

tinue

d)

Pain

Referenc

eIM

Con

ceptNam

eValidation

Dec

ision

Com

men

tDefinition

Pain

ScalePa

nel

Add

Thisisthena

meof

apa

nel

Theassessmen

ttool/metho

dus

edto

evalua

tethepa

tien

t’s

pain

basedon

agean

dab

ility

toself-repo

rt.

Che

cklistof

Non

verbal

Pain

Indicators(CNPI)Score

Add

CRIES

Score

Add

Critica

l-carePa

inObs

ervation

Tool

(CPO

T)Sc

ore

Add

FACES

(Won

g-Bak

er)Ra

ting

ScaleScore

Retain

Face

sPa

inSc

ale–Re

vised

(FPS

-RScale)

Score

Add

Pain

Intens

ityself-rep

orttool

(3þ

yrs),c

ulturally

sens

itivetool

(witho

uttears)

FLACCPa

inAssessm

entScore

Retain

Retained

only

score–remov

ed6ind

ividua

litemsfor

cons

istenc

y

Neo

natalInfan

tPa

inScale

(NIPS)

Score

Retain

Retained

only

score–remov

ed6individua

litemsfor

cons

istenc

y

Neo

natalP

ain,

Agitation&

Seda

tion

Scale(N

-PASS

)Score

Retain

Retained

only

score–remov

ed5individua

litemsfor

cons

istenc

y

Num

eric

Pain

Rating

Scale0–

10Score

Retain

Syno

nymou

swithNum

eric

Rating

Scale(0–1

0)Pa

tien

tsrate

theirp

ainon

asimplescalefrom

0to

10,w

here

0is’nopa

in’an

d10

is’w

orst

pain

imag

inab

le’

PAIN

Adv

ance

dDem

entia

(PAINAD)Sc

ore

Retain

Retained

only

score–remov

ed5individua

litemsfor

cons

istenc

y

Prem

atureInfant

Pain

Profi

le(PIPP)

Score

Add

RevisedFLACCPa

inAssess-

men

t(rFLACC)Score

Retain

Retained

only

score–remov

ed6individua

litemsfor

cons

istenc

y

Pain

GoalsPa

nel

Thisisthena

meof

apa

nel

Thepa

tien

t’sde

siredpa

ingo

al

Accep

tableCom

fort

Leve

l(num

eric)

Add

Rating

of0-1

0ba

sedon

wha

tthepa

tien

tstates

isacce

ptab

leforman

agingpa

in

Accep

tableCom

fort

Leve

l(nom

inal)

Retain

Narrative

descriptionof

anacce

ptab

lelevelo

fpa

instated

bythepa

tien

t

Pain

Interven

tion

sRe

tain

Anac

tion,

trea

tmen

t,or

proc

edureto

prev

entor

alleviatepa

indo

neforthepa

tien

t

Pain

Outco

meDescription

Resp

onse

toInterven

tion

Retain

Com

bine

dAccep

table

Com

fort

Levelfor

Acu

tePa

inwith

Resp

onse

toInterven

tion

Percep

tionof

patien

t’sresp

onse

tothepa

ininterven

tion

s



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

Table

2(Con

tinue

d)

Pain

Referenc

eIM

Con

ceptNam

eValidation

Dec

ision

Commen

tDefinition

Original

Model

Conc

epts

Remove

d

Pain

ScaleUsed

Remov

eRe

place

dwithapa

nelfor

pain

scalean

dthen

includ

edthe

scoreforea

chpa

inscale

PreferredPa

inAssessm

entTo

olRe

mov

eLimited

use

Pain

Descriptors

Remov

eMerged

withPa

inQua

lity,

overlapin

valuesets

Expression

ofPa

inRe

mov

eOve

rlap

invaluesets

withNonv

erbal

Pain

Indicators

Pain

Fluc

tuation

Remov

eSimila

rco

ncep

tto

pain

course

Pain

History

(narrative

)Re

mov

eRe

dun

dant

ofothe

rco

ncep

ts

Loca

tion

-multiplesites

Remov

eLimited

use

Pain

RiskFactorsPa

nel

Remov

eLimited

use–Th

isinclud

ed4co

ncep

tsof

trea

tmen

t,ph

ysio-

logic,

person

al,an

den

vironm

entalrelated

Physiological

Pain

IndicatorPa

nel

Remov

eLimited

Use

–Th

isinclud

ed4co

ncep

tindica

tors

–bloo

dpressure,he

artrate,respiration

,oxyge

nsaturation

Clin

ically

Alig

nedPa

inAssessm

ent

(CAPA

)Pa

nel

Remov

eLimited

Use

–Th

isinclud

ed5co

ncep

tsof

comfort,ch

ange

inpa

in,pa

inco

ntrol,func

tion

ing,

slee

p

Pain

/Toleran

ceforH

andling(N

ICU)

Pane

lRe

mov

eLimited

Use

–thisinclud

ed6co

ncep

ts–co

mfort,p

ositioning

,problem

siden

tified

,arou

salstate,calm

ingtech

nique

s,an

da

narrative

Resp

onse

toInterven

tion

Remov

eCom

bine

dinto

Pain

Outco

meDescription

Resp

onse

toPa

inEd

ucation

Remov

eEd

ucationwas

notinclud

edin

thePa

inIM

Pain

med

ication(freetext)

Remov

eTh

isispa

rtof

themed

icationad

ministrationreco

rd

Pain

med

icationda

telast

dose

Remov

eTh

isispa

rtof

themed

icationad

ministrationreco

rd

Elec

trod

ePlacem

ent(TEN

S)Re

mov

eTh

isisinclud

edin

pain

interven

tion

s

Abbrev

iations

:FLA

CCscore,Face,Leg

s,Activity,Cry,C

onso

labilityscore;

IM,informationmod

el;LOINC,Log

icalObservation

Iden

tifiersNam

esan

dCod

es;N

ICU,n

eona

talinten

sive

care

unit;T

ENS,

tran

scutan

eous

elec

trical

nervestim

ulation.



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

Table

3Descriptive

statistics

andLO

INCco

deforva

lidated

conc

epts

inthefina

lpainIM

Con

ceptna

me

Flow

shee

trowsmap

ped

%Organ

iza-

tions

Patien

tsNumber

ofob

servations

#Valuesetitem

s

Min

Avg

Max

Number

%Original

Fina

l

Current

Pain

118

5575

869,38

013

28,284

,945

94

Pain

Type

712

3288

980,28

715

17,895

,687

115

Con

text

ofPa

inRa

ting

111

3038

557,99

89

16,296

,187

45

Pain

Qua

lity

45

625

187,31

33

2,08

2,82

130

41

Nonv

erbal

Pain

Indica

tors

17

1650

56,254

170

3,59

827

37

Pain

ExacerbatingFactors

13

438

139,17

92

1,24

8,67

926

38

Pain

Alle

viatingFactors

223

6138

40,658

14,06

3,02

228

37

Pain

PatternPa

nel

Spee

dof

Pain

Ons

et1

58

8883

1,08

113

12,355

,821

115

Pain

Duration

27

1988

2,80

6,58

243

29,006

,751

44

Pain

Freq

uenc

y1

1131

6326

2,09

44

2,61

0,20

37

5

Pain

Cou

rse

14

838

126,07

62

823,07

44

Pain

Loca

tion

Pane

l

Body

Site

628

104

883,74

9,89

457

46,098

,215

8491

Body

Loca

tion

Qua

lifier

614

3263

763,38

012

10,969

,940

NA

NA

Body

Laterality

00

00

�0

�NA

NA

Pain

ScalePa

nel

Che

cklistof

Nonv

erbal

Pain

Indicators(CNPI)Score

209

CRIESScore

206

Critical-carePa

inObs

erva

tion

Tool

(CPO

T)Score

505

FACES

(Won

g–Ba

ker)

Rating

ScaleScore

113

2738

73,525

11,80

6,27

06

6

Face

sPa

inScale–

Revised

(FPS

-RScale)

Score

306

6

FLACCPa

inAssessm

entScore

17

2088

277,68

24

4,67

0,83

76

6

Neo

natalInfan

tPa

inSc

ale

(NIPS)

Score

11

163

258,25

44

2,72

3,49

27

7

12

250

111,32

32

4,78

1,12

96

6



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

Table

3(Con

tinue

d)

Con

ceptna

me

Flow

shee

trowsmap

ped

%Organ

iza-

tions

Patien

tsNumber

ofob

servations

#Valuesetitem

s

Min

Avg

Max

Number

%Original

Fina

l

Neo

natalP

ain,

Agitation

&Se

dationSc

ale(N

-PASS

)Sco

re

Num

eric

Pain

Rating

0–10

Score

423

8110

06,56

1,15

010

015

3,34

0,44

811

11

PAIN

Advanc

edDem

entia

(PAINAD)Score

28

2038

34,380

189

7,54

65

5

Prem

atureInfant

Pain

Profile

(PIPP)

Score

208

RevisedFLACCPa

inAssess-

men

t(rFLACC)Score

22

213

1,66

40

47,532

66

Pain

Goa

lsPa

nel

Accep

tableCom

fort

Leve

l(num

eric)

13

1163

2,71

6,90

641

65,625

,525

1111

Accep

tableCom

fort

Leve

l(nom

inal)

14

775

388,19

96

4,01

7,35

614

5

Pain

Interven

tion

s1

1344

100

2,63

9,96

440

43,971

,596

6667

Pain

Outco

meDescription

14

950

115,76

92

2,18

0,85

35

5

Abbrev

iation

s:Avg

,ave

rage

;FLA

CCscore,

Face,Leg

s,Activity,Cry,C

onso

labilityscore;

IM,informationmode

l;LO

INC,Log

ical

Observation

Iden

tifiersNam

esan

dCod

es;M

ax,m

axim

um;M

in,m

inim

um;N

A,n

otap

plicab

le.



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

Table 4 Comparison of validated pain information model with LOINC nursing physiological assessment panela

Concepts in pain IM and LOINC Nursing Physiological Assessment (n ¼ 13)

32419–4 Pain Quality

38209–3 Pain Exacerbating Factors

38210–1 Pain Alleviating Factors

38203–6 Speed of Pain Onset

38207–7 Pain Duration

38206–9 Pain Course

39111–0 Body Site

39112–8 Body Location Qualifiera

20228–3 Body Lateralitya

80316–3 Pain Scales

38221–8 FACES (Wong–Baker)a

38208–5 Pain Rating 0–10 Scale

38213–5 FLACC Pain Assessment

Concepts In LOINC Nursing Physiological Assessment not in pain Reference IM (n ¼ 6)

38201–0 Pain Onset [Date and Time] – Reported

38202–8 Pain Onset [Hours Ago] – Reported

38204–4 Pain Primary Location – Reported

38205–1 Pain Radiation

38211–9 Pain Initiating Event Narrative – Reported

80317–1 Pain Assessment [Interpretation]

New concepts not in the Nursing Physiological Assessment (n ¼ 17)

Current Pain

Pain Type

Context of Pain Rating

Nonverbal Pain Indicators

Pain Frequency

Checklist of Nonverbal Pain Indicators (CNPI) Score

CRIES Score

Critical-care Pain Observation Tool (CPOT) Score

Neonatal Pain, Agitation & Sedation Scale (N-PASS) Score

Neonatal Infant Pain Scale (NIPS) Score

PAIN Advanced Dementia (PAINAD) Score

Premature Infant Pain Scale (PIPP) Score

Faces Pain Scale – Revised (FPS-R Scale) Score

Revised FLACC Pain Assessment (rFLACC) Score

Acceptable Comfort Level (numeric)

Acceptable Comfort Level (nominal)

Pain Outcome Description

Abbreviations: FLACC score, Face, Legs, Activity, Cry, Consolability score; IM, information model; LOINC, Logical Observation Identifiers Names andCodes.aConcepts in LOINC but not in Nursing Physiological Assessment Panel.



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

custom query provides a method for extracting comparabledata for interoperability and cross-organization pain research.

Using real-world evidence from large data sets is anincreasing trend in research due to the potential cost-sav-ings.13However, if research was conducted evaluating a vitalsign such as patients’ pain and it used only one of themultiple flowsheet rows mapped to a concept like “NumericPain Rating 0–10 Score,” then the study’s effectiveness forevaluating medications or nursing pain interventions couldresult in false conclusions because the pain rating datacontained in the other flowsheet rows mapped to “NumericPain Rating 0–10 Score” would be missing. Implementationof an IM can reduce redundancy and increase the usefulnessof the data.

While it might be ideal to have a single pain scale, weretained 12 unique pain assessment scales which includeboth self-report and observational assessments. Nurses andother clinicians need to select the appropriate tool based onage, setting, and clinical condition. The pain assessmentscales identified address these wide variety of circum-stances. One essential point, however, is the importance ofconsistent use of the same scale over time to evaluatepatient’s progress.

Results of our study extend the concepts needed in LOINCfor interoperability.14 For example, there are some importantpain concepts that are missing from LOINC such as “CurrentPain,” “PainType,” and “Acceptable Comfort Level (numeric).”However, there are LOINC concepts not found in our study,such as “Pain Onset,”which is a date and time stamp. AnotherLOINC term that was not found in our organizations’datawas“Pain Primary Location.” This is likely due to the fact thatpatients can have multiple pain locations, each with its ownassessment, so it is not used in practice.

There are several future steps planned including addingstandard terminology mappings to the concepts, validatingthe pain IM with additional organizations and settings, andapplying the process to validate IMs for other clinical areas.The terminology standards include LOINC for assessmentsand some outcomes and Systematized Nomenclature ofMedicine–Clinical Terms (SNOMED CT) for value sets asso-ciated with assessments, problems, and interventions.14

Additional terms will need to be submitted to LOINC andSNOMED CT when codes do not exist. Once this work iscompleted, broad dissemination is needed. The NursingKnowledge Big Data Science Initiative is developing anopen source repository for sharing work such as the painIM.15 Additional research is needed in several areas. Valida-tion of the IM with a broader set of stakeholders includinghome care, hospice, long-term care, and others would bebeneficial. The IM could also be validated with multipleclinical experts specific to pain using a Delphi technique orother consensus approach. Further IMmapping is needed onother nurse-sensitive measures such as falls, catheter-asso-ciated urinary tract infections (CAUTI), and central lineassociated bloodstream infections (CLABSI). Once themodelshave been validated, research is needed on implementationof the models and the coded data elements in EHRs tounderstand what worked, what problems and issues were

uncovered, how documentation is impacted, and if anydifferences exist based on vendor solutions. Ultimately, wewant to know if the IMs and use of coded key data elementsincreases interoperability and our ability to enable large,multicenter research including comparative effectivenessresearch.

Our work has several limitations. A volunteer sample oforganizations participated, and thus, the model may not begeneralizable to all organizations. While this was a conve-nience sample, which can limit generalizability of findings,the geographic locations, population size, and variety ofpractices provides a foundation for a generalizable pain IMthat can be used to support research. The pain IM is abeginning and it is anticipated that it will evolve overtime. In particular, there are additional concepts neededfor cardiac services and pain clinics may have more specia-lized assessments and interventions. Therewas variability inthe data extraction approaches and criteria used at eachparticipating organization that could influence the results.No attempt was made to dictate how to implement the painIM in an EHR, and thus organizations need to determine thebest practice for doing this. While the researchers consultedtheir pain experts, amore conscious effort is needed in futurework to include domain experts. Another limitation is thatFloMap is not yet available publicly nor is the SQL script fordata extraction. If others are interested in its use, they cancontact S.J., one of the authors on this article.

Conclusion

The purpose of our research was to validate and refine a painIM by using a data-driven approach across multiple healthsystems. The resulting pain IM is a consensusmodel based onactual EHR documentation in the participating health sys-tems. The pain IM captures the most important conceptsrelated to pain.

Clinical Relevance Statement

Secondary use of EHR data must be standardized for com-parison within and across organizations. Our study resultedin 30 concepts, definitions, and associated value sets agreedupon by 10 organizations as useful for building or optimizingan EHR. Our methods also allowed agencies to map theirflowsheet data to these concepts to support future research.

Multiple Choice Question

Variation in flowsheet data are often due to

a. The content and guidelines provided by vendorsb. Professional guidelines that influence contentc. Limited resources and rapid deployment of EHRsd. Available guidelines from terminologies of how to build

EHRse. All of the above

Correct Answer: The correct answer is e, all of the above.Vendorsprovide generic content andguideorganizations in



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

using consensus-based approaches to configure the bulk oftheir system to meet the clinical requirements of theorganization. Much of the documentation, however, iscaptured in flowsheet format, using nonstandardized,semistructured data in a matrix format for patient assess-ments, goals, problems, interventions, and outcomes ofcare. Limited resources and rapid deployment timelinesprovide little time for organizations to identify and adoptstandardized terminologies and use IMs to design flow-sheets for future data sharing. Further, the informaticiansare required to choose frommultiple terminologies3,4withlimited reference standards to guide flowsheet builds.

Protection of Human and Animal SubjectsThe data were considered “metadata” and representeddescriptions of how the organization’s EHR was designedand aggregated counts for frequency of use; no patient-identifiable data were included. Each participant con-sulted with their organization to determine whetherInstitutional Board Approval was needed. If InstitutionalReview Board (IRB) approval was required, it wasobtained prior to data extraction and transmission to asecure database at the University of Minnesota.

Conflict of InterestNone.

AcknowledgmentWe would like to acknowledge the organizations thatwere willing to share their data and committed their stafftime to collaborate on this project over an extendedperiod of time.

References1 GoossenW, Goossen-Baremans A, van der Zel M. Detailed clinical

models: a review. Healthc Inform Res 2010;16(04):201–214

2 Johnson SG, ByrneMD, Christie B, et al.Modelingflowsheet data forclinical research.AMIAJt Summits Transl SciProc2015;2015:77–81

3 Rutherford M a. Standardized nursing language: what does itmean for nursing practice? Online J Issues Nurs 2008;13(01):1–7

4 Westra BL, Delaney CW, Konicek D, Keenan G. Nursing standardsto support the electronic health record. Nurs Outlook 2008;56(05):258–266.e1

5 Westra BL, Christie B, Johnson SG, et al. Modeling flowsheet datato support secondary use. Comput Informatics Nurs 2017;35(09):452–458

6 Chow M, Beene M, O’Brien A, et al. A nursing information modelprocess for interoperability. J AmMed Inform Assoc 2015;22(03):608–614

7 Harris MR, Langford LH, Miller H, Hook M, Dykes PC, Matney SA.Harmonizing and extending standards from a domain-specific andbottom-upapproach:anexamplefromdevelopment throughuse inclinical applications. J AmMed InformAssoc 2015;22(03):545–552

8 American Nurses Association & the American Society for PainManagement. Pain Management Nursing: Scope and Standards ofPractice. 2nd ed. Silver Spring, MD: American Nurses Association;2016

9 Nahin RL. Estimates of pain prevalence and severity in adults:United States, 2012. J Pain 2015;16(08):769–780

10 Gaskin DJ, Richard P. The economic costs of pain in the UnitedStates. J Pain 2012;13(08):715–724

11 The Joint Commission. Joint Commission Enhances Pain Assess-ment and Management Requirements for Accredited Hospitals.The Joint Commission Perspectives 2017;37(7):1-3. Available at:https://www.jointcommission.org/assets/1/18/Joint_Commis-sion_Enhances_Pain_Assessment_and_Management_Require-ments_for_Accredited_Hospitals1.PDF

12 Delaney CW, Pruinelli L, Alexander S, Westra BL. 2016 NursingKnowledge Big Data Science Initiative. Comput InformNurs 2016;34(09):384–386

13 Jarow JP, LaVange L, Woodcock J. Multidimensional evidencegeneration and FDA regulatory decision making: defining andusing “real-world” data. JAMA 2017;318(08):703–704

14 Matney SA, Settergren TT, Carrington JM, Richesson RL, Sheide A,Westra BL. Standardizing physiologic assessment data to enablebig data analytics. West J Nurs Res 2016;39(01):63–77

15 Carter-Templeton H, Effken J, Weaver C, Cochran K, Androwich I,O’Brien A. Toward a central repository for sharing nursing infor-matics’ best practices. Comput Inform Nurs 2016;34(06):245–246



Thi

s do

cum

ent w

as d

ownl

oade

d fo

r pe

rson

al u

se o

nly.

Una

utho

rized

dis

trib

utio

n is

str

ictly

pro

hibi

ted.

https://www.jointcommission.org/assets/1/18/Joint_Commission_Enhances_Pain_Assessment_and_Management_Requirements_for_Accredited_Hospitals1.PDF



Documents

Validation and Re ﬁnementofaPainInformation Model from …...Validation and ReﬁnementofaPainInformation Model from EHR Flowsheet Data Bonnie L. Westra1 Steven G. Johnson1 Samira