HISTORICAL TECHNOLOGY UTILIZATION AND INTENDED BEHAVIORS OF

NEONATAL INTENSIVE CARE UNIT (NICU) NURSE STAFF TOWARDS

ELECTRONIC HEALTH RECORDS AS A FRAMEWORK FOR FUTURE

TECHNOLOGY UTILIZATION ACCEPTANCE

A DISSERTATION

SUBMITTED ON THE FIFTEENTH DAY OF NOVEMBER 2012

TO THE DEPARTMENT OF GLOBAL HEALTH SYSTEMS AND DEVELOPMENT .

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS

OF THE SCHOOL OF PUBLIC HEALTH AND TROPICAL MEDICINE

OF TULANE UNIVERSITY

FOR THE DEGREE

OF

DOCTOR OF PHILOSOPHY

BY

Michael Thad Phillips, MSHA, MSHI, MBA

Claudia Campbell, PhD

I. Abstract

Background and Significance- The patient safety and quality of care movement of the

past decade has advanced to include and be somewhat dependent on new information

technologies in the form of electronic health r~cords. The healthcare policy environment

has both quality and information technology tied very tightly together in an effort not

only to improve healthcare quality, but also to save money. In order to achieve these

goals, a better understanding ofhow clinical staff utilizes technology is necessary as it

relates to electronic health record investment success. The present research study aims to

measure technology end users' past electronic health record objective use in conjunction

with their subjective feelings towards past and future teclmologies in order to better

understand how to predict future system utilization. This study is being performed in

order to 1) Test the relevance of expanding the cutTent IT theory based TAM to include

objective historical use data, to more fully explain the variance in actual technology use;

2) To provide an expanded framework for future application use models of this nature;

and 3) To serve as a quality improvement collection tool, utilizing narrative feedback

from study patiicipants to identify organizational attributes that positively or negatively

impact actual performance when utilizing an information technology (IT).

Literature Review- A comprehensive literature review was performed in the following

areas: 1) Quality of care and patient safety from a healthcm·e policy perspective; 2)

Information technology theories that have been applied to the healthcare setting; and 3)

2

An integrated review of performance measures as it relates to information technology in

neonatal intensive care units (NICUs).

Conceptual Models- The study utilizes the core conceptual model of TAM (see Figure

1), described as a measure of end user technology acceptance through two subjective

variables, perceived usefulness and perceived ease of use. The study measures the

acceptance of an electronic health record technology, Cribnotes, through the perspective

ofNICU nursing staff by measuring not only the traditional TAM subjective variables of

perceived ease of use and perceived usefulness by individual nurses, but also utilizing

objective performance data of the same nurses on a past information technology,

Medication Administration Checking (MAK). Additional independent variables of

nursing staff measured will be age, gender, race, professional degree, andjob title.

Lastly, actual system use of the Cribnotes technology will serve as the dependent variable

and will be measured by objective performance data.

Hypothesis and/or research questions- HI: The MAK objective nursing performance

score will be the primary predictor, or will have the most impact, on the Cribnotes

objective nursing performance score when compared individually with all other

independent variables. H2: The MAK objective nursing performance score will be the

primary predictor, or will have the most impact, on the Cribnotes objective nursing

performance score when compared with all other independent variables in aggregate; H3:

The MAK objective nursing performance score will work in conjunction with RN

behavioral intentions to create a more explanatory framework; and H4: The full model

3

consisting ofMAK objective nursing performance score and the pre-implementation

subjective RN behavioral intentions will explain all the variation in the dependent

variable, Cribnotes objective performance score. Research questions include: 1) Will RN

historic system perfonnance be a better predictor of future system use than utilizing their

attitude, or behavioral intention to use it? 2) Will RN historic system performance work

in conjunction with RN behavioral intentions to explain more R squared variability

together, rather than independently? 3) Were clinical, social, and/or technical factors

identified that can be added as subjective variables to enhance future analysis ofR

squared findings related to RN performance?

Methods and Materials- The study takes place at the University of South Alabama's

Children's and Women's Hospital in the NICU Department. The sample size is 69 nurses

and the. duration of the study is a 24 months of software utilization timeframes. The basic

research design revolves around the development and testing of a healthcare information

technology prediction model framework. This prediction model framework is based on

the traditional TAM with the enhancement of an added core variable portraying historic

objective use of an information system. The objective use data is measured in terms of

nursing performance on an individual basis when quality checks specific to breast milk

are performed prior to administration to neonates. In addition, pre-post survey data was

collected from the nurse participants related to their attitudes, or intentional behaviors,

towards information teclmology of the past, present, and future. The surveys are directly

linked to the traditional TAM with four survey questions specific to their perceived ease

of use and perceived usefulness of two information teclmologies, MAK and Cribnotes.

4

Multiple regression analysis was conducted in order to explain different combinations of

outcome variability related to future use of the Cribnotes system.

Results and Discussion- The Quantitative regression analysis results show that the full

model is statistically significant (p<.05) and explains on a very conservative level 14.7%

of the variation in the Cribnotes performance score. Related to the individual coefficients,

the objective measure of performance scores on the old system (MAK) is significantly .

positively related to the performance scores on the new system (Cribnotes) (p < 0.05).

Similarly, the pre-implementation survey-based subjective measure result for Pre­

Cribnotes Ease of Use is also significantly positively related to the performance scores on

the new system (p<.05). The qualitative content analysis reveals two negative high level

categories represented by Overall Clinical Impact (Negative) and Overall IT Impact

(Negative) that suggest unexplained variability may be due to a combination of clinical

workflow and IT Infrastructure concerns. Based on these findings, a new model

framework was proposed.

Conclusion- Hypotheses 1, 2, and 3 were accepted because the addition ofthe objective

performance score measurement was the most powerful of all the variables. It also

worked in combination with subjective variables to enhance the TAM. Hypothesis 4 was

rejected due to the remaining variability left unexplained by the full model. All research

questions echoed Hypotheses 1-3 and were thus affirmed.

5

II. Background and Significance

For more than a decade, the theme of improved patient safety and quality of care in the

United States healthcare system has been a common thread in healthcare policy decision

making and overall strategy. Specifically, the on-going push for the implementation and

meaningful use of electronic health records (EHR) seems to be gaining more ground than

ever and is strongly supported by the federal government. EHRs as a fundamental

component in the patient safety and quality of care movement will not only provide an

electronic medium for what in the past was a paper based system, but has the potential to

fundamentally change the workflows and practices of clinical end users. For these

reasons, the importance of understanding how to become successful users of teclmology

across the healthcare spectrum has never been greater.

The present research study aims to measure technology end users' past electronic health

record objective use in conjunction with their subjective feelings towards past and future

technologies in order better understand how to predict future system utilization. It is

significant in that it applies and expands upon an Information Systems Theory, the

Technology Acceptance Model (TAM) (Davis, 1989), to an academic teaching hospital

based Neo-natal Intensive Care Unit (NICU). The NICU is located in the Southeast

region ofthe United States at the University of South Alabama Children's and Women's

Hospital. This NICU is considered large due to its regional presence and because it

averages seventy-five neonates daily. NICUs are by nature complex and as the research

literature suggests, it is very impmiant to understand how users adopt a new technology

in order to be successful (Rikli, 2009). This study was performed in order to model the

6

relevance of expanding the cun·ent IT theory based TAM to include objective historical

use data, to more fully explain the variance in actual technology use. This study is not

intended to be directly used across other departments within USA Hospitals or at other

institutions, but rather to provide an expanded framework for future application use

models of this nature. In addition, this study will serve as a quality improvement

collection tool, utilizing narrative feedback from study participants to identify

organizational attributes that positively or negatively impact actual performance when

utilizing an information technology (IT). The research questions are as follows: 1) Will

RN historic system performance be a better predictor of future system use than utilizing

their attitude, or behavioral intention to use it? 2) Will RN historic system performance

work in conjunction with RN behavioral intentions to explain more R squared variability

together, rather than independently? 3) Were clinical, social, and/or technical factors

identified that can be added as subjective variables to enhance future analysis ofR

squared findings related to RN performance? The study sample consists of an estimated

one hundred nursing participants. Participants were administered surveys to gain an

understanding of their attitudes, or behavioral intentions, towards use of both the MAK

and Cribnotes information systems. Pruiicipant data specific to how well they used the

information systems was measured in terms of perfonnru1ce quality checks that occur

before administering breast milk to neonates. This data was gathered via system

utilization reports spanning a consecutive eighteen month period oftime for the MAK

system, followed by a 6 month period of time for the Cribnotes system.

7

This study first draws from the literature the linkages between healthcare quality

initiatives, information systems, and associated financial implications. In so doing, a

number of information systems theories will be reviewed with the TAM being selected

and directly applied to the framework for this study. A frniher in depth review of TAM is

then conducted as it relates to the theory. In addition, a review ofNICU teclmology

performance studies related to clinical outcome and end users are reviewed. Next,

conceptual models ofT AM, study variables and interventions are developed and

explained. The methodology comprising the study design is then addressed by defining

the setting, sample, data sources, intervention, scoring calculations, and associated data

elements. The analytical approach is that of a model based on past experience where

quantitative regression analysis will be conducted using different combinations of the

variables to explain the different levels of variability related to use of the Cribnotes

system. In addition, qualitative based surveys will be analyzed and content analysis will

be performed to interpret the impact of nursing behavioral intentions related to Cribnotes.

III. Literature Review

The approach to this comprehensive literature review encompasses a review of health care

policy in the areas of patient safety and quality care, then reviews theories related to

teclmology in healthcare, and lastly, identifies themes and trends discovered as a result of

an integrated review of varying types ofEHR performance literature specific to NICUs.

Healthcare Policy- Patient safety and quality momentum over the past decade

In 1999, the repmi, "To Err is Human: Building a safer healthcare system" was released

by the Institute of Medicine (IOM) providing insight into the deep rooted patient safety

8

concerns facing the US health care system. The report describes common problems that

occur in healthcare everyday including but not limited to adverse drug events, improper

transfusions, surgical injuries, wrong-site surgery, restraint-related injuries, falls, burns,

pressure ulcers, mistaken patient identities, suicide, and death (Kohn, 1999). Further, the

report identifies that intensive care units (ICUs), operating rooms, and emergency

departments are locations within healthcare delivery systems that are most prone to high

error rates with dire consequences (Kohn, 1999). Other components of the repmi that

have great bearing are the resulting quantified costs in terms of monetary and human

lives. At the time of publication of this repmi, the nationwide costs of preventable

medical errors were estimated to be between $1 7 billion and $29 billion per year in

hospitals across the United States, and a staggering number between 44,000 and 98,000

deaths (Kohn, 1999).

While patients and practitioners alike were still digesting the IOM report, business

owners took notice of one recommendation from the report that urged large employers to

provide more market reinforcement for the quality and safety of health care. SpUITed by

the patient safety quality movement initiated by the IOM's 1999 report, the Leapfrog

Group was organized in fall of 2000 by a constituency of large employers with suppmi

from the Business Roundtable (BRT) (The Leapfrog Group, 2011). The primary goal was

to create a rewards system built aroUI1d improvements related to safety and quality for

hospitals. The Leapfrog Group measures and rates hospitals on best practices, in

accordance with the National Quality Forum (The Leapfrog Group, 2011). If a hospital

follows these best practices related to patient safety, overall ris!< to the patient, provider,

9

and the institution should decrease in the care environment. Lastly, the Leapfrog Group

takes the consumer point of view when determining how to approach organizational

standards directly related to patient safety. US Hospitals are not mandated by any Federal

or State laws to participate with the Leapfrog Group and compliance with the standards

are on a voluntary reporting basis (The Leapfrog Group, 2011).

In the spring of 2001, IOM published "Crossing the quality chasm- a new healthcare

system for the 21st century." The report's recommendations call on Healthcare

Organizations (HCOs) to take a hard look at their processes of care delivery and adjust

them as necessary to work in conjunction with information teclmologies. In addition, an

overall leadership review from a managerial and clinical perspective is recommended in

the rep01t. More specifically, the development of efficient and effective teams who will

better manage clinical knowledge and skill-sets, and coordinate care across patient

conditions, services, and settings over time was recommended (Institute of Medicine-

Committee on Quality of Health Care in America, 2001). Lastly, an improved checks and

balance system was recommended to instill performance based outcome measurements

for purposes of greater accountability and overall improvement (Institute of Medicine-,..

Cmrunittee on Quality of Health Care in America, 2001).

As the patient safety and information technology charge continued to grow over the next

two years, so too did the importance of protecting health information. As it relates to

safeguarding protected patients' healthcare ·information, the Health Insurance Pmtability

and Accountability Act compliance deadlines for the Privacy portion were instituted in

10

2003 (Department of Health and Hmnan Services- Office of the Secretary, 2003). In

addition, President Bush's State of the Union address places the healthcare quality

conundrum as the second most important goal of his administration. The focus is

broadened in that high quality of care is one aspect, but the goal also encompasses access

and affordability for the American people (President George W. Bush, 2003). That is too

say, showing the importance of quality in a manner greater than clinical outcomes, and

delving into core issues such as access and affordability of healthcare for all Americans.

This shows the President's agenda is truly focused on improving quality of care for those

being treated in the US healthcare system today, as well as, those who are not able to be

treated appropriately.

Once again, making a consistent stand on patient safety, President Bush's State of the

Union address in 2004, echoes that of2003, but is more focused on the computerization

ofhealthcare data and operations to play a key role in improving healthcare with

reductions in both medical mistakes and costs (President George W. Bush, 2004).

Following President Bush's statements and goals, the IOM published another article in

2004 titled, "Patient Safety: Achieving a New Standard for Care" (Aspden, 2004). This

report further stresses the importance of IT in contributing to the improvement of patient

safety and sets the stage for standards related to the development of a national health

information infrastructure (NHII). The need for the inception of the NHII was described

as critical for specifically addressing quality of care related patient safety problems

(Hjmi, 2005). President Bush then made an aggressive move by signing an executive

order requiring the Department of Health and Hmnan Services (HHS) to lead the charge

11

to advance the American people closer to obtaining secme access to EHRs by the year

2014. President Bush also called for the adoption ofhealthcare information teclmology

(HIT) interoperability standards and improved quality measures across internal Federal

Agencies such as HHS, the Depmiment of Defense (DOD), the Depm·tment of Veterans

Health Affairs (VHA) and the Office of Personnel Management (OPM) (American

College of Emergency Physicians, 2011). Lastly, President Bush also called for the

creation of the Office ofthe National Coordinator for Health Infmmation Teclmology

(ONCHIT) at the Federal level. Executive Order 13335 essentially demanded a strategic

plan at the national level within a ninety-day timeframe that would lay out a course of

action for public and private sector implementations of HIT (Department ofHealth and

Human Services- Office of the National Coordinator for Health Information Technology

(ONCHIT), 2004).

From President Bush's 2005 State of the Union Address, he once again urges the nation

that the use infmmation technology is key to preventing medical enors. He also likens

this to cost savings in not only the quality improvement arena, but also points tci a

comprehensive health care agenda and medical liabilities reform (President George W.

Bush, 2005). Also, in 2005, The Patient Safety and Quality Improvement Act of2005

pointed to teclmology as a catalyst for healthcare process improvement. Data collection,

analysis, and dissemination would be carried out by newly created Patient Safety

Organizations (Chuo, 2008) .. In the spring of2005, The Department of Health and Human

Services (HHS) released final HIP AA Security regulations that becan1e mandatory on

April20, 2005 and required healthcare providers to implement administrative, physical

12

and technical practices to protect the security of individually identifiable health

information that is electronically maintained or transmitted (Department of Health and

Human Services- Office ofthe Secretary, 2003).

In 2006, President Bush addresses patient safety once again in his State of the Union

speech, stating, "We will make wider use of electronic records and other health

information technology, to help control costs and reduce dangerous medical errors"

(President George W. Bush, 2006).

Again, and for the fifth consecutive year in a row, President Bush's State of the Union

Address in 2007 impresses upon the nation his patient safety goals. He once again ties the

need for improved information teclmology directly to medical enor and cost reductions

within the US healthcare system (President George W. Bush, 2007).

In 2009, the Obama administration further pushes the agenda by subsequently signing

into law The Health Information Technology for Economic and Clinical Health Act

(HITECH) within the larger American Recovery and Reinvestment Act of 2009 (ARRA)

(President Barrack Obama, 2009). The HITECH ACT is a federal attempt to infuse

billions of reimbursement dollars in the form of incentives to eligible providers and

hospitals if they adopt and use certified technology in a meaningful way. The Medicare

and Medicaid incentive programs for electronic health records are thus being driven by

"meaningful use". The Centers for Medicare and Medicaid Services (CMS) defines

meaningful use as a three stage approach where EHRs are used as an integral component

of a larger information technology infrastructure necessary to improve patient safety and

overall quality ofhealthcare (CMS Office of Public Affairs, 2010). The first stage of

13

meaningful use is to essentially collect electronic healthcare information in a

standardized way in order to better trend clinical conditions, coordinate patient care, and

begin repmiing clinical quality measures at both the local and public health levels (CMS

Office of Public Affairs, 2010). The stage one criteria proposes that eligible providers

completing twenty five objectives, fifteen of which are core, and eligible hospitals

completing twenty four objectives, fomieen of which are core. The second stage of

meaningful use is focused on expanding and building upon stage one quality criteria by

\ improving management of diseases, medications, clinical decision support, and care

transitions at the local level, while also further integrating at a state level through the

exchange of infonnation with public health agencies (CMS Office of Public Affairs,

201 0). The third stage is postured to begin seeing results for meaningful system use in

terms of efficiencies and effectiveness represented by patient safety, and overall quality

improvements at the local, state, and national levels (CMS Office of Public Affairs,

2010). Quality improvements are focused not only on the metrics outlined for eligible

providers and eligible hospitals relative to their care of patients, but also to improve

population health by empowering the patients to have more autonomy through access to

their electronically protected health information (EPHI) (CMS Office of Public Affairs,

2010). In addition to the three stages of meaningful use criteria, the software technology

used by eligible providers and eligible hospitals must be ce1iified by an Office of the

National Coordinator (ONC) Authorized Testing and Certification Body (ATCB) to be

eligible for incentive payments (Centers for Medicare and Medicaid Services, 201 0).

Incentive payments are based on Medicare and Medicaid patient encounters acl'oss the

14

aforementioned three phases spanning multiple years depending on the payer (President

Barrack Obama, 2009).

The Medicare EHR Incentive Program began in 2011 and offers eligible providers up to

$44,000 with a decreasing annual sliding incentive payment scale over a five year period

(CMS, 2010). For eligible hospitals, there is a base incentive payment of$2,000,000 and

additional monies depending on the number and type of patient encounters (CMS, 2010).

Begim1ing in 2015, however, physicians and hospitals that do not use certified products

in a meaningful way will be penalized by Medicare on a sliding scale of 1% in 2015, 2%

in 2016, and three-five percent for subsequent years (President Barrack Obama, 2009).

The Medicaid EHR Incentive Program provides similar incentive payments for eligible

providers, but the provider or practice must choose either Medicare or Medicaid, not

both. Under Medicaid, the eligible providers are allowed to receive up to $63,750 with a

decreasing ammal sliding incentive payment scale over a six year program pmiicipation

period. For eligible hospitals, they too begin with a $2,000,000 base payment, and have

the choice of pmiicipating in both the Medicm·e and/or Medicaid EHR Incentive

Programs (CMS, 2010). CmTently, there are no payment adjustments under the Medicaid

EHR Incentive program for eligible providers and hospitals that are unable to achieve

their goals within the given timeline (CMS, 201 0).

The HI-TECH Act also provides two billion dollars for discretionary spending under the

management ofONCHIT, and establishes a goal aforementioned in the President's state

of the union adch-ess, that the people of the United States of America will have electronic

health records being utilized by 2014 (President BmTack Obama, 2009).

15

Further, in 2010, the Patient Protection and Affordable Care Act (PPACA), was signed

into law. One of the goals of the PP ACA is to better connect the quality of care with

reimbursements for that care in an effort to reduce wasteful spending and decrease related

costs. This is to be achieved by creating new payment methodologies, delivery models,

and value-based purchasing programs, also known as pay for performance (P4P). There

will be mandated checks and balances in place specific to quality reporting (American

College ofEmergency Physicians, 2011). The act also establishes the Center for

Medicare and Medicaid Innovation (CMI). CMI is working with providers, payers, and

advocacy groups at the Federal, State, and local levels to gather their input in an effort to

remodel and propagate new models of more cost effective delivery of quality healthcare.

In so doing, CMI' s focus on improving the quality of care and coordination provided to

patients and their resulting outcomes will be the foremost task at hand, while also looking

forward at the formulation of effective community care models (Sharamitaro, 2011).

Dentzer (2011) takes a pulse ofhow much improvement has been made as it relates to

patient safety and quality of care in the US healthcare industry since the publication of

the IOM's "To Err is Human" report. Notably, the article suggests that while there have

been significant quality improvements made, they are slow coming and that the initial

problems identified could be much worse than first anticipated. Dentzer alludes to the

fact that as the research methodologies advance so too does the uncovering of more

quality issues (Dentzer, 2011). Of the notable quality improvements that have been made

across the country, one common denominator is that healthcare quality is impmiant

enough to be addressed consistently at a national level. National organizations such as

16

the Centers for Disease Control and Prevention (CDC), Institute for Healthcare

Improvement, and the Joint Commission on Accreditation ofHealthcare Organizations

(JCAHO) are working in conjunction with providers throughout the country on quality

improvement processes (Pronovost, 2011; Chassin, 2011). Similarly, this can be also seen

in national healthcare provider organizations, such as Ascension Healthcare Delivery

System and Veteran's Affairs (VA) through their commitment to improving quality of

care (Pryor, 2011; Trivedi, 2011).

Both public and private stakeholders, comprised of state and federal agencies, providers,

payers, and regulators across the US have found middle ground in developing common

quality standards to measure in reducing Intensive Care Unit (ICU) based central line

blood stream infections (Pronovost, 2011). Forty-five states participated in the effort and

a 62% reduction outcome was validated by the Centers for Disease Control and

Prevention, between the years 2001 and 2009 (Pronovost, 2011). The success was

attributed to the ability of interdisciplinary teams in local hospitals working together at

the state level, states sharing with one another to the point that a national social

community was formed, and then formalized by the American Hospital Association

(AHA) (Pronovost, 2011). Support from Quality associations such as JCAHO in

conjunction with support by payers both private and public supplied the regulatory and

reimbursement initiatives to assist providers in developing goals and reaching them

successfully in a standard, unified approach (Pronovost, 2011).

17

Ascension Healthcare Delivery System sought quality improvement in 2003 for a number

of areas such as preventable deaths, trauma related to births, pressure ulcers, ventilator

associated pneumonia, and hospital acquired infections (Pryor, 2011). The Board of

Directors backed a clinical excellence team who was responsible for overseeing the

quality standards (Pryor, 2011). Executive compensation was further tied with quality

performance and each hospital's results were compared nationally with one another

(Pryor, 2011). From this, hospital pminerships created quality teams called Affinity

Groups m1d developed quality indicators called Change Packages that were used

throughout their sixty-nine hospitals that span twenty states (Pryor, 2011). The quality

initiatives were encompassed in organizational govemance structure and most notably,

during the yem·s 2004-2010, preventable deaths were significantly decreased on average

of one thousand five hundred per year (Pryor, 2011 ). In addition, Ascension's

performance when compared with national averages in 2010, revealed reductions in

system neo-natal mortality rates of 89 percent, birth trauma rate of 65 percent, pressure

ulcers rates of 94 percent, ventilator associated pneumonia rates of 7 4 percent, and

hospital acquired infection rates of 43 percent (Pryor, 2011).

Trivedi and colleagues focused on another national quality endeavor from the perspective

ofthe VA (Trivedi, 2011). While the VA has made great strides in improving healthcare

quality for our nation's veterans, the outcomes related to racial disparities were studi~d

tln·ough process of care delivery and clinical outcome measures (Trivedi, 2011 ). The

quality indicators measured were comprised of diabetes, cardiovascular disease,

hype1iension, and cancer screenings from years 2000-2009 in most cases (Trivedi, 2011).

18

From the data, all quality performance rates showed improvement over time, other than

mammography (Trivedi, 2011). However, there were findings of significant racial

disparities in clinical outcomes between whites and blacks with the blacks having more

problems in the areas of blood pressure, glucose and cholesterol (Trivedi, 2011). While

this study is specific to the VA, it alludes to other identical publicized trends outside the

VA as it relates to the US population where it seems improvements are being made for

blacks, but a lower rate than for whites (Trivedi, 2011).

Recent research with a focus on adverse advents in hospitals, as they relate to patient

safety, found that current measures in the United States could be missing the majority of

adverse events and therefore only truly account for ten percent of the total (Classen,

2011). This was exemplified by a study comparison of measures across three hospitals

consisting of volunteer reporting, the Agency for Healthcare Research and Quality's

(AHRQ) Patient Safety Indicators, and the Institute for Healthcare Improvement's Global

Research Tool (Classen, 2011). Of the three measures, the implementation of the Global

Trigger Tool, which is defined as a new and improved intense chali review, revealed that

ten times more adverse events than usual were found throughout the records in

comparison with the other measures described that are used and accepted today (Classen,

2011). The bottom line is that there is plenty of room for improvement not only in

quality, but also in how it is measured across US Hospitals.

With exan1ples like these in mind, JCAHO is looking to the future and focused on a

consistent approach to preventing even the smallest of enors in hopes to achieve a more

19

reliable quality experience for patients (Chassin, 2011). As it stands today, there are

described pockets of excellence across the US where organizations are surpassing quality

standards in some areas and then failing in others (Chassin, 2011). With this in mind,

JCAHO has its sights set on further establishing a high reliability model by developing

standards that are achievable, but are more consistently met over time across all quality

standards, not just some of them (Chassin, 2011 ). This is based on the principle of

collective mindfulness, when applied to healthcare would mean that all workforce would

understand the necessity of care quality and the high stakes involved when something

goes awry (Chassin, 2011). The creation of high reliability organizations can be achieved

when an organization's leadership commits to the goal, develops and allows the

organizational culture to support the goal, and enables the workforce to adopt robust

process improvements such as six sigma and lean management to reach the stated goal

(Chassin, 2011).

With more than ten years of focused growth in the patient safety and quality movement,

mandated use of electronic health records through an incentive, and then disincentive

program for non-participants, and reduction in Federal Medicare reimbursements with no

budgetary end in sight, it has become clear that the stage is being set for the shift to pay

for performance (P4P) in an incremental, yet swift fashion.

Theoretical perspectives - Healthcare Information Technology

Primary literature findings on Electronic Health Record (EHR) adoption, diffusion, and

acceptance theories yielded varying theories and models comprised of the following: 1)

Diffusion oflnnovations Theory (Rogers, 1962; Ford, 2006); 2) Bass Diffusion Model

20

(Bass, 1969; Ford, 2006); 3) Technology Acceptance Model (TAM) (Davis, 1989; Hyun,

2009); 4) Task-Technology Fit Model (TTFM) (Goodhue, 1995; Hyun, 2009); 5)

Grounded Theory (Glaser, 1967; Yoon-Flannery, 2008); 6) Game Theory (Nash, 1950;

Klarreich, n.d.; Woodside, 2007); 7) Resource Dependency Theory (Pfeffer, 1978;

Bramble, 2010); and 8) Combinations therein.

For purposes of this study, the researcher chose TAM as the theoretical framework

because it is an accepted technology theory in the literature related to healthcare

information technology implementations specific to electronic health records and nursing

documentation systems. Ten percent of all information systems publications use the

TAM, with thiliy to f01iy percent of IT acceptance attributed to reviews ofthe basic

theory (Holden, 2010).

TAM is an information systems theory that models how user attitudes or behavioral

intentions to use a technology compare with how they actually use the teclmology. TAM

hypothesizes that a user's intended behavior predicts actual system use, and that external

variables, such as human and social factors, indirectly determine an individual's attitude

toward technology acceptance by influencing perceived usefulness and perceived ease of

use (Morton, 2009). Davis defined perceived usefulness (PU) as "the degree to which a

person believes that using a particular system would enhance his or her job performance",

and perceived ease-of-use (PEOU) as "the degree to which a person believes that using a

patiicular system would be free from effort" (Davis, 1989a, pg. 320). Both vm·iables

perceived usefulness and perceived ease-of-use are used in conjunction with one another

21

to formulate the person's, or end user's, intentional behavior towards a technology.

Intentional behavior is defined as "the degree to which a person has formulated conscious

plans to perform or not perform some specified future behavior" (Warsaw, 1985 pg. 214).

The final variable is actual system use and it is essentially the end result of how the end

user did in fact use the system. It is postulated that actual system use is derived in part

from the end user's behavioral intention towards an information teclmology (Davis,

1989b).

Motion and Wiedenbeck (2009) used the TAM in conjunction with the Diffusion of

Innovations Theory to identify and determine which factors contribute to physician

acceptance of an EHR system. They analyzed 239 usable self-reported online

questionnaire responses out of an eligible 802 total physicians at the University of

Mississippi Medical Center in 2007 (Morton, 2009). The study was open for participation

across all thirty one specialties and the online survey was communicated to physicians

via three different email notifications (Morton, 2009). The Like11 scales based survey

used five response levels consisting of the following: Strongly agree, Agree, Neutral,

Disagree, and Strongly Disagree (Morton, 2009). These factors are represented across

three categories: physician characteristics, social factors, and teclmical factors.

Specifically, theses factors included management support, physician involvement,

adequate training, physician autonomy, doctor-patient relationship, perceived ease of use,

perceived usefulness, and attitude about EHR usage. The goal was to measure the factors

of these characteristics for influential attitudes related to EHR use, using the variables PU

and PEOU. Results showed that "Perceived usefulness (PU) had the strongest impact on

22

attitude about EHR use, with physician involvement. Perceived ease of use (PEOU) did

not directly impact attitude about EHR use as hypothesized" (Morton, 2009).

In a follow-up study, Morton and Wiedenbeck (2010) again were measuring which

factors, social or technical, would impact ambulatory EHR adoption attitudes in

physicians. Prior data was utilized and comprised of 23 9 physician responses to a survey

across thirty-one specialties at the University of Mississippi Medical Center (Morton,

2009). Similarly, the specific factors explored in this acceptance study include physician

perceptions of computer skills and training, management support, physician involvement

and participation in the process, physician autonomy, the doctor-patient relationship,

perceived ease of use, and perceived usefulness. Physician attitudes were assessed before

and after EHR implementation in the pre-post fashion. Results revealed perceived

usefulness to be the significant predictor before and after implementation. In addition,

"concerns regarding patient privacy, interference with physician-patient rapport,

workflow, efficiency, and autonomy were found" (Morton, 201 0).

Lastly, the TAM is utilized in conjunction with Task-teclmology Fit Theory by Hyun

(2009). This study uses nurses' perceptions related to their documentation needs in order

to design an effective electronic nursing documentation application (Hyun, 2009). The

study took place at Columbia University College of Nursing and academic teaching

hospital. Methods consisted of a brainstorming sessions with nursing, n= 2, followed by

an: interactive collaboration in system design on paper, next, the user interface was

created and evaluated by nursing, n= 5, and lastly nurses pmiicipating in the study

23

completed surveys related to perceived usefulness and perceived ease of use on a seven

point Like1i scale (Hyun, 2009). Study specific factors were quality, locatability,

authorization, compatibility, ease ofuse/training, production timeliness, systems

reliability, and relationship with users. Results yielded perceptions of ease of use and

usefulness of interface screens for nursing staff. In addition, the interface screens

fulfilled the need for nursing documentation related to Nursing Admission Assessment,

Blood Administration, and Nursing Discharge Summary. By developing the system with

the clinical end user, most documentation needs, described as documentation efficiency

and patient safety as functional requirements, were understood and met thereby leading to

a successful endeavor.

NICU user performance and associated technologies- Integrated Review

Past topics of performance measurement, as it relates to actual infonnation teclmology in

the literature are portrayed in multiple ways. The broadly described components from

which the measurements are comprised, include a number of different technologies,

clinical settings, end users, and related patient outcomes over the past decade. From a

teclmical perspective, the technologies being used are in the form of software, hardware,

or a combination therein with free-standing and/or mobile configurations. The clinical

settings are primarily hospital NICUs of varying size. The clinical setting is focused on

NICU sub-specialty units in hospitals. The end users are generally comprised of primary

clinical staff in the form of physicians, physician residents, and pharmacy staff, and

various nursing staff. Clinical outcomes are comprised of actual patients whose

24

treatment was delivered both with and without the assistance of technology. Provider

outcomes are compromised of how well the user of the technology performed

individually or in aggregate, and lastly, technology outcomes as they relate to effectively

eliminating problems they are implemented to solve. Put simply, the effectiveness of

different technology interventions are described next and are generally measured by the

performance of the end user, the clinical outcome experienced by the patients, or a

combination ofboth.

A study was conducted at Fairfax Hospital NICU to measure the turn around times on

ordering, processing, delivery, and the overall quality of parenteral nutrition (PN)

therapies for neonates (Puangco, 1997). For each PN order, a data collection sheet was

attached and filled out by the ordering physician neonatologist and the processing

pharmacy technologist (Puangco, 1997). After one year time intervals, data was

collected for two week periods before and after the implementation of an in-house

developed interventional software used to automated the process (Puangco, 1997). Mean

differences across both time periods were measured using unpaired t-tests (Puangco,

1997). Thirty nine patients received PN prior to the intervention and forty two patients

received PN after the intervention. Results revealed a decrease of three minutes per order

for neonatologists (Puangco, 1997). In addition, improvements were shown in pharmacy

staff efficiencies and dietician calculations which played a pmi in increasing the nutrient

quality and thus energy levels of neonate patients (Puangco, 1997).

25

Another study focused on PN en-or prevention and teclmology user satisfaction at Johns

Hopkins Hospital NICU (Lehmann, 2002). An order entry software program was in­

house developed to replace the paper based methodology. This was coined the total

parenteral nutrition (TPN) calculator. This instrument served as a quality improvement

and was designed in conjunction with the decade old existing paper form. Data collection

for the baseline period consisted of forty three days, with the intervention period lasting

46 days (Lelunmm, 2002). The data elements were defined as the number of nutrition

orders, frequency and en-or type. Afterwards, a personal experience and opinions survey

was conducting online to physicians, pharmacists, and nurses. In the baseline period 557

orders were compared with 471 orders from the intervention period (Lehmmm, 2002).

Results showed a 61% en-or reduction rate, in that for every 1 00 TPN orders, m1 average

of 10.8 were errors prior to the intervention compm·ed with 4.2 after the intervention

(Lehmmm, 2002). The survey results revealed that the TPN calculator was scored as a

1.5, or easy to use based on a numeric ranking scale of 1-5 with one being the easiest

(Lehmann, 2002). A Likert scale of strongly agree, agree, neutral, disagree, agree was

used to measure how the end users liked the paper form compared with the TPN

calculator (Lehmmm et al., 2002).

Further, Physician residents at the University of Washington NICU were evaluated for

performance with clinical documentation software while using a Personal Digital

Assistant (PDA) in order to improve accuracy and consistency (Cm-roll, 2004). The study

duration was eight months, consisting of a base time period four months prior to the

implementation and four months after, with forty days being randomly selected and taken

26

into consideration for analysis. Measures included patient weights, medications and

vascular lines documented in the progress note that did not match pre-defined standards

from nursing flow-sheets, assessments and medication administration records collected

prior (Carroll, 2004). The baseline period evaluated 339 progress notes and the

interventional period evaluated 432 progress notes. Logistic regression yielded initial

increases, but once covariates were controlled, there were varying results in that PDA

related documentation for patients weights was found to have less discrepancies, while

there was no substantive change for medications and vascular lines (Carroll, 2004).

Ultimately, the PDA's were removed from the NICU and they moved back to the

baseline documentation, while recommending more research in the use ofPDAs was

necessary.

As it relates to clinical outcome based performance measures, The Ohio State University

Medical Center NICU implemented Computerized Physician Order Entry (CPOE) in

their NICU in an effo1i to reduce medication error rates and improve medication turn

around times. Data variables included medications of caffeine and gentamicin. Both the

baseline and intervention samples contained 1 00+ very low birth weight infants, with the

study ranging from six months prior to six months after implementation (Cordero, 2004).

Statistical analysis were comprised of descriptive statistics (frequency counts,

percentages, mean and standard deviations), unpaired t-tests, and chi-square tests. Results

showed significant decreases in both medication rates and turn around times attributed to

the CPOE software functionality (Col'dero, 2004). Average turn around times decreased

fi·om 10.5 hours to 2.8 hours and were statistically significant, p<.Ol. Specifically, the

27

volume of neonates receiving medications within the 2 hour and three hour ranges

increased from 10% to 35% and 12% to 62% (Cordero, 2004). As it relates to medication

errors, the baseline period accounted for 13% dosing mostly due to overdoses and

underdoses (Cordero, 2004), compared to the interventional period where all medication

dosing errors were non-existent (Cordero, 2004).

During 2004 and 2006, a study at the Madigan Army Medical Center (MAMC) NICU

was conducted in accordance with the implementation of a CPOE system (Taylor, 2008).

The study measured the medication administration variances over a period of 21 months

with the orders being collected by hand for the first twelve months and then using the

interventional CPOE system for the remaining nine months (Taylor, 2008) . The

frequency of variance during both periods was identified through an representative

observational study whereby research nurses actively watched the nursing participants

administer medications to patients (Taylor, 2008). Thi1iy three nurses out of forty two

total participated in the study and over 250 medication administrations were observed in

both time periods (Taylor, 2008). Results showed that prior to the implementation of

CPOE, there were 19.8% variances detected during medication administrations and

afterwards this decreased to 11.6% (Taylor, 2008). Early or late administrations, defined

as those given 60 minutes early or late, comprised 53.1% of all variances (Taylor et al.,

2008).

Another notable measure of performance is described as system performance, as

portrayed in a study measuring self reported medication errors in the MEDMARX

28

anonymous national database reporting tool for hospitals. NICU enors related to

computer entry systems and more advanced CPOE systems were queried with a total of

343 results across 48 hospitals found (Chuo, 2008). Of those enors, 248 were found in

the computer entry systems, while 45 were attributed to CPOE (Chuo, 2008). The

associated phases of the medication process in which the enor occuned consisted of

procurement, prescribing, transcribing/ documenting, dispensing, administering, and

monitoring. Over sixty percent of the errors in the computer entry systems can be linked

to the transcribing/ documenting phase, while over seventy five percent of enors can be

tied to the prescribing phase in CPOE (Chuo, 2008). Of these errors, the most influential

error types were comprised of improper dose/ quantity at 36.2% in the computer based

entry system and prescription error at 62.2% in the CPOE system (Chuo, 2008).

In2009, Rikli performed a qualitative study at Helen DeVos Children's Hospital NICU

centered around the concept of using micro systems and quality improvements to

successfully implement an electronic documentation system following a failed attempt.

Microsystems are described as any participant involved in the healthcare of a patient.

Thus clinical linkages within departments amongst providers and patients are significant

when trying to problem solve. After the failed implementation attempt two years prior,

staff was surveyed about problems and what was needed to be successful. Of the

findings, technical problems in terms of access to workstations and speed of computers

were identified, along with the need for extended practice time and more experienced

trainers to help them use the software program (Rikli, 2009). An interdisciplinary quality

29

team was formed and monitored the progress throughout the second project

implementation which lead to a successful endeavor (Rikli, 2009).

Lastly, a lesson's learned miicle related to the implementation of CPOE at Mission's .

Children's Hospital NICU. The NICU is comprised of a 50 bed unit and was chosen as

the pilot site (Ramirez, 201 0). The critical aspects of the implementation described relate

to clinical team building and communication, physician leadership, sequencing every

aspect of each order set, direct access to programmers for system changes, m1d real time

testing scenarios with self paced training (Ramirez, 201 0). These lessons learned lead the

NICU to a successful implementation of CPOE along with a successful impact on patient

cm·e and safety (Ramirez et al., 2010).

In summary, the quality of care patient safety movement is transforming like never before

in the field ofhealthcare. This focus is easily seen through the spurred implementation of

electronic health records, the evolution of performance and quality of care standards, and

lastly, the linkages with direct reimbursement for healthcare providers. Thus, the

importance ofunderstm1ding how to become successful users ofteclmology across the

healthcare spectrum has never been greater.

IV. Conceptual models

For the study, I am going to utilize the core conceptual model of TAM (see Figure 1a),

described as a measure of end user teclmology acceptm1ce through two subjective

vm·iables, perceived usefulness and perceived ease of use (Davis, 1989). Again,

30

behavioral intention to use a system is comprised of these variables and relates

specifically to .actual system use (Davis, 1989b ).

Figure la: Core Conceptual Model- TAM

Perceived Usefulness \ Behavioral

v Intention to Perceived Use Ease ofUse

Figure lb: Core Conceptual Study Model

Perceived Usefulness

MAK Objective Performance Score (T-18m)

Behavioral Intention to Use

Actual System Use

Actual System Use Technology 2 (Cribnotes t+6m)

More specifically, I will be looking at the acceptance of an electronic health record

technology, Cribnotes, through the perspective ofNICU nursing staff by measuring not

only the traditional TAM subjective variables of perceived ease of use and perceived

usefulness by individual nurses, but also utilizing objective performance data of the same

nurses on a past information technology, Medication Administration Checking (MAK)

(see Figure 1b) (Grand Rounds Software, 1994; Siemens, 2002). Additional independent

31

variables of nursing staff measured will be age, gender, race, professional degree, and job

title. Lastly, actual system use of the Cribnotes technology will serve as the dependent

variable and will be measured by objective performance data. This is depicted in

Appendix 1- Figure 2 in detailed fashion. I am interested in the value of measuring my

conceptual model across one year of time, as users' attitudes and behaviors may change

over time. The first measure will encompass a time period of six months prior to the

implementation of the Cribnotes system and the second measure will be six months after

the system is implemented. My hypothesis states'that MAK objective nursing

performance data will be the primary predictor, or will have the most impact, of the

Cribnotes objective nursing performance data. If this holds true, the TAM as it stands

today needs revision, either to 1) include the objective use data variable in an expanded

model, or 2) to eliminate the need for the TAM and rely solely on the objective use data

as a stand-alone predictor.

V. Research Questions and hypotheses

Research questions for this study include: 1) Will RN historic system performance be a

better predictor of future system use than utilizing their attitude, or behavioral intention to

use it? 2) Will RN historic system performance work in conjunction with RN behavioral

intentions to explain more R squared variability together, rather than independently? 3)

Were clinical, social, and/or teclmical factors identified that can be added as subjective

variables to enhance future analysis ofR squared findings related toRN performance?

32

HI: The MAK objective nursing performance score will be the primary predictor, or will

have the most impact, on the Cribnotes objective nursing performance score when

compared individually with all other independent variables.

H2: The MAK objective nursing performance score will be the primary predictor, or will

have the most impact, on the Cribnotes objective nursing performance score when

compared with all other independent variables in aggregate.

H3: The MAK objective nursing perfonnance score will work in conjunction with RN

behavioral intentions to create a more explanatory framework.

H4: The full model consisting ofMAK objective nursing performance score and the pre­

implementation subjective RN behavioral intentions will explain all the variation in the

dependent variable, Cribnotes objective performance score.

VI. Methods and Materials

This study uses quasi-experimental research design with data being collected before the

implementation of a new technology and six months after the implementation.

The study is quantitative and spans across two time periods in a longitudinal fashion

within the course of one year. Other characteristics are relevant in that the approach of

the study is pre-post in nature, as the researcher is describing and measuring both

subjective behavioral attitude scores utilization patterns across two different types of

technologies over time. Data is already being captured as part of system functionality so

the costs of the study are minimal.

33

The researcher will assign nursing performance scores for specific, time sensitive

variables within each system. The performance scores will be measured across time

periods throughout the year relative to the Cribnotes Go-Live date of July 11, 2011 (T),

begilming with the use of historic MAK performance scores in January, 2010 (T-18m),

and then 18 months later upon the intervention capturing the Cribnotes performance

scores through January, 2012 (T+6m). Cribnotes perfonnance scores will be measured at

time periods T +6m. In addition, TAM based nursing surveys were given to the

participants prior to the implementation of Cribnotes in order to gain an understanding of

their behavioral intentions towards these current and future technologies. In so doing, the

users' behavioral intentions can be measured independently and in conjunction with the

perfom1ance scores of the MAK technology in an effmi to predict and explain nursing

utilization pattems of the Cribnotes technology. Additionally, the same survey, with

slight modifications adapting it to past tense, will be administered a second time to the

same nursing staff six months post-implementation of Cribnotes. Figure 3 depicts the

time periods of the before and after design study.

34

Figure 3- Study Design

Time

T- 18 months 0 T+ 6 months

(Pre-objective data-) (Post- objective data) MAK Cribnotes (Pre-subjective data) (Post-subjective data) Post-MAK and Pre-CribnotesPost-MAK and Post-Cribnotes

Study Group Variables Objective (MAK) Objective (Cribnotes) Subjective (Both) Objective (Cribnotes Exp.)

Study Setting

01

01

X X X

02 02 02

This study is being performing in an effort to model NICU nurse utilization of electronic

health record documentation technologies, one of which is currently used hospital wide,

including NICU, and the latter, to be utilized as a specialization system only within the

NICU. The NICU's current module of documentation teclmology, more specifically,

medication administration checking at the point of care (MAK), will continue to be used

in conjunction with a new modular technology, Cribnotes, a nursing documentation

system, as the NICU moves to a comprehensive electronic· health record. This pre-post

study will use breast milk verification/administration times by nurse as the variable of

measurement. The breast milk variable was selected because it is a nationwide quality

35

goal and is· indicative of a key NICU measure directly tied to nursing documentation

performance (AAP, 2012). This variable is used in the MAK system for the first six

months of the study, and then is used in the Cribnotes system for the remaining six

months of the study. Both of the applications, although developed by different

companies, maintain a modem user interface related to aesthetics, and very similar

functionality in terms of measurements ofbreast milk verification/ administration as

quality checks.

IRB approval

The University of South Alabama Institutional Review Board approval was requested and

granted prior to·this human-subjects research study. University of South Alabama

Children's and Women's Hospital nurses participating in the study did so at their

discretion and could leave the study at any time.

Sample

The sample population consists of the NICU nursing staff at the University of South

Alabama Children's and Women's Hospital, Department of Pediatrics, Division of

Neonatology. The sample size consisted of 146 potential participants whom were

requested to be a part of the initial study. The initial participation level prior to the

implementation of Cribnotes consisted of 95 out of the 146 potential participants or 65%

response rate. The sample size from the second administration of surveys six months after

the implementation of Cribnotes consisted of 14 7 potential participants. Of this group

130 out of the 147 or 78% chose to pmiicipate. Those whom pmiicipated in the original

36

survey were all included as potential participants for the second survey. Their response

rate was 81 out of the initial 95, with 14 of the participants leaving the study due to

reasons such as they simply didn't want to fill out another survey, were out on maternity

leave, or were no longer employed by USA Children's and Women's Hospital NICU.

Upon review, an additional 12 participants were un-eligible for the study because they

had not utilized the MAK system. Therefore, there were a total of 69 study participants.

The nursing staff participants identified were kept confidential, thus minimizing the

study's risk, and have been re-coded with an assigned number to track their performance

throughout the stuqy.

Data Sources

Data sources for this study are comprised of two separate clinical application system

databases and survey data from the NICU Nurse Survey instrument. The MAK system is

primarily used to perform quality checks on medications administered and document all

associated information electronically. It serves as a critical module for a comprehensive

EHR and is utilized hospital-wide by nurse staff. The data from the MAK information

system is collected real-time and is currently utilized by the NICU Nurse Manager to

assess nursing performance as it relates to patient safety and quality of care. The data is

also utilized by the hospital's Department of Quality Management for JCAHO purposes.

The Cribnotes information system captures additional data fields unavailable 1n MAK

that were historically recorded in paper format. Upon the implementation of Cribnotes,

the NICU will be capturing information related to nursing documentation in the areas of

assessments, nursing diagnosis, goals or expected outcomes, nursing actions or

37

interventions, evaluations, I/Os, and overall care plan. Cribnotes will serve as the

comprehensive EHR for the NICU, to be used in combination with the MAK system.

Similarly, reports from the system will be utilized in the same fashion. The survey

instrument will measure intended behavior of the nursing staff as it relates to utilizing

both technologies, specifically, ease of use, described as the impact on nursing workload,

and usefulness, described as the quality of patient care.

Data Source 1- MAK Database

The first database used in the study is the MAK database, which houses patient

information directly related to point of care medication administration performed by

NICU nursing staff. The data for medication administration delivery times is tied directly

to each nurse user within the information system. The primary variable to be analyzed

will be breast milk verifications/administrations given by the nurse to the patient. The

researcher will analyze this data by organizing and presenting the information by nurse

user that will be the basis for individual nursing performance scores. The primary MAK

variable will be measured by verification/ administration times that were either delivered

"on-time" or "early/ late".

Data Source'2- Cribnotes Database

The second database will be that of Cribnotes, which began on July 11, 2011 and will

collect all data input into the system. Again, all the nursing documentation performed by

users will be directly linked to them individually. This will allow the data to be tracked

by the researcher, in order to identify quality metrics that will be the basis for individual

38

nursing performance scores. The primary variable to be analyzed will be breast milk

verifications/administrations given by the nurse to the patient. The data will be organized

into a report through access to the Cribnotes SQL database. The primary Cribnotes

variable will be measured by verification/administration times that were either "on-time"

or "delivered early/ late".

Data Source 3- Pre and Post Survey Instrument

Please see Appendix 2 to see the survey instruments used in this study. The pre­

implementation survey instrument consists of basic demographic information such as,

age, gender, race, professional degree, position title. In addition, the survey contains four

questions that gage the nurses' perception of how the technology would affect their

. workload and quality of care given to patients. Questions 1 and 3, point specifically to the

ease of use of each technology as they relate to the MAK technology's current impact on

nursing workload, as well as the expectation of future impact of the Cribnotes

technology. In addition, the remaining questions (2 and 4) are directly related to the

MAK technology's current usefulness of improving quality patient care, as well as, the

expectation of future impact ofthe Cribnotes technology.

The post-implementation survey instrument consists of basic demographic information

such as, age, gender, race, professional degree, position title. The survey contains the four

same questions as the pre-survey correcting for pre/ post tenses and asks the nurse's level

of agreement with how the two technologies affected their work load and quality of care

given to patients. A comments section was added in an effort to allow nurses to provide

39

feedback about their experience over the past six months using the MAK and Cribnotes

systems. The comments section was left open intentionally in order to not push the RNs

to an answer, but rather allow them to speak freely after having experienced the systems.

Also taken into consideration was the impact of nurses being somewhat restricted in

answer choices when filling out the agree/disagree portions of the survey, and thus a

broad approach was instituted to capture anything on their mind about their experience.

The survey was self administered and completed in the NICU during the patiicipants'

work shift. Confidentiality was maintained in that I collected the surveys from the

participants throughout 2 different shifts (12 hr shifts; 7atn-7pm; and 7pm-7am) and

stored them in a locked drawer in a locked room, in a separate secure building. The

surveys were then input into excel, de-identified, and m·e backed up and stored on a

secure server. The survey was a stand alone, not linked to other data sources. I am able to

link the surveys and the system use data (MAK and Cribnotes breast milk administration

documentation reports) to the individual nurses by their names, which have been de­

identified/ changed to numbers for confidentiality purposes.

Data Collection Time Intervals

Data analysis on the defined data sources will take place with the primary reference point

of "Cribnotes Go-Live" (Time period = T) once the intervention system has been

implemented a11d users begin working on the system. The Cribnotes system was

implemented on July 11, 2011. A two year time period measured as snapshots of nursing

utilization at averaged time periods will be at T- 18 months for the MAK system

40

specifically, and Cribnotes at T+6 months. As it relates to the pre-implementation survey,

the data collection time period spanned a 15 day timeframe that took place prior to the

Cribnotes Go-Live (Time period= T). The post-implementation survey was also

administered over a fifteen day timeframe that took place six months after the Cribnotes

Go-Live.

Intervention

The primary intervention used for this study is the implementation of Cribnotes

information system. This electronic documentation system will serve as the second

significant piece of patient documentation software to be adopted by NICU nurses. This

intervention will be used to test hypothesis H1, H2, H3, and H4.

Scoring Calculation

The measurements for each technology will begin with deriving an objective and

subjective performance scoring methodology for MAK and Cribnotes. This study will

measure these NICU nursing perfmmance scores over time. The objective use data is

specific to measuring nursing documentation performance and will be analyzed across

both systems. Each system provides modular functionality as it relates to a

comprehensive electronic health record for the NICU and as such, are used in conjunction

with one another to assist in providing specialty patient care. For purposes of this study,

the variable to be measured within the MAK system will be breast milk

verification/administration times, which shifts from MAK after the Cribnotes intervention

and is then measured in the Cribnotes system for the second period. The breast milk

41

quality measure is a nationwide goal and is indicative of a key NICU measure directly

tied to nursing documentation performance (AAP, 2012).

The scoring methodology related to the documentation performance ofMAK medication

administration times is comprised of nursing timeliness of breast milk

verification/administration given by assigning a two tiered point system for being either

"on-time" or "early/late". The data will consist of all verifications/administrations by

each RN within the base period (T -18 months). Each nurse breast milk

verification/administration will be assigned a "1" or a "0". A score of a "1" represents the

breast milk being verified/ administered on time. "On time" is defined as a medication

being verified/ administered less than thirty minutes early and not more than thi1iy

minutes late. A "0" will be assigned for all medication verification/administration times

where medications were given beyond this one hour threshold and were thus early or late.

For measurement purposes, the scoring system's associated calculation will be based on

the outcomes related to the proportion oftimes the nurse verified the medication on-time

divided by the total number of times they performed the task.

The scoring methodology for nursing documentation performance, related to the new

Cribnotes system, will align with that fi:om the MAK system in that the primary variable

to be measured will be breast milk verification! administration documentation times that

resulted in a status of being "on-time" or "early/late". The data will consist of all

verifications/administrations by each RN in the interventional time period (T+6 months).

Each nurse breast milk verification/administration will be assigned a "1" or a "0". A

42

score of a "1" represents the breast milk being verified/ administered on time. "On time"

is defined as a medication being verified/ administered less than thirty minutes early and

not more than thirty minutes late. A "0" will be assigned for all medication

verification/administration times where medications were given beyond this one hour

threshold and were thus early or late. For measurement purposes, the scoring system's

associated calculation will be based on the outcomes related to the proportion of times the

nurse verified the medication on-time divided by the total number of times they

performed the task.

The survey scoring methodology applied to both the pre and post surveys will also follow

a "1" and "0" assignment. On the baseline, pre-implementation survey, the first survey

question, "Has the implementation ofMAK increased your nursing workload?" will be

assigned a "1" if the participant agrees and a "0" if the participant disagrees. The second

survey question, "Has the utilization ofMAK improved quality of care for patients?" will

be assigned a "1" if the participant agrees and a "0" if the participant disagrees. The third

survey question, "Has the implementation of CRIBNOTES increased your nursing

workload?" will be assigned a "1" if the participant agrees and a "0" if the participant

disagrees. The fourth survey question, "Has the utilization of CRIB NOTES improved

quality of care for patients?" will be assigned a "1" if the participant agrees and a "0" if

the participant disagrees.

On the post-implementation survey, the first survey question, "Has the implementation

ofMAK increased your nursing workload?" will be assigned a "1" if the patiicipant

43

agrees and a "0" if the participant disagrees. The second survey question, "Has the

utilization of MAK improved quality of care for patients?" will be assigned a "1" if the

pruiicipant agrees and a "0" if the participant disagrees. The third survey question,

"Expectation that CRIBNOTES will increase your nursing workload?" will be assigned a

"1" ifthe pruiicipant agrees and a "0" ifthe participant disagrees. The fornih survey

question, "Expectation that CRIBNOTES will improve quality of care for patients?" will

be assigned a "1" if the participant agrees and a "0" if the participant disagrees.

Analytical Approach

Data elements representing each of the three aforementioned data sources as they relate

to the general and specific theoretical constructs are shown below in Appendix 3- Tables

1, 2, and 3. Linear regression analysis (R-squared), correlations and descriptive statistics

will be utilized. The dependent variable (Y) will be represented as a proportion of times

RNs verified/ administered breast milk "on-time" divided by the total number of times it

was verified/ administered. An linear regression analysis will then be conducted to

measure the independent variables 1 (B 1 ), described as Technology 1 (MAK) actual use

performance scores, and independent variables 2, 3, 4, a11d 5 (B2, B3, B4, and B5),

described as Nursing Survey attitudes independently and in combination against the

dependent variable, described as Teclmology 2 (Cribnotes) actual use performance

scores. In addition, independent variable (B6) describing individual RN frequency, or the

number of times, breast milk verification/ administration occurred in time period T+6.

The model will utilize the full sample of 69 RNs.

44

Additional independent variables collected by the surveys include age, gender, race,

p1~ofessional degree, and position title. These demographic independent variables will be

examined using descriptive statistics in an effort to justify their relevance while

identifying any confounding impacts. Further, of the survey respondents, those eligible to

participate in the study and non-respondents will be compared in aggregate to gain a

better understanding of the NICU departmental demographics and adjust for any impacts

that may be realized.

Statistical Equation

Ordinary Least Squares (OLS) Regression Analysis

Y (t+6 months)= bo + biXt(t-IS) + b2X2(t) + b3X3(t) + b4X4(t) + bsXs(t) + b6X6(t)

t-Is= cribnotes pre-implementation time period of eighteen months

t+6= 6 months post cribnotes implementation time period

y = Cribnotes Performance Score

XI= MAK objective use historic performance score

x2 =Study participant agreed (1) or disagreed (0) with MAK ease of use

x3 = Study pmiicipant agreed (1) or disagreed (0) with MAK perceived usefulness

X4 =Study pm·ticipant agreed (1) or disagreed (0) with Cribnotes ease of use

x5 =Study participant agreed (1) or disagreed (0) with Cribnotes perceived usefulness

x6 = RN frequency of breast milk verification/ administration in time period T +6

As it relates to the open ended comments section of the post-implementation survey,

results will be qualitatively evaluated through the use of content analysis. Each comment

45

will be characterized independently. This qualitative analysis will be performed from the

perspective of and relevant to management decision making and technology use. This

effmi will be unde1iaken to better understand the problems that need to be addressed, if

any, and which variables could be incorporated in a follow-up study in terms of adding

more measures to increase understanding of variance.

Data elements used to represent theoretical constructs

Constructs of interest and how they will operationalize (Confounders, IV, DV) (See

Appendix 3- Tables 1, 2, and 3).

VII. Study Results

In order to begin to address the study's research questions, a number of related

quantitative and qualitative statistical analyses were conducted. Results related to study

questions 1 and 2 were achieved tlu·ough quantitative analysis, while study question 3

relied on qualitative analysis. The approach to the quantitative analysis was to first,

explain the study sample and the composition of each study pruiicipant by analyzing the

demographic variables age, gender, race, professional degree, and position title. The

second approach was to describe the distributions of the pre-post implementation

subjective and objective variables related to the dependent variable. Third, correlations

amongst the subjective variables were also conducted. Lastly, linear regression analysis

was performed on all independent variables in order to compare the strength of their

relationships when explaining the varim1ce in the dependent variable. The approach to

46

the qualitative analysis was to first discuss the number ofresponses related to the entire

sample. Second, content analysis based on inductive reasoning was used for coding study

pruticipant comments obtained from the post-implementation survey. Third, the coded

statements were categorized in a hierarchical fashion in order to identify trends in

positive or negative statements related to the user experience.

Quantitative Analysis

Sample

The study population consists of the NICU nursing staff at the University of South

Alabama Children's and Women's Hospital, Department of Pediatrics, Division of

Neonatology. There werel46 potential pru·ticipants recruited to be a part of the initial

study. The initial participation level before the implementation of Cribnotes consisted of

95 out of the 146 potential participants or 65% response rate. The participation level from

the second administration of surveys six months after the implementation of Cribnotes

consisted of 147 potential participants. Of these, 130 out or 78% chose to pruticipate.

Those who participated in the original survey were all included as potential participants

for the second survey and 81 out of the initial 95 respondents pmticipated in the post­

implementation survey. Reasons for dropping out included: Didn't want to fill out

another survey, were out on maternity leave, or were no longer employed. An additional

12 participm1ts were removed from the study because they had not used the MAK

information system, and thus did not have any scores to measure that were necessary for

47

purposes of the study. The final sample consisted of 69 NICU nursing staff for which

there was complete data (see Table 4).

Table 4: Study participant pre-post response composition

Pre-Implementation Post-Implementation Total Total

Responses Surveyed Responses Surveyed 95 (65%) 146 130(78%) 147

Participated in both Pre-Post Implementation Surveys 81 (55%) 146 81 (55%) 147

Actual Use Pre-Post performance scores T-18 months 69 (47%} 146 69 (47%) 147

Table 5 presents descriptive statistics of study pmiicipants, including age, gender, race,

professional degree, and position title. Age ranged from 25 to 65 yem·s old with a mean of

41 years old and a standard deviation of 10.079. Ninety-five percent ofthe sample was

female. Education level ranged from 68 (98.6%) RNs and 1 (1.4%) LPN, with 7

education levels ofRNs comprised of 1) RN-ADN, or Associate Degree in Nursing; 2)

RN-ADN-RNC, or Associate Degree in Nursing with an additional specialized nurse

training ce1iificate; 3) RN-ADN-RRT, or Associate Degree in Nursing with additional

training and participation on the rapid response team; 4) RN, Registered Nurse with no

specification of Associate or Bachelor degree; 5) RN-BSN, Bachelor of Science Degree

in Nursing; 6) RN-BSN-MSN, or Bachelor of Science Degree in Nursing and Master of

Science in Nursing; and 7) RN-BSN-RNC, or Bachelor of Science in Nursing with an

additional specialized nurse training certificate. Lastly, there were three categories related

to position title. These include: 1) LPN II-StaffNurse; 2) RN-StaffNurse; and 3) RN-

StaffNurse-Working Supervisor.

48

Table 5: Study participant demographic composition

Pre-1m :>lamentation Post-Implementation Eligible Total Survey Eligible Total Survey Responses Responses Responses Responses

Total 69 (47.2) 95 69 (47) 136 Gender M 2 (2.9) 3 (3.2) 2 (2.9) 5 (3.7) F 66 (95.7) 91 (95.8) 66 (95.7) 128 (94.1)

Race Ethnicity White 61 (88.4) 82 (86.3) 61 (88.4) 114 (83.8) White/ Hispanic 1 (1.4) 1 (1.1) 1 (1.4) 1 (.7)

0

Asian 1 (1.4) 1 (1.1) 1 (1.4) 1 (.7) Black 2 (2.9) 6 (6.3) 2 (2.9) 11 (8.1) American 1 (1.4) 1 (1.1) 1 (1.4) 1 (.7) Other 1 (1.4) 1 (1.1) 1 (1.4) 1 (.7)

Professional Degree LPN 1 (1.4) 1 (1.1) 1 (1.4) 2 (1.5) RN 7(10.1) 12 (12.6) 7(10.1) 16 (11. 7) RN-ADN 15 (21.7) 21 (22.1) 15 (21. 7) 28 (21) RN-ADN-RNC 2 (2.9) 2 (2.1) 2 (2.9) 3 (2.2) RN-ADN-RRT 1 (1.4) 1 (1.1) 1 (1.4) 1 (.7) RN-BSN 32 (46.4) 42 (44.2) 32 (46.4) 68 (50) RN-BSN-MSN 2 (2.9) 3 (3.2) 2 (2.9) 3 (2.2) RN-BSN-RNC 7 (10.1) 11 ( 11 .6) 7 (10.1) 14(10.3)

Job Title LPN 11-Staff Nurse 1 (1.4) 1 (1.1) 1 (1.4) 1 (1.4) RN-Staff Nurse 66 (95.7) 86 (90.1) 66 (95.7) 129 (94.9)

RN-Staff Nurse-Working Supervisor 2 (2.9) 6 (6.3) 2 (2.9) 6 (4.4)

Age N 64 88 64 126 Mean 40.91 39.67 40.91 39.76 Standard Deviation 10.079 10.22 10.079 11.03 Min 25 25 25 23 Max 65 65 65 65

49

Multivariate Analysis

Subjective Use

Survey variables from pre-implementation and post-implementation of Cribnotes were

analyzed. Table 6 presents the distribution frequencies for each variable in terms of

whether or not the study participant agreed or disagreed.

Table 6: Survey response distribution frequencies

Pre-study Post-study Agree Disagree Agree Disagree 57 12 42 27

MAK perceived ease of use (82.6%) (17.4%) (60.9%) (39.1 %) 45 24 48 21

MAK perceived usefulness (65.2%) (34.8%) (69.6%) (30.4%) 60 8 42 27

Cribnotes perceived ease of use (88.2%) (11.8%) (60.9%) (39.1%) 29 33 30 38

Cribnotes perceived usefulness (46.8%) (53.2%) (44.1%) (55.9%)

All survey variables were then independently measured against the dependent variable

and represented through the use of histograms (Figures 4 and 5). The histograms resulted

in negatively skewed distributions as the Cribnotes performance scores generally ranged

from .4 to 1, or scores of 40% to 1 00%.

50

Figure 4: Histograms- Pre-survey Independent Variables with Cribnotes

Performance Score

12"

1o-

a- /, .. 1\.

4

noA{ >

~

:I: ~ s: ;:: ~

~ ~ '"0 '"0 ~ ~ In In ~ ~

< ~ . > '< '<

10

8

m ~ m ~ 2-!l. ~ c c: ~

. ;;;· . • c1l g 4

> n n

.020 0.00 0.20 0.40 0£0 0,80 1.00 120

Crlbnotes Performance Score Cribnotes Performance Score

Cribnotes Periormance Score Cribnoles Performance Score

51

Figure 5: Histograms- Post-survey Independent Variables with Cribnotes

Performance Score

~~========~~~~~~~~====~ .t

-0.20 0.20 0.40 0,60 0.80 1.00 1.20

Cribnotes Performance Score

-0.20 0.00 0.20 0.40 0.60 0,80 1.00 1.20

Cribnotes Performance Score

Crosstabs

-D.20 0.00 0.60 0.80 1.00 1.20

Crlbnotes Performance Score

Cribnotes Performance Score

Survey response variables were correlated with one another in three different scenarios

using six combinations (Tables 7, 8, and 9). For purposes of this study, the variable of

interest measured within the MAK system was breast milk verification/administration

times, which shifts from MAK after the Cribnotes intervention, and is then measured in

the Cribnotes system for the second period. In other words, the nurses performed breast

52

milk verification qmi.lity checks first using MAK, and then using Cribnotes. The breast

milk quality measure is a nationwide goal and is indicative of a key NICU measure

directly tied to nursing documentation performance (AAP, 2012).

The first scenario is described as the relationship between nursing perceptions on "ease of

use," or increased workload, and "perceived usefulness," or improved quality of care,

related specifically to the old system (MAK) (Table 7). Nursing perceptions directly

related to MAK workload increase across both time periods of the study were compared

both with and without the breast milk variable of interest. Meaning more specifically,

that prior to the intervention, the breast milk verification was included in MAK and post­

intervention it was not, because it was moved to Cribnotes. Chi-Square Tests revealed a

Pearson Chi-Square value of association at 4.624, p = 0.032. As it relates to effect size,

ranging from -1 to 1, Phi= .259, p = .032, represented a moderately strong association

between pre-post MAK ease of use. When both pre- and post implementation periods are

compared, nurses who perceived the system to be easy to use before they used it were

more likely to perceive it as easy to use after they had used it.

Nursing perceptions directly related to MAK quality of care improvement across both

time periods of the study were compared both with and without the breast milk variable

of interest. Chi-Square Tests revealed a Pearson Chi-Square value of association at 9.789

and a significance of .002. As it relates to effect size, ranging between -1 to 1, Phi= .377,

p = .002, represents a moderately strong association between pre-post MAK perceived

usefulness. When both pre- and post implementation periods are compared, nurses who

53

perceived the system to be useful before they used it were more likely to perceive it as

useful after they had used it.

Table 7: Pre-Post Survey MAK Perceived Ease of Use and Usefulness Crosstabs

Post-survey MAK Ease of Use Agree Disagree Total

Pre-survey MAK Perceived Ease of Use Agree 38 19 57

Disagree 4 8 12 Total 42 27 69

Pearson Chi Square 4.624 p 0.032 Phi 0 .. 259

Post-survey MAK Perceived Usefulness Agree Disagree Total

Pre-survey MAK Perceived Usefulness Agree 37 8 45

Disagree 11 13 24 Total 48 21 69

Pearson Chi Square 9.789 p 0.002 Phi 0.377

The second scenario is described as the relationship between nursing perceptions on

"ease of use," or increased workload, and "perceived usefulness," or increased quality of

care, related specifically to Cribnotes (Table 8). Nursing perceptions directly related to

Cribnotes expected and actual workload increases across both time periods of the study

were compared. Chi-Square Tests revealed a Pearson Chi-Square value of association at

8.651 and a significance of .003. As it relates to effect size, ranging from -1 to 1, Phi=

.357, p = .003, represents a moderately strong association between pre-post Cribnotes

ease of use. When both pre- and post implementation periods are compared, those nurses

54

who expected the new system to be easy to use before they used it were more likely to

perceive it as easy to use after they had used it.

Nursing perceptions directly related to Cribnotes expected and actual perceptions on

improved quality of care for patients across both time periods of the study were

compared. Chi-Square Tests revealed a Pearson Chi-Square value of association at 7.044

and a significance of .008. As it relates to effect size, ranging between -1 to 1, Phi= .340,

p = .008, represented a moderately strong association between pre-post Cribnotes

perceived usefulness. When both pre- and post implementation periods are compared,

those nurses who expected the system to not be of use before they used it were more

likely to perceive it to be un-useful after they had used it.

Table 8: Pre-Post Survey Cribnotes Perceived Ease of Use and Usefulness Crosstabs

Post-survey Cribnotes Ease of Use Agree Disagree Total

Pre-survey Cribnotes Perceived Ease of Use Agree

Disagree Total

Pearson Chi Square 8.651 p 0.003 Phi 0.357

Pre-survey Cribnotes Perceived Usefulness Agree

Total

Pearson Chi Square p

Phi

Disagree

55

7.044 0.008 0.340

40 20 60 1 7 8

41 27 68

Post-survey Cribnotes Perceived Usefulness Agree Disagree Total

18 10 28 10 28

23 33

33 61

The third scenario is described as the relationship between nursing perceptions on "ease

of use," or increased workload, and "perceived usefulness," or improved quality of care

to the patient, both after having used each respective system (Table 9). Put simply, the

nurses that used MAK before Cribnotes implementation report on whether or not their

workload and/or quality of care increased, and then report on Cribnotes in the same

manner post-implementation. Chi-Square Tests revealed a Pearson Chi-Square value of

association at 2.249 and a significance of0.134. As it relates to effect size ranging

between -1 to 1, Phi = .181, p = .134, represents the strength of association as a moderate

relationship between pre-survey MAK and post-survey Cribnotes ease of use, however,

insignificant. Even though, when both pre- and post implementation periods are

compared, nurses who used the old system and found it easy to use were more likely to

find they new system easy to use after they had used it.

Similarly, nursing perceptions represented by the variable "perceived usefulness" or

quality of care increase were compared after the individual implementations ofMAK and

Cribnotes. Chi-Square Tests revealed a Pearson Chi-Square value of association at 5.499

and a significance of .019. As it relates to effect size, ranging between -1 and 1, Phi=

.284, p = .019, represented a moderately strong association between pre-survey MAK and

post survey Cribnotes perceived usefulness. When both pre- and post implementation

periods are compared, nurses who used the old system and found it useful were more

lik~ly to find they new system as useful after they had used it.

56

Table 9: Pre-Post Survey MAK and Cribnotes Perceived Ease of Use and Usefulness

Crosstabs

Pre-survey MAK Perceived Ease of Use

Total

Pearson Chi Square p Phi

Pre-survey MAK Perceived Usefulness

Total

Pearson Chi Square p Phi

Objective Use

Agree Disagree

2.249 0.134 0.181

Agree Disagree

5.499 0.019 0.284

Post-survey Cribnotes Ease of Use Agree Disagree Total

37 20 57 5 7 12

42 27 69

Post-survey Cribnotes Perceived Usefulness

Agree Disagree Total

24 20 44 6 18 24

30 38 68

The MAK performance score was plotted against the dependent variable Cribnotes

performance score (Figure 6). Both MAK performance score and Cribnotes performance

score were defined and formulated from the proportion of times a nurse verified/

administered breast milk within the proper time frame of 1 hour to a patient divided by

the total number oftimes the nurse verified/ administered breast milk in each respective

system. The scatterplot suggests a positive correlation of .265 between the variables.

Possible outliers investigated included study pa.J.iicipants 45, 47, 48, 53, 58, 59, 61. Each

57

outlier was investigated by looking up the actual performance score tied to the study

participant. In all cases, the outliers resulted from low performance scores and thus there

was no evidence that warranted the removal of these study participants scores from the

study.

Figure 6: Objective MAK Performance Score vs. Objective Cribnotes Performance

Score

e 8

en fl

1.00

0.80

c 0.60

~ ~ Cll c.. Ul 0.40

-E 0 c .c ·;: (.)

0.20

0.00

0.00

0

0

0.20

0

0

0

0

0

0 0 0

0

0

0.40 0.60 0.80 1.00

MAK Performance Score

In addition, a bivariate analysis was conducted on the MAK performance score and

Cribnotes performance score variables. This yielded a p value of .028, and thus a

statistically significant Pearson Correlation of .265, p< .05.

58

Regression Analysis

A linear regression analysis was conducted of the full model with the inclusion of an

additional variable signifying the number of times each nurse used the Cribnotes system,

or experience, for breast milk verification purposes (Table 1 0).

Table 10: Regression and ANOV A statistics- Full Model plus Cribnotes Experience

Covariates (CV)

MAK Obj. Pre-MAK Ease of Use

Pre-MAK Perceived Usefulness Pre-Cribnotes Ease of Use

P re-C rib notes Perceived Usefulness CN Experience

R= .481 R sq=.231 Adj R sq=.146 p <.05= * .023

B Std. Error B

0.298 0.139 -0.007

0.061 -0.073

0.046 0.153

0.063 -0.062

0.042 -5.61 E-

5 0.000

*Statistically significant (p < .05) **Statistically significant (.05> p < .1)

Beta p

0.265 *0.037 -0.017 0.903

-0.215 0.117

0.314 *0.019

-0.188 0.148

-0.028 0.833

From the regression results it can be seen that the model is statistically significant (p<.05)

and explains on a very conservative level 14.7% of the variation in the Cribnotes

performance score. Related to the individual coefficients, the objective measure of

performance scores on the old system (MAK) is significantly positively related to the

performance scores on the new system (Cribnotes) (p < 0.05). Similarly, the pre-

59

implementation survey-based subjective measure result for Pre-Cribnotes Ease of Use is

also significantly positively related to the performance scores on the new system (p<.05).

Qualitative Analysis

To derive the results pertaining to research question 3, a qualitative content analysis was

conducted by using study participants subjective, post-implementation, survey-based

narrative comments. Of the 69 study participants, 35 responded with comments. In order

to identify the possibility that additional variables could enhance the model in future

iterations or applications, the comments were analyzed through an inductive reasoning

approach. All of the comments received were reviewed individually and coded into

subcategories from the ground up. The 35 comments were broken down even fu1iher into

73 sub-comments due to respondents making multiple statements. These sub-comments

were coded individually and if applicable, sometimes included multiple codes.

There were a total of 31 codes applied to the sub-comments. Frequency counts were

performed on each code at the base, or low level category. These 31 codes were then

grouped into 9 mid-level categories consisting of the following: 1) Clinical Application

Response (Positive); 2) Clinical Ease ofUse (Positive); 3) Clinical Quality Improvement

(Positive); 4) Clinical Impact (Negative); 5) Teaching Impact (Negative); 6) Workflow

Dependencies (Negative); 7) Application Specific (Negative); 8) IT Infrastructure

(Negative); and 9) Other. The 9 mid-level categories were then grouped into 4 high-level

categories that consisted of 1) Overall Clinical Impact (Positive); 2) Overall Clinical

60

Impact (Negative); Overall IT Impact (Positive); and 4) Overall IT Impact (Negative).

The results for the low, mid and high level categories are displayed in Figure 7.

Positive and negative connotations related to the initial comment response were used

within the categorizations. From a clinical perspective, examples of Overall Clinical

Impact (Positive) include positive comments such as, "MAK has increased quality of

patient care", and, "Cribnotes does make it easier to look back to keep continuity of

care". Conversely, examples of Overall Clinical Impact (Negative) include negative

cornn1ents such as, "A large amount of time is spent at the computer documenting and

less time is spent at the bedside", and, "Both MAK and Cribnotes has increased our

worldoad".

From a technology perspective, examples of Overall IT (Positive) include positive

comments such as, "MAK is good when the system works", and, "When systems are

operating they are fine". Conversely, examples of Overall IT (Negative) includes

negative comments such as, "Both systems are frequently down, making it difficult to

chart in a timely manner", and, "With logging in and out of the computer all day, less

time can be devoted to actual patient care". Additional negative technology comments

related to computer response time include, "Citrix needs to be revamped or switch to a

server that doesn't require freeze up at least two to tlu·ee times per shift", and, "I wish the

system was faster and could handle the large volume of traffic. The locking up and time it

takes to change between screens really slows up the process

61

Figure 7: Cootent Analysis- Post-lmplanen!ation Surw-y Comments

Category- Codes (Low-lev<>ij

Ease of use- darity Ease of use- learnhg cr ... ical duties unchar.;;ed (irtdl•.idual}

Chsrtfng c>Jmpliano: lmprcwsment Patient C3.re' imprmi'ement Crfumf.es-Future Qu..'1Jily Improvement Crrmetles- E~-clving Quafit•Jin-pra.•e.nF-nt

Charting w::rldoa':l Charting b'rn:linE:Ss Inadequate IT slaffugl .sup~cr! {V!nicians troub:~'!oo1ing}

Social Gull lnadeq<Jale iea;:bng tine (pare.'!ts)

AppE<:aiion /<C>e6s. {non- NICAJ sl:a.'l) Phamlacy Depl?ndeD.."Y ~JW::. c~.ter-rro'e

MAK appro'.'al. Critnoles. apprCiia!

Critnetles app'fcaf>:n func'Jo.'li :2.App1cat.'cn integra\i:n. Cnoooles App'fc;F.}:;n S~tlmgs Co.'lf:1.JUraiioo MI<K m'ssirg fuul:lfTTI3tion CriOOotes. rn'Ssirg i'nfoonalian Critnoles, th:mt>Jhnes.s

MI<K ihs!abiltt~· MPK abwniime cflnical impli::atiori5 CM;noles dm•mthr.e cflnie<~l irr.pacl Campuler response tin-.P­Jnaeeq~~ate Sy5lems Acce:ss. Timainess. Ciiril: inl'ras!ructure settings c.::n6gurat:oo

C.o1ecli.oo twf fe8:1back

#'Times Ca.iegory {Mid-Le•,oeJ}

~ lv.rnical Ease of use (Posiu\~e}

! I Clinical Quality lrn,;:roveme."ll. t (Positive) t

91 ·1~ Clinical lirnpact (Negative}

i ITeachir.Y,~Impaet{Negalive) 2

31 ! Clinical Workflow Oe;:er.d {NE>l!}

51Ap. plicat.'oo Specific 7 (Posl!ill'e)

?

~ Ap¢ca!l•oo Specrfic (Negai1~) 51 2 f

11 m 10 15 lT k!fras1ruclure {Negative} 5 9 :;

2IOiher

62

#Tones Category [High-Le>Jel) #limes

lw•'~~~~-) Hi

nl 3. O•tera1 Clincallmpac1 {Negative)'

5

:31.

1210vera'l11T {Posm>P-} 12:

14

Overall IT {Neganve) 67

53

2

To further analyze the data, Table 11, was constructed in order to show comparisons of

demographic groupings with both systems' quantitative performance score aspects, as

well as, the qualitative results from the post-survey comments.

Table 11: Study Participant Demographics by Performance Scores and Post-Survey Comments

Study Post-Survey Participants Performance Scores Comments

MAK- Cribnotes-Total 69 Mean Mean Count/ Percentage Gender M 2 (2.9) 0.635 0.52 0 (0) F 66 (95.7j_ 0.587 0.783 37 (56.1)

Ethnicity White 61 (88.4) 0.585 0.774 33 (54.1) Non-White 6 (8.5l 0.62 0.76 4 (66.6)

Professional Degree LPN 1 (1.4) 0.45 0.83 1 (1 00) RN 7 (10.1) 0.64 0.86 5 (71.4)_ RN-ADN 18 (26) 0.61 0.69 8 (44.4) RN-BSN 41 (59.4) 0.571 0.803 22 (53. 7)

Age 20-34 22 (31.8) 0.616 0.775 12 (54.5) 35-49 29 (42.0) 0.61 0.8 14 (48.3) 50-64 13(18.8) 0.484 0.709 7 (.538)

These findings reveal that females scored nearly 20% points higher on the Cribnotes

system than on the MAK system and were also the sole respondents to the post-survey

comments at 37. Similarly, when evaluating ethnicity, white study participants increased

their performance score in the Cribnotes system by 19%, and were also the primary

respondents to the post-survey comments at 33. Professional degrees related to RN-BSN

group, representative of 60% of the study p3.1.iicipants, showed the most improvement in

the Cribnotes system, and were also the primary respondents to the post-survey at 22.

Age Groups 20-34 and 35-49 increased performance scores in Cribnotes by 17-19%, and

were also the primary respondents to the post-survey at 12-14. While the Age Group 50-

64 increased performance scores were representative of fewer patiicipants, their

performance scores increased by 20-22%, and were the least responsive to the post­

survey comments at 7.

Lastly, a more detailed representation of the study population's results at an individual

level are repmied on in Table 12. This table contains individual based perfonnance scores

relative to both information systems, as well as, multi-tiered specifics of the linked

comments at the sub-levels associated by positive and/or negative connotations. The

purpose of this table is to clearly represent how the sub-comments and associated sub­

codes can impact the qualitative analysis outcomes of the study. Put simply, there can be

multiple sub-comments derived from lengthy survey comments that are represented by

multiple codes. These numeric counts in association with there positive or negative

connotations are compared with the performance scores in order to evaluate the

associations between the objective and subjective results by individual. From the results

the negative codes far outweigh the positive codes in terms of count. Further, regardless

of the positive and/or negative cmmotations represented, 63, or 91.3%, of the study

participants realized a positive increase from MAK performance scores to Cribnotes

performance scores. Only 6 study participants noticed a decreased performance score on

Cribnotes when compared with MAK. Upon investigation, these 6 accounted for a

combined 6 positive codes and 20 negative codes out of the total 120 representative

codes.

64

Table 12: Individual Study Participant by Performance Scores, Sub-comments, Sub-Codes, and Connotation

Sub- Sub-ID Performance Scores Comments Codes Connotation

MAK- Cribnotes-Unique Mean Mean Count Count Pos/Neg

1 0.63 0.85 3 3 3-Ne_g_ative 3 0.65 0.83 2 2 2-Negative 5 0.58 0.9 3 4 1-Pos/ 3-Neg 7 0.65 0.81 5 6 3-Pos/ 3-Neg 8 0.66 0.94 1 1 1-Negative 10 0.42 0.88 1 1 1-N~ative

13 0.57 0.86 6 9 9-Negative 14 0.66 0.88 2 6 2-Pos/ 4-Neg 17 0.61 0.92 4 5 5-Ne_gative 18 0.77 0.92 2 3 3-Negative 21 0.66 0.73 1 2 1-Pos/ 1-Neg 22 0.71 0.61 4 4 1-Pos/ 3-Neg 23 0.51 0.83 1 1 Other 24 0.5 0.85 1 3 3-Negative 26 0.74 0.8 1 4 4-Negative 28 0.72 0.87 1 5 5-Negative 30 0.83 0.64 2 5 1-Pos/ 4-Ne_g 34 0.57 0.68 1 1 1-Positive 35 0.74 0.77 5 7 5-Pos/ 2-Neg 38 0.84 0.74 4 6 6-Negative 40 0.7 0.82 2 3 2-Pos/ 1-N~ 41 0.75 0.58 2 4 4-Negative 43 0.48 0.93 2 4 4-Negative 44 0.52 0.84 1 4 4-Negative 49 0.58 0.69 1 2 2-Ne_gative 51 0.55 0.95 2 4 2-Pos/ 2-Neg 53 0.59 0.21 1 1 1-Negative 54 0.77 0.81 1 6 6-Ne_gative 55 0.51 0.89 1 2 2-Ne_g_ative 56 0.48 0.88 2 4 1-Pos/ 3-Neg 57 0.35 0.96 1 1 1-Positive 60 0.87 0.7 1 3 3-Postive 61 0.27 0.02 2 2 1-Pos/ 1-Neg

1-0ther/ 1-62 0.71 0.62 2 2 Neg 64 0.65 0.95 1 1 1-Positive 66 0.46 0.68 3 6 6-Negative 69 0.66 0.86 1 1 1-Negative

65

VIII. Discussion

This section will first utilize the results found in order to discuss the study variables and

how they relate to the outcomes of the four hypotheses and research questions. Second,

the implications of the study findings associated with policy, theory, overall methodology

to include limitations, and management will be addressed. Third, the study's contribution

to research literature and future research opportunities are reviewed.

The three research questions for which this study seeks to provide answers are as follows:

1) Will RN historic system performance be a better predictor of future system use than

utilizing their attitude, or behavioral intention to use it? 2) Will RN historic system

performance work in conjunction with RN behavioral intentions to explain more R

squared variability together, rather than independently? 3) Were clinical, social, and/or

technical factors identified that can be added as subjective variables to enhance future

analysis ofR squared findings related toRN performance? IS prediction models have

been applied to healthcare settings in an effort to measure adoption, diffusion, and

acceptance ofEHR teclmologies. The TAM was utilized due to its presence in ten percent

of all information systems publications, with thi1iy to forty percent of IT acceptance

attributed to reviews ofthe basic theory (Holden, 2010). The TAM theory thus has a

proven track record in terms of theoretical constructs with which to measure and use in

testing hypotheses. However, the TAM, at its core, primarily focuses on subjective

variables such as perceived ease of use and usefulness to explain technology utilization,

and while useful, this study seeks to test a more objective approach.

66

Results of Hypothesis

Quantitative- Regression Analysis

There were four hypothesis to be tested. Put simply, the intentions of the research were

to assess whether or not the findings related to the addition of the historic objective

performance on the old software would overpower the TAM subjective variables to the

point where TAM was not worthwhile to perform, or if it would be additive in making

the TAM more powerful. Fmiher, these variables were expected to explain all variability

in current performance on the new software. The stated hypotheses are as follows: 1) H1:

The MAK objective nursing performance score will be the primary predictor, or will have

the most impact, on the Cribnotes objective nursing performance score when compared

individually with all other subjective RN behavioral intentions; 2) H2: The MAK

objective nursing performance score will be the primary predictor, or will have the most

impact, on the Cribnotes objective nursing performance score when compared with all

other subjective RN behavioral intentions in aggregate; 3) H3: The MAK objective

nursing performance score will work in conjunction with RN behavioral intentions to

create a more explanatory framework; and 4) H4: The full model consisting ofMAK

objective nursing performance score and the pre-implementation subjective RN

behavioral intentions will describe all the variation in the dependent variable, Cribnotes

objective performance score. Quantitative analysis consisting of linear regression

analysis was utilized to evaluate Hypotheses 1, 2, 3, and 4. In addition, qualitative

analysis consisting of content analysis was utilized to fu1iher evaluate Hypothesis 4.

67

From the quantitative regression analysis, the independent variables of interest measured

against the dependent variable, Cribnotes performance score, consist of the following: 1)

MAK objective use performance score; 2) Pre-MAK perceived ease of use; 3) Pre-MAK

perceived usefulness; 4) Pre-Cribnotes perceived ease of use; 5) Pre-Cribnotes perceived

usefulness; and 6) Post-Cribnotes experience. The coefficient on Pre-MAK objective

performance score was statistically significant at p= .037 (p<.05). This coefficient of .298

indicates that as the number ofPre-MAK objective performance scores increase by one,

the number ofCribnotes objective performance scores increases by 29.8%. The

coefficient on Pre-MAK perceived ease of use was statistically insignificant at p= .903.

This indicates that the nurses who perceive the software as easy to use perform no

differently when measured objectively. The coefficient on Pre-MAK perceived

usefulness was statistically insignificant at p= .11 7. This indicates that the nurses who

perceive the software as useful performed no differently when measured objectively. The

coefficient on Pre-Cribnotes perceived ease of use was statistically significant at p= .019

(p<.05). This coefficient of .153 indicates that the nurses who perceive the software as

easy to use, perform better on Cribnotes objective performance score by an increase of

15.3%. The coefficient on Pre-Cribnotes perceived usefulness was statistically

insignificant at p= .148. This indicates that the nurses who perceive the software as useful

perform no differently when measured objectively. Lastly, the coefficient on Cribnotes

Experience was statistically insignificant at p= .833. This indicates that nurses with

different experience levels with Cribnotes perform no differently when measured

objectively.

68

As it relates to the hypotheses, the statistically significant coefficient of .298 for MAK

objective performance score suggests that if nurses performed well in the MAK system,

then they will perform even better in the Cribnotes system. Thus, high performers in

MAK will surely be high performers in Cribnotes. Those users who were mid performers

in MAK will most likely perform higher in Cribnotes. This predictor was the strongest

individually and in aggregate as Hypothesis 1 and 2 suggest. Thus, these hypotheses are

accepted (Table 13). The coefficient for Pre-Cribnotes perceived ease of use is

statistically significant at .153, and is thus in alignment with the traditional TAM theory

that suggests if a user perceives a system to be easy to use, than that will be a strong

predictor of actual system use. Thus, this variable can be considered as having a positive

impact on the Cribnotes Performance Score, albeit 15.3% in comparison to the 29.8% of

the MAK objective perfonnance score. Note that this variable was reverse coded for

clarity due to the negative connotation when considering nursing responses to the pre­

survey question, "Expectation that Cribnotes will increase your nursing workload." In

this case, when the nurses agreed, they had actually disagreed with the respective TAM

variable ease of use. In addition, Hypothesis 3 was accepted because the TAM model

represented by Pre-Cribnotes Ease ofUse, is made much stronger, and thus explains

more variance, tlu·ough the additive presence ofMAK objective performance scores

(Table 13). When these two variables are investigated more closely, the nurses who

perceived the software as easy to use, with the exception of one, also displayed MAK

performance scores ranging from 50%-66%, which is above both the mean and high end

of the standard deviation. This suggests that MAK performance scores may be

69

influencing future perceptions related to ease of use for the Cribnotes application, thus

further strengthening the MAK performance score construct. Lastly, the statistically

significant (p=.023) full regression model yielded the following: R= .481; R squared=

.231; and adjusted R squared= .146. The R value was used to indicate the strength of the

association between the six independent variables at 48.1 %. The R squared value

represents the propmiion of variance in the Cribnotes performance score, explained by

the independent variables at 23.1 %. The Adjusted R square value corrects for positive

bias and estimates effect size equals .146, or 14.6%. When adjusted, the independent

variables account for 14.6% of variation in the dependent variable, Cribnotes

performance score. Thus, Hypothesis 4 is rejected because there was still 85.4%

unexplained variability in the Cribnotes performance score (Table 13). In order to try to

address the remaining 85.4% variability from the quantitative analysis, a qualitative

content analysis was performed.

70

Table 13- Hypothesis Summary Results

DV- Cribnotes Performance Hypothesis Construct- IV Score Supported?

MAK Objective Performance Score

H1 Increase (29.8%) Yes MAK Objective Performance Score

H2 Increase (29.8%) Yes Pre-MAK Ease of Use Decrease Pre-MAK Perceived Usefulness Decrease Pre-Cribnotes Ease of Use

Increase (15.3%) Pre-Cribnotes Perceived Usefulness

Decrease CN Experience Decrease MAK Objective Performance Score

H3 Increase (29.8%) Yes Pre-MAK Ease of Use Increase (15.3%) Full Model No (85.4%

H4 Insufficient (Adj R sq= 14.6%) unexplained)

Qualitative- Content Analysis

From the qualitative content analysis results, derived specifically by the post-

implementation survey comments, the mid-level and high-level categories revealed

applicable findings directly associated with Hypothesis 4. Overall, clinical and IT

comments that were positive totaled 22, or 18.3%, while the negative comments totaled

71

98, or 81. 7%. The 1 :3 ratio of positive to negative responses represents roughly half of

the study pruiicipants. Each high level category is discussed in conjunction with its

associated mid-level categories in order to gain a more detailed understanding of the

coding composition.

Positive Clinical and Technical Impacts

In the first high-level category, the frequency of codes related to Overall Clinical Impact

(Positive) resulted in only 10 instances, or 8.3%. This number represents the sum of mid­

level categories Clinical Ease of Use and Clinical Quality Improvement. There were only

three instances after having used both software applications where nursing felt like it was

easy to use. Further, only seven felt like the software applications were useful in the areas

of quality improvement and compliance. These results are in alignment with the rejection

of Hypothesis 4 as the Overall Clinical Impact (Positive), represented 8.3% of all coded

comments collected.

In the second high-level category, the frequency of codes related to Overall IT Impact

(Positive) resulted in 12 instances, or 10%. This number represents the sum of mid-level

category of Application Specific, with zero coded results related to IT Infrastructure. In

this case, there were only twelve approvals for the softwru·e, five of which were for MAK

and seven of which were for Cribnotes.

72

Negative Clinical and Technical Impacts

In the third high-level category, the frequency of codes related to Overall Clinical Impact

(Negative) resulted in 31 instances, or 25.8%. This number represents the sum of mid­

level categories Clinical Impact, Teaching Impact, and Clinical Workflow Dependencies.

There were twenty three instances where nurses felt like chmiing workload, timeliness,

and time trouble-shooting IT problems negatively impacted their clinical duties. Along

the same lines, there were three instances where nurses felt that their ability to teach

parents of neo-natal patients was negatively impacted. Lastly, there were five instances

where nurses expressed concern over clinical workflow dependencies among staff outside

the NICU, pharmacy, and clinical situations that necessitated system over-rides. As

Hypothesis 4 was rejected, a portion of the remaining 85% variability may very well be

explained by measuring these three mid-level categories. Further investigation as to the

negative impact they may be having on existing full model regression variables or the

possible addition of them to the regression model may be warranted.

In the fourth high-level category, the frequency of codes related to Overall IT Impact

(Negative) resulted in 67 instances, or 55.8%. This level represents the sum of mid-level

categories Application Specific and IT infrastructure. There were fourteen instances

where nurses did not like the MAK and Cribnotes functionality, lack of integration,

thorouglmess, and system missing information. There were fifty-three instances of IT

Infrastructure where the nurses complained of computer response times, access

timeliness, system instability, associated downtime implications, infrastructure software

configurations, and system inadequacy. This mid-level category alone comprised of 44%

73

of the total coded category frequencies. As Hypothesis 4 was rejected, a portion of the

remaining 85% variability will most likely be explained by measuring these two mid­

level categories. There is an obvious workflow barrier in the area ofiT Infrastructure that

needs to be addressed. Further investigation as to the negative impact these may be

having on existing full model regression variables and the possible addition of them to

the regression model may be warranted.

Major Implications

The three research questions addressed by this study are as follows: 1) Will RN historic

system performance be a better predictor of future system use than utilizing their attitude,

or behavioral intention to use it? 2) Will RN historic system performance work in

conjunction with RN behavioral intentions to explain more R squared variability together,

rather than independently? 3) Were clinical, social, and/or teclmical factors identified that

can be added as subjective variables to enhance future analysis ofR squared findings

related to RN performance? Each question has been addressed and can be answered with

a "yes," in that research questions 1 and 2 are positively supported by the results from the

quantitative regression analysis, and question 3 is supported by the qualitative content

analysis results. The implications of these answers contribute to health services research

areas of policy, theory, management, and methodologies in a number of ways, as

described next in the study.

74

Policy Implications

For more than a decade, the theme of improved patient safety and quality of care in the

United States healthcare system has been a common thread in healthcare policy decision:..

making and overall strategy. Specifically, the on-going push for the implementation and

meaningful use of EHRs continue to become more of a reality and are strongly supported

by the federal govermnent. EHRs as a fundamental component in the patient safety and

quality of care movement will not only provide an electronic medium for what in the past

was a paper based system, but have the potential to fundamentally change the workflows

and practices of clinical end users. For these reasons, the importance of understanding

how to become successful users of technology across the healthcare spectrum has never

been greater.

With these points in mind, the results of this study and proposed framework suggest that

policy makers whom support the need for improved quality of care through the

implementation of EHRs need to strongly consider the TAM in conjunction with prior

system performance by clinicians at an individual, departmental, and ultimately

organizational level, as a benchmark by which to predict the timing for requiring greater

quality based metrics and enforcing them through future system enhancements. In other

words, there will have to be a measureable balance by which policy makers decide to

increase quality of care standards that an organization can reasonably meet. For example,

if a large percentage ofhealthcare organizations across the United States are performing

well on properly utilizing the EHR and achieving quality standards, then it may be time

to implement higher quality standards. These standards would then be captured in future

75

software with reimbursement based quality metrics. This is not to say that the software

the clinician is using is the cause of the quality, but rather it is the medium they are using

to assist with the provision of quality care. So if how one performs on a prior system in

conjunction with how one feels about a future system are a representation of current

quality, and thus indicators of future performance, then it seems that policy makers,

associated software certification boards, third party vendors, healthcare organizations,

and ultimately the end user are all in this together.

Theory Implications

The study results suggest that the TAM, while widely accepted as an explanatory model,

is incomplete. The study's measurement results of the historic objective use performance

score proved to be a more powerful predictor than the TAM model as a whole. In

addition, the data suggests that a portion ofthe study results related to the TAM may be

derived from the end users prior experience on the historic system. The TAM, while still

valuable is greatly enhanced by three times the normal explanatory strength as a

prediction model, when it is used in conjunction with the objective performance score.

With this in mind, the objective performance score should be the new basis of the

prediction model. The TAM should not be completely done away with because it has

proven to be a useful predictor, but it should compliment, or be secondary, to the

objective performance score serving as the primary predictor. In addition, the comments

section that was added to the TAM survey was of great value in identifying other

concerns·that may be directly impacting the TAM, but more impmiantly, the objective

perfonnance score. The TAM, as measured through the data collection survey tool in this

76

study, can be made cyclical in a process improvement fashion in that it can be

administered at any time and has the ability to identify related staff concerns. As

concerns are addressed, and thus interventions take place, the resulting impact can be

checked periodically against the pre-post on-going performance scores. So it seems that

while TAM's value as a prediction variable secondary to objective performance scores is

. valid for new pre-post system implementations, it may better serve as an on-going

process-improvement quality management tool in the areas of clinical and IT workflow

throughout the application's life-cycle working within a software generation. Thus, this

revised model framework allows for continuous measurement in preparation for 1) small

incremental changes in the current generation of software (ie. versions) and also 2) for

complete overhauls (ie. replacing legacy systems, changing vendors, software platform

changes) in new generations of software (Appendix 4- Figure 8).

Management Implications

Implications of this study affect a number of management facets across healthcare

organizations. Administrators, as leaders of the organization, are focused on providing a

good product in the form ofhealthcare services and being paid for those services in order

to remain financially viable. The bottom line for leadership is that reimbursement by

payers represented by State and Federal govermnents, as well as, private third parties, is

going to be increasingly dictated by the quality of care provided to those insured. The

way these payers are going to measure organizational quality will be driven by reports

derived from the actual care provided and documented in EHRs. Put simply, an

organization will only be as good as its data. Therefore, the results of this study are

77

compelling in that a framework with which to measure performance at the individual,

departmental, and organizational levels, on key quality measures is a necessity and

should not be taken lightly. If administrators choose to purchase, implement and attest to

State and Federal governments for incentive payments related to meaningful use in hopes

of not being penalized after 2016 in some cases, then they need to understand how to be

successful with their care providing producers, or end users in this case, accepting the

technology and using it appropriately.

From the perspective of mid-level management, quality management, who is responsible

for reporting on these matters, may find themselves in a conundrum of finger pointing

amongst other mid-managers as they try to harness the necessary data in order to check

the box, so to speak, on mandated quality metrics. In this case, workflow dependencies,

describing how specialized hospital departments must work together to ultimately

perfonn a service, are extremely important. These workflow dependencies between the

direct and indirect providers represented by physicians, nursing staff, ancillary clinical

services, such as Pharmacy, Radiology, and Laboratory, and the IT Department all play

vital roles in the delivery of quality care services. In this study, the impacts of

dependencies on nurses related to Pharmacy, other non-NICU providers, and an unstable

IT infrastructure may be primary reasons nurses are performing less than optimal in some

cases. Alternatively, perhaps these clinical and IT related workflow dependencies are

umelated in terms of current constructs measured and need to be tested within the

framework to quantify ban'iers to greater performance scores. Regardless, the onus of

responsibility for staff performance scores will fall on clinical department managers who

78

manage the provision of the actual care taking place and related documentation in the

EHR. Their staffs individual performance scores, as a component of their departmental

score, will ultimately make up the organizational score. Knowing this, clinical

department managers need to stay abreast of workflow issues by continuing to formally

survey their staff members in order to try and isolate current problems before they are

escalated and directly impact their department's performance score.

Methodology Implications and Limitations

For purposes of this study, the goal was to measure the biggest department in the health

system, which in this case was the NICU, in order to yield a large enough sample size to

run a full prediction model. However, the primary implication to the overall methodology

and also a limitation ended up being the sample size. The sample size began at 146 and

ended at 69 due to a number of constraints. The confines of the study methodology were

such that study pmiicipants data was only utilized if all of the data collection was

obtained. Mem1ing, that an eligible or measureable study participant must have all four of

the data points as follows: 1) Utilized MAK in the past when verifying/administering

breast-milk to an NICU infant; 2) Participated in the pre- implementation survey; 3)

Utilized Cribnotes when verifying/administering breast-milk to ail NICU infm1t; and 4)

Participated in the post-impleni.entation survey. Due to the small sample size a true

prediction model could not be run.

When selecting the base objective performance score to serve as the primary predictor, in

this case represented by MAK as a first generation nursing documentation system, it is

79

key to define and determine what makes an end user good at a system prior to measuring

it. This study used proportional base scores that statistically made sense, but did not have

a thorough understanding of how these base scores were impacted by internal or external

sources prior to and during the use of the MAK system. More specifically, there is a

critical need to know what made the end user either good or bad on the MAK system that

translated into their associated base performance score.

Further, the regression analysis needed more study participants in the sample in order to

improve accuracy. Within the regression analysis the experience of the study pa1iicipants,

or the frequency of times they used the system, was measured related to the Cribnotes

information system but was not performed for the MAK system. The addition of an

experience based independent variable for MAK may have better explained why some

users score higher or perhaps lower within the MAK system based on their experience

level with the system.

Other implications were seen in the descriptive statistics from the study results. These

revealed a homogenous population of nursing staff not just limited to study participants,

but to the depmiment as a whole. The department consists of primm·ily white, female

registered nurses (RNs), holding the same positions as staff nurses, and thus performing

the smne tasks. The ages differed throughout, but were evenly distributed.

Lastly, the data collection tool in the form of a pre-post survey was created and utilized

with a small scale that allowed nurses to only agree or disagree. A like1i scale was

80

actually discussed with the NICU nurse manager and IT Clinical Application Coordinator

prior to the creation, but it was decided to keep it simple in terms of the number of

questions and the possible answers. As stated previously in the hypothesis results, the

question related to ease of use represented by the nurse's expectation that workload

would be increased was phrased in a negative fashion. While it was effective at capturing

data, the responses had to be reverse-coded in order to reflect the ease of use definition

that TAM is based on. The final limitation is that ofthe post-survey comments section. In

an overcompensated effort to allow the users to express themselves outside of the

confines of the survey questions and answer options, the comments section was

comprised of open lines with absolutely no guidance. The resulting comments from

roughly half of the study participants were indeed very open ended, but were qualified

through content analysis into a coded categorical hierarchy to make sense of them. This

actually ended up being very informative and was aforementioned as a recommended

management practice for on-going performance monitoring at the clinical managerial

level.

Contributions to Literature

From the study results, measuring EHR historic objective performance scores of nurses in

the NICU yielded a statistically significant theoretical construct as pmi of a newly

formulated prediction framework that may be utilized at the individual, depmtmental,

orgm1izational, and perhaps even at state and national levels. This framework may also be

applied to other clinical disciplines as well. The identification: of a strong singulm·

quantitative predictor is integral to better understanding teclmology acceptance and

81

ultimately achieving quality goals through the use of the EHR medium. When

considering the TAM framework that is already widely proven and accepted in the

literature, it actually becomes a secondary predictor shadowed by the additive value of

the objective performance. The newly proposed framework would primarily utilize

objective performance as the prediction tool, but also use the TAM as an on-going

process-improvement quality management tool in the areas of clinical and IT workflow

throughout the application's life-cycle working within a software generation. By utilizing

this proposed framework, organizations can continuously measure objective performance

both within and across software generations, while making specific adjustments to

necessary workflow concerns along the way. (Appendix 4- Figure 8).

IX. Conclusions and Recommendations

From the study, Hypotheses 1, 2, and 3 were accepted because the addition of the

objective performance score measurement was the most powerful of all the variables. It

also worked in combination with subjective variables to enhance the TAM. Hypothesis 4

was rejected due to the remaining variability left unexplained by the full model. All

research questions echoed Hypotheses 1-3 and were thus affirmed.

This study measured performance on first and second generation nursing documentation

systems. Knowing that this newly proposed model framework is based on very early

stages of EHR use, there is a definite need and opportunity to measure multiple

generations of clinician performance on multi-generation EHRs into the future.

82

Additional research is necessary to further test the reliability of this proposed framework

both within present and across future EHR generations, as well as, seek out additional

performance specific explanatory construct variables. Clearly, a good stmiing place for

future measurements would be to begin with the concerns identified in the qualitative

results of this study, that include various aspects of workflow dependencies amongst

clinical services, as well as, IT facets such as infrastructure and specific application

functionality. Another opportunity is to investigate experience levels within past and

current generations ofEHRs moving forward is to evaluate whether or not a better

understanding related to the frequency of EHR use has an impact on user performance.

Lastly, when assessing base performance scores with which to measure and predict future

system use, there needs to be a more thorough understanding of what this consists of and

how it is derived. In other words, what made the end users good or bad users of the

technology the first time around and what role if any did management play that may have

had m1 impact.

83

Appendix 1

Figure 2- Detailed Conceptual Model

Actual Utilization Data Technology 1 (MAK)

NICU Nursing Timeliness of "Breastmilk verification"

+I d . ocumentatwn

H RN Performance Score Technology 1 (MAK)

-4

Actual Utilization Data Technology 2 (Cribnotes)

J NICUNursing Timeliness of "Breast milk verification" documentation-# of times performed

----.-RN Performance Score Technology 2 (Cribnotes) Time = 6 months

Appendix 2

Survey instruments

Pre-Cribnotes Implementation Survey

NICU Nursing Survey Questions- (please circle Agree or Disagree for each question)

Name: ------------------------------- Age: ______ _

Professional degree: ---------------------- Gender:

Position Title: Race: -------------------------- -------

Questions:

1. Has the implementation ofMAK increased your nursing workload?

Agree Disagree

2. Has the utilization ofMAK improved qu'ality of care for patients?

Agree Disagree

3. Expectation that CRIBNOTES will increase your nursing workload?

Agree Disagree

4. Expectation that CRIBNOTES will improve quality of care for patients?

Agree Disagree

Appendix 2 (continued)

Post-Cribnotes Implementation Survey

NICU Nursing Survey Questions- (please circle Agree or Disagree for each question)

Name: ------------------------------- Age: ___ _

Professional degree: ____________________ _ Gender:

Position Title: Race: -------------------------- -----

Questions:

1. Has the implementation ofMAK increased your nursing workload?

Agree Disagree

2. Has the utilization of MAK improved quality of care for patients?

Agree Disagree

3. Has the implementation ofCRIBNOTES increased your nursing workload?

Agree Disagree

4.. Has the utilization of CRIBNOTES improved quality of care for patients?

Agree Disagree

Comments:

86

Appendix 3

Table 1

Actual Technology Use Data- NICU Nursing Staff

Dimension, Variable Type ofVariable

Construct

Technology 1 Actual Use- Independent

(MAK) RN breast milk Variable

Performance Score administration

timeliness (T -18 Numerical

months)

Technology 2 Actual Use- Independent

(Cribnotes) RN breast milk Variable

Cribnotes administration

Experience- timeliness (T +6 Numerical

RN frequency months)

(# oftimes)

Teclmology 2 Actual Use- Dependent Variable

(Cribnotes) RN Breast Milk

Performance Score Verification Numerical

Documentation-

Critical Patients

(T+6 month)

Level of

Measurement

Continuous

Proportion of on-

time verifications

divided by total

times

Continuous

Number of times by

individual

Continuous

Proportion of on-

time verifications

divided by total

times

Table 2

Nursing Technology (Pre- Implementation) Survey- NICU Nursing Staff

Dimension, Variable Type of Variable Level of

Constmct Measurement

Nursing Technology Age Independent/ Continuous

Attitudes (Survey) Confounder

Numerical

Nursing Technology Gender Independent/ Nominal

Attitudes (Survey) Confounder

Categorical

Nursing Technology Race Independent/ Nominal

Attitudes (Survey) Confounder

Categorical

Nursing Technology Professional degree Independent/ Ordinal

Attitudes (Survey) Confounder

Categorical

Nursing Technology Position Title Independent/ Ordinal (Control)

Attitudes (Survey) Confounder

Categorical

Ease of Use (MAK) The implementation Independent Nominal

of MAK increased Variable Agree

your nursing Disagree

88

workload Categorical

Usefulness (MAK) The utilization of Independent Nominal-

MAK improved Variable Agree

quality of care for Disagree

patients Categorical

Ease ofUse Expectation that the Independent Nominal-

(Cribnotes) implementation of Variable Agree

CRIBNOTES will Disagree

mcrease your Categorical

nursing workload

Usefulness Expectation that the Independent Nominal-

(Cribnotes) utilization of Variable Agree

CRIBNOTES will Disagree

improve quality of Categorical

care for patients

89

Table3

Nursing Technology (Post- Implementation) Survey- NICU Nursing Staff

Dimension, Variable Type of Variable Level of

Construct Measurement

Nursing Technology Age Independent/ Continuous

Attitudes (Survey) Confounder (Control)

Numerical

Nursing Technology Gender Independent/ Nominal (Control)

Attitudes (Survey) Confounder

Categorical

Nursing Technology Race Independent/ Nominal (Control)

Attitudes (Survey) Confounder

Categorical

Nursing Technology Professional degree Independent/ Ordinal (Control)

Attitudes (Survey) Confounder

Categorical

Nursing Technology Position Title Independent/ Ordinal (Control)

Attitudes (Survey) Confounder

Categorical

Ease ofUse (MAK) The implementation Independent Nominal

of MAK increased Variable Agree

your nursmg Disagree

workload Categorical

90

Usefulness (MAK) The utilization of Independent Nominal-

MAK improved Variable Agree .

quality of care for Disagree

patients Categorical

Ease ofUse The implementation Independent Nominal-

(Cribnotes) of CRIBNOTES Variable Agree

increased your Disagree

nursing workload Categorical

Usefulness The utilization of Independent Nominal-

(Cribnotes) CRIBNOTES Variable Agree

improved quality of Disagree

care for patients Categorical

91

---- ~ ~~--~----"~---~~,-~~-=~------ ·~-~= --~·~~~~

Appendix 4

Figure 8- Objective Use Model

Quality based soflware version Enhancements,

Technical based software/ infi·astructure/ platform Changes

Workflow Efficiencies

Software Generation 1

v v

Quality based solhvare version Enhancements,

Technical based software/ infrastnacture/ platform Changes

Workflow Efficiencies

Quality based software version Enhancements,

Technical based software/ infrastructure/ platform Changes

Workflow Efficiencies

Quality based software version Enhancements,

Technical based software/ infrastructure/ platform Changes

Software Generation 2

v v

Technical based software/ infrastmcture/ platform Changes

Workflow Efficiencies

Quality based software version Enhancements,

Technical based software/ infrastructure/ platform Changes

Software Generation 3

Quality based software V v version Enhancements,

Technical based software/ infrastructure/ platform Changes

Workflow Efficiencies

Technical based software/ infrastructure/ platform Changes

Workflow Efficiencies

Quality based software version Enhancements,

Technical based software/ infrastructure/ platform Changes

Workflow Efficiencies

References

American College of Emergency Physicians. (2011). Health Information Teclmology.

American College of Emergency Physicians. Advocacy- Federal Issues. Retrieved

from http:/ /www.acep.org/Content.aspx?id=21664

Aspden, P. (2004). Patient Safety: Achieving a New Standard for Care. Washington, DC:

The National Academies Press.

Bass, F. (1969). A new product growth model for consumer deliverables. Management

Science, 15, 215-227.

Bramble, J., Galt, K., Siracuse, M., Abbott, A., Drincic, A., Paschal, K., & Fuji, K.

(20 1 0). The relationship between physician practice characteristics and physician

adoption of electronic health records. Health Care Management Review, 35(1),

55-64.

Carroll, A., Tarczy-Hornoch, P., O'Reilly, E., & Christakis, D. (2004). The Effect of

Point-of-Care Personal Digital Assistant Use on Resident Documentation

Discrepancies. Pediatrics, 113(3), 450-454.

Centers for Medicare and Medicaid Services. (2010). Certified EHR Teclmology.

Retrieved from

http://www. ems. gov /pf/printpage.asp ?ref= http://www. ems. gov /EHRincenti vePro

grams/25 _Certification. asp

Chassin, M., & Loeb, J. (2011). The Ongoing Quality Improvement Journey: Next Stop,

High Reliability. Health Affairs, 30(4), 559-568.

Chuo, J., & Hicks, R. (2008). Computer-related medication errors in neonatal intensive

care units. Clinical Perinatology, 35(1), 119-39, ix.

Classen, D., Resar, R., Griffin, F., Federico, F., Frankel, T., Kimmel, N., Whittington, J.,

et al. (2011). "Global Trigger Tool" Shows That Adverse Events In Hospitals

May Be Ten Times Greater Than Previously Measured. Health Affairs, 30( 4),

581-589.

CMS. (2010). EHR Incentive Program. Retrieved from

http://www.cms.gov/ehrincentiveprograms/

CMS Office of Public Affairs. (2010). CMS Finalizes Definition ofMeaningful Use of

Certified Electronic Health Records (EHR) Teclmology. Centers for Medicare and

Medicaid Services. Retrieved from

http://www.cms.gov/pf/printpage.asp?ref=http://www.cms.gov/apps/media/press/f

actsheet.asp?Counter=3 794&chkN ewsType=6&intPage=&intNumPerPage= 1 O&s

rchOpt=O&checkDate=&checkKey=&p Y ear=&srchType= 1 &numDays=3 500&sh

owAll=&srchData=&keywordType=All&year=&cboOrder=date&desc=

Cordero, L., Kuehn, L., Kumar, R., & Mekhjian, H. (2004). Impact of Computerized

Physician Order Entry on Clinical Practice in a Newborn Intensive Care Unit.

Journal ofPerinatology, 24, 88-93.

Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of

information teclmology. MIS Quarterly, 13(3), 319-340.

Davis, F., Bagozzi, R., & Warshaw, P. (1989). User Acceptance of Computer

Technology: A Comparison of Two Theoretical Models. Management Science,

(35), 982-1003.

94

Dentzer, S. (2011). Still Crossing the Quality Chasm- Or Suspended Over It? Health

Affairs, 30(no. 4), 554-555.

Depmiment of Health and Humam Services- Office of the National Coordinator for

Health Information Technology (ONCHIT). (2004, November). Executive

Summary. Retrieved from http://www.himss.org/handouts/executiveSummary.pdf

Depmiment of Health and Human Services- Office of the Secretary. (2003). Health

Insurance Reform: Security Standards- Final Rule (Federal Register No. 68 (34)).

Ford, E., Menachemi, N., & Phillips, M. (2006). Predicting the Adoption of Electronic

Health Records by Physicians: When Will Health Care be Paperless? Journal of

the American Medical Informatics Association, 13(1), 106-112.

Glaser, B., & Strauss, A. (1967). The Discovery ofGrounded Theory: Strategies for

Qualitative Research. New Jersey: Transaction Publishers.

Goodhue, D., & Thompson, R. (1995). Task-Technology Fit and Individual Performance.

MIS Quarterly, 19(2), 213-236.

Grand Rounds Software. (1994). Cribnotes Integrated Neonatal Software Solution.

Retrieved from www.cribnotes.com

Hjmi, B. (2005). The HIM Role in Patient Safety and Quality of Care (AHIMA Practice

Brief). Journal of AHIMA, 76(1), 56A-G.

Holden, R., & Karsh, B. (2010). The Technology Acceptance Model: Its Past and its

Future in Healthcare. Journal of Biomedical Informatics, 43(1).

Hyun, S., Jolmson, S., Stetson, P., & Bakken, S. (2009). Development and evaluation of

nursing user interface ·screens using multiple methods. Journal of Biomedical

Informatics, 42, 1004-1012.

95

Institute of Medicine- Committee on Quality of Health Care in America. (2001).

Crossing the Quality Chasm,: A New Health System for the 21st Century.

Washington, DC: National Academies Press.

Klarreich, E. (n.d.). The Mathematics of Strategy. Proceedings of the National Academy

of Sciences, Classics of the Scientific Literature. Retrieved .from

http://www. pnas.org/si te/misc/classics5 .shtml#Refl

Kohn, L., Corn, J., & Donaldson, M. (Eds.). (1999). To Err Is Human: Building a Safer

Health System,. Washington, DC: National Academies of Press.

Lehmann, C., Conner, K., & Cox, J. (2002). Provider Error Prevention: Online Total

Parenteral Nutrition Calculator. AMIA Annual Symposium Proceedings, 435-439.

Morton, M., & Wiedenbeck, S. (2009). A Framework for Predicting EHR Adoption

Attitudes: A Physician Survey. Perspectives in Health Information Management,

6.

Morton, M., & Wiedenbeck, S. (2010). EHR Acceptance Factors in Ambulatory Care: A

Survey of Physician Perceptions. Perspectives in Health Information

Management.

Nash, J. (1950). Equilibrium Points inN-Person Games. Proceedings of the National

Academy of Science, 36, 48-49.

Pfeffer, J., & Salancik, G. (1978). The external control of organizations: A resource

dependence perspective. New York: Harper and Row.

President Barrack Obama. (2009, January 6). American Recovery and Reinvestment Act

of2009. One Hundred Eleventh Congress ofthe United States of America.

96

Retrieved from http://frwebgate.access.gpo.gov/cgi­

bin/getdoc.cgi?dbname=lll_cong_bills&docid=f:hlenr.pdf

President George W. Bush. (2003, January 28). State of the Union Address. Presented at

the Joint session of Congress. Retrieved from

http://usgovinfo.about.com/library/weekly/aasou2003_text.htm

President George W. Bush. (2004). State ofthe Union Address. Retrieved from

http://dollar.biz.uiowa.edu/~street/research/OJNI09.pdf

President George W. Bush. (2005). State ofthe Union Address.

President George W. Bush. (2006, February 1). State ofthe Union Address. Retrieved

from http:/ /articles.cm1.com/2006-0 1-31/politics/sotu.transcript_l_ union-speech­

misguided-idealism-president-bush-s-state/9?_s=PM:POLITICS

President George W. Bush. (2007, January 24). State ofthe Union Address. Retrieved

from

http://www .himss. org/ ASP /ContentRedirector .asp?Contentid=664 51 &type= HIM

SSNewsltem

Pronovost, P., Marsteller, J., & Goeschel, A. (2011). Preventing Bloodstream Infections:

A Measurable National Success Story In Quality Improvement. Health Affairs,

30( 4), 628-634.

Pryor, D., Hendrich, A., Henkel, R., Beckmmm, J., & Tersigni, A. (2011). The Quality

"Journey" At Ascension Health: How We've Prevented At Least 1,500 Avoidable

Deaths A Year- And Aim To Do Even Better. Health Affairs, 30(4), 604-611.

97

Puangco, M., Nguyen, H., & Sheridan, M. (1997). Computerized PN ordering optimizes

timely nutrition therapy in a neonatal intensive care unit. Journal of the American

Dietetic Association, 97(3), 258-261.

Ramirez, A., Carlson, D., & Estes, C. (2010). Computerized Order Entry: Lessons

Learned from the Trenches. Neonatal Network, 29(4).

Rikli, J., Huizinga, B., Schafer, D., Atwater, A., Coker, K., & Sikora, C. (2009).

Implementation of an Electronic Documentation System Using Microsystem and

Quality Improvement Concepts. Advances in Neonatal Care, 9(2).

Rogers, E. (1962). Diffusion of innovations. New York: Free Press of Glencoe.

Sharamitaro, A. (20 11 ). CMS Develops Center for Medicare & Medicaid Innovation.

(Burt, J., Ed.)Health Capital Topics, 4(2).

Siemens. (2002). Med Administration Check. Retrieved from

http:/ /www.medical.siemens.com/webapp/wcs/stores/servlet/ProductDisplay~q_ c

atalogid~e _-1 ~a_ catTree~e _1 00010,1008631,1025982,1 025978~a _langid~e _-

1~a _productld~e _191 002~a _ storeid~e _1 000 l.htm

Taylor, J., Loan, L., Kamara, J., Blackburn, S., & Whitney, D. (2008). Medication

Administration Variances Before and After Implementation of Computerize

Physician Order Entry in a Neonatal Intensive Care Unit. Pediatrics, 121(1), 123-

128.

The Leapfrog Group. (20 11 ). Hospital Recognition Program. Retrieved from

www.leapfroggroup.org

98

Trivedi, A., Grebla, R., Wright, S., & Washington, D. (2011). Despite Improved Quality

of Care In The Veterans Affairs Health System, Racial Disparity Persists For

Impmiant Clinical Outcomes. Health Affairs, 30(no. 4), 707-715.

Woodside, J. (2007). Economic externalities ofhealth information technology. A game

theoretic model for electronic health record adoption. Journal of Healthcare

Inforamation Managem.ent, 21 ( 4 ), 25-31.

Yoon-Flannery, K., Zandieh, S., Kuperman, G., Langsam, D., Hyman, D., & Kaushal, R.

(2008). A qualitative analysis of an electronic health record (EHR)

implementation in an academic ambulatory setting. Informatics in Primary Care,

16, 277-284.

99