1-s2.0-S1386505611001985-main

8/12/2019 1-s2.0-S1386505611001985-main

1/13

i n te r na t io n al j o ur n al o f m e di c al i nf o rm a ti c s 8 0 ( 2 0 1 1 ) 828840

j o u rn a l ho m ep age : www. i jm i jou rna l . com

Key IT management issues in hospitals: Results of a Delphistudy in Canada

Mirou Jaana a , , Haitham Tamim b, Guy Par c, Mari Teitelbaum da Telfer School of Management, University of Ottawa, Ottawa, Canadab Sprott School of Business, Carleton University, Ottawa, Canadac Canada Research Chair in Information Technology in Health Care, HEC Montreal, Montreal, Canadad BORN Ontario, Childrens Hospital of Eastern Ontario, Ottawa, Canada

a r t i c l e i n f o

Article history:Received 16 January 2011Received in revised form30 April 2011Accepted 18 July 2011

Keywords:IT management issuesRanking-type DelphiChief information ofcersHospitals

a b s t r a c t

Purpose: To develop an authoritative list of IT management issues faced by CIOs and ITdirectors in public hospitals.Methods: The ranking-type Delphi method, which elicits the opinion of a panel of expertsthrough iterative controlled feedback, was used in this study. Data collection involved threemain steps during which three panels of experts in Ontario ( n = 13 in rural hospitals; n = 9in community hospitals; n = 8 in academic hospitals) provided their input about the key ITmanagement issues in hospitals and their relative importance. Attrition rate was minimal;28 out of the 30 experts who participated in the study completed all three phases of thesurvey.Results: During the brainstorming phase, the responses from the three panels were consol-idated resulting in a list of 36 IT management issues; eleven were overlapping between thethree panels. In the narrowing down phase, 18 issues were retained in the rural panel; 20issues in thecommunity panel;and 17 issuesin theacademic panel.The topissues retainedin this phase included: managing demand and expectations for IT services (76% of partic-ipants); having sufcient funds (69% of participants); recognizing IT as a key stakeholderin major hospital decisions (65% of participants); and implementation of an EMR (62% of participants). In the ranking phase, a moderate level of consensus was obtained for all threepanels: W = 0.41 (rural panel); W = 0.54 (community panel); and W = 0.43 (academic panel).Despite the differences in the preoccupations between the three groups, there were similar-ities on certain IT management issues. Besides having sufcient funds, three of the top ITmanagement issues relate to the strategic positioning of IT within the hospital: managing demands and expectations for IT services; IT competing with other clinical priorities forscarce resources; and recognizing IT as a key stakeholder in major hospital decisions.

Corresponding author at : Health Administration, Telfer School of Management, University of Ottawa, 55 Laurier Ave. East, Ottawa, ON,K1N 6N5 Canada. Tel.: +1 613 562 5800x4928; fax: +1 613 562 5164.

E-mail address: [email protected] (M. Jaana).1386-5056/$ see front matter 2011 Elsevier Ireland Ltd. All rights reserved.doi:10.1016/j.ijmedinf.2011.07.004
http://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004http://www.ijmijournal.com/mailto:[email protected]://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004http://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004mailto:[email protected]://www.ijmijournal.com/http://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004

8/12/2019 1-s2.0-S1386505611001985-main

2/13

i nt er na ti on al j ou rn al o f m ed ic al i nf or m at i cs 8 0 ( 2 0 1 1 ) 828840 829

Conclusions: This study is the rst to systematically and rigorously identify and prioritizecritical IT management issues in hospitals, which may be generalizable to similar environ-ments in other industrialized countries. The prioritized lists of IT management issues maybe used as a benchmark and diagnostic tool to support internal strategic decision making related to IT. The broadening of understanding of the challenges faced by IT executives inhospitals would support a more systematic evaluation of these issues over time, and allowmanagement, educational, and research resources to be invested in the appropriate areas.

2011 Elsevier Ireland Ltd. All rights reserved.

1. Introduction

Health care organizations today are continuously looking forand introducing new information technologies (IT) that arecapable of improving staff productivity and process efciencyand enabling better quality of services [1,2]. Emerging tech-nologies are intensively used to address some of the majorchallenges faced in health care systems around the world. Forexample, telehomecare is considered as a promising patientmanagement approach in light of the shortage in registerednurses and health care professionals in many industrializedcountries [36]. As another example, clinical decision sup-port systems have also become critical tools for addressing patient safety issues, which may reduce medication errorsand support clinical decision making [7,8]. In addition, therehas been an increasing focus on the importance of hav-ing an integrated infrastructure that supports the ow andexchange of patients and information within health care sys-tems and the building of an interoperable electronic healthrecords (EHR) system [9,10]. In light of all these changes andthe increasing role of technology in health care organizationsin general, and hospitals in specic, IT executives and deci-sion makers are facing various challenges and issues, whichunless recognized andaddressed, may hinder progress in thisarea.

In Canada, the health care system has undergone majortransformations in recent years [11,12], which have been par-alleled by a signicant evolution in the implementation of IT in this sector. A national initiative for investment in IThas been underway since 2001, which is driven by signicantefforts for the implementation of standardized EHR systems[9]. The province of Ontario, which has the highest numberof health care organizations and networks in the country [13],has been among the leading provinces in the national move-

ment toward the implementation of electronic medical record(EMR) systems and the adoption of the state-of-the-art tech-nologies that support health care delivery. Consequently, andin light of these changes, IT managers in this province arefacing newrealities and issues that requiresystematic assess-ment and evaluation.

IT management issues have been thoroughly examined inthe Information Systems (IS) literature. As early as the 1980s,the business literature has recognized the importance of investigating these issues from the perspective of chief infor-mation ofcers (CIOs) in order to gain a better understanding of the challenges and concerns in this domain and makesound judgments in areas requiring management, educa-

tional, and research resources [1425].

From a methodological

standpoint, theDelphi approach hasbeenconsistentlyused toexamine key management issues in this eld (e.g., [14,1618]given its merit in surfacing new issues in exploratory studies,and its ability to move a group of experts toward reaching aconsensus [26].

Some of the major IT management issues that have beenreported in the IS literature in the past include: attract-ing, developing, and retaining IT professionals; improving ITstrategic planning; facilitating and managing business pro-

cess reengineering; improving the effectiveness of softwaredevelopment; developing and implementing an informationarchitecture; and keeping updated with new trends in IT[16,20,2325]. These concerns have been thoroughly assessed,and efforts have been invested to address them over theyears. Based on a recent survey of IT leaders who aremembers of the Society for Information Management, theve most important issues reported nowadays in businessorganizations include: (1) business productivity and costreduction; (2) IT and business alignment; (3) business agilityand speed to market; (4) business process reengineering; and(5) IT cost reduction [25]. Interestingly, a shift in the focusof these issues was noticed compared to previous years,

with a higher emphasis on the role of IT in cutting costsand improving productivity in time of economic recession[25].

The medical informatics literature has lagged behind withrespect to the evidence-based assessment and analysis of contemporary IT management issues from the perspective of IT executives in health care organizations. A review of themedical informatics literature reveals scattered discussionsof challenges in thecontext of health IT projectsand thesuccessof their implementation. Some of the themes that commonlyemerge as hindering the realization of benets from health ITinclude: lack of sufcient resources [27]; absence of processreengineering [28]; privacy and security concerns [28,29]; lack

of IT knowledgeby clinicians [27]; andabsence of standards tosupport system interoperability [30]. Despite the limited evi-dence in this area, no prior study to date has specically andthoroughly examined IT management issues, nor assessedtheir priorities in this sector.

This study addresses this gap and presents the resultsof a Delphi study conducted with CIOs and IT directorsin rural, community, and academic hospitals in Canada.The results support the development of an authoritativelist of issues that can be used in future research to builda comprehensive framework of IT management issues inhealth care, examine their evolution over time, and iden-tify areas requiring management, educational, and research

resources.
http://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004http://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004

8/12/2019 1-s2.0-S1386505611001985-main

3/13

830 i nt er na ti on al j ou rn al o f m e di ca l i nf or ma ti cs 8 0 ( 2 0 1 1 ) 828840

2. Methods

The Delphi method has been used since the early 1960s toinvestigate a broad range of research problems in various aca-demic elds [26,31]. It is a valuable data-driven approach inexploratory studies where limited evidence exists on specic

topics and questions [26]. The ranking-type Delphi, whichis the most commonly used Delphi method in the IS eld,involves a set of linked questionnaires and aims to elicitthe opinion of a panel of experts through iterative controlledfeedback [16,32,33]. The iterative feedback is usually solicitedthrough three main steps, namely, brainstorming, narrowing down and ranking, with the purpose of reaching consensusabout the relative importance of items related to a given topicor question [16,32,33].

2.1. Composition of the panels

In order to reect the variation in respondents settings, werecruitedpanelists fromthree target groups of public hospitalsin Ontario, Canada: rural hospitals, community hospitals, andacademic hospitals. As previously mentioned, the provinceof Ontario represents one of the largest health care jurisdic-tions in Canada with the biggest population, highest numberof health care organizations and networks, and highest totalhealth expenditures and IT budget in the country [13].

The three types of hospitals are similar in terms of the jurisdictionto which theybelong and the regulations and poli-cies that they follow. Nevertheless, they differ with respect totheir characteristicsandenvironments (e.g., size, nancialandhuman resources), and their level of IT sophistication [34,35].Therefore, we expect to nd differences across the differenthospital types in relation to thecritical IT management issuesfaced by their IT executives.

In order to identify the potential participants in this study,we used a list of hospitals in Ontario generated by the OntarioHospital Association (OHA), excluding mental health andlong-termcarehospitals [36]. These were excluded given theirlimited budgets and therefore limited variance in terms of IT sophistication level. Rural hospitals are dened as singlecommunity providers with a total inpatient acute, complexcontinuing care, and day surgery weighted cases under 2,700[37]. Academic hospitals include acute and pediatric hospi-tals that are members in the Council of Academic Hospitalsof Ontario (CAHO), are afliated with a medical or healthsciences school, provide complex patient care, and supportsignicant research activity and post-graduate training [37].Last, community hospitals consist of regional hospitals thatrange in size between the rural and academic hospitals, arenot afliated with medical schools and usually provide lesscomplex services than academic hospitals [37].

In total, we solicited the participation of 102 CIOs/IT direc-tors in Ontario hospitals (38 rural hospitals; 51 communityhospitals; and 13 academic hospitals). Phone calls were rstmade to all potential respondents in the three groups to intro-duce the research project, explain the Delphi survey processand invite themto participate in the study. Whenrespondentswere not available, voicemails were left with a return phonenumber and e-mail address to allow them to reply at their

convenience. Individuals who responded negatively or couldnot be reached after three consecutive attempts wereexcluded. Overall, seven individuals refused to participate (6in rural hospitals and one in a community hospital), and thetotal number of CIOs/IT directors who agreed to be involvedin the Delphi study was 41, namely, 15 in rural hospitals; 17 incommunity hospitals; and 9 in academic hospitals. Out of the41 experts who agreed to participate in the study, 30 actuallyresponded to the rst round of the Delphi survey, representing 73% of the original sample.

2.2. Prole of the experts

As indicated in Table 1 , the three panels of experts differedsignicantlywith respect to theprole of their respective hos-pitals. The averagenumber of beds in therural hospitalspanelwas 48, as compared to 337 beds in the community panel, and587 beds in the academic panel. Rural and community hos-pitals also had signicantly lower hospital and IT budgets, aswell as smaller numbers of IT staff, as compared to academichospitals. Therefore, it is evident that the sample of partic-ipants included in this study represented different types of hospitals with varying organizational resources and capaci-ties.

The characteristics of the participants also varied acrossthe panels although the three groups were only signicantlydifferent in relation to participants title, education andtenurein current position ( Table 1 ). The majority of the partici-pants were men (82%), between 40 and 60 years of age (93%),had technical and IT background (57.5%), and spent on aver-age 11 years in their respective hospitals and 14 years inthe IT domain. It is important to note that only two of theparticipants had a medical informatics background and theremaining varied in specialization(e.g., management, nance,computer science, and information management). As shownin Table 1 , the majority of the panelists in the rural and com-munity groups had the title of IT director as opposed to CIO inthe academic group. Interestingly, 67% of the panelists fromrural hospitals reported high school/college as their highestlevel of education ascompared to14%in thecommunitygroupand none in the academic group; 71% and 50% of the pan-elists from community and academic hospitals, respectively,reportedhaving masters anddoctoral level of education. Last,the average tenure in current positionwas signicantlyhigherin the rural group (15 years) as compared to the communityand academic groups (9 and 5 years, respectively).

2.3. Data collection and analysis

Fig. 1 presents the phases of the Delphi survey conductedwith the three panels of experts. Data were collected betweenMarch and September 2010. In the rst phase of the Delphi,brainstorming , allpanelistswho hadagreedto participatein thestudy received via email the rst questionnaire asking themto generate a list of 510 key IT management issues over thenext three years. The panelists were also asked to provide abrief description of each identied issue to clarify ambiguousresponses [26,32,38]and support the reconciliationof answersin preparation for the second phase. Based on Yang [19], akey issue was dened as a concern, threat or problem that is

8/12/2019 1-s2.0-S1386505611001985-main

4/13

Table 1 Characteristics of the three panels.

Hospitals characteristics Rural hospitals ( N = 12) Community hospitals ( N = 8) Academic hospitals ( N = 8) OMean [range] Mean [range] Mean [range]

No. of beds a 48 [1088] 337 [157515] 587 [1471151]Hospital budget (millions) 20.35 [10.536] 209.69 [80330] 680.86 [1801200]IT budget (millions) 0.59 [0.11.3] 4.86 [1.39.5] 16.7 [430]No. IT staff 2 [13] 22 [935] 116 [23288]

Participants characteristics Rural hospitals ( N = 12) Community hospitals ( N = 8) Academic hospitals ( N = 8)N (%) N (%) N (%)

Participants title a

Director 8 (67) 7 (87) 3 (38)Other 4 (33) 1 (12) CIO 5 (63)

GenderMale 12 (100) 5 (62.5) 6 (75)Female 3 (37.5) 2 (25)

EducationHigh school/college 8 (67) 1 (14) Undergraduate 3 (25) 4 (50)

Masters 4 (57) 3 (37.5)Doctoral 1 (14) 1 (12.5)Other 1 (8) 1 (14)

Age3039 1 (8) 4049 6 (50) 2 (25) 6 (75)5059 4 (33) 6 (75) 2 (25) 60 1 (8)

SpecializationMedical informatics 1 (12.5) 1 (12.5)Networking and development 4 (40) 1 (12.5) 5Information technology/systems 4 (40) 3 (37.5) 3 (37.5) 10 (Management/administration 1 (10) 1 (12.5) 2 (25) 4Others 1 (10) 2 (25) 2 (25)

Participants characteristics Rural hospitals ( N = 12) Community hospitals ( N = 8) Academic hospitals ( N = 8)Mean [range] Mean [range] Mean [range]

Years in hospital 12 [329] 11 [127] 8 [215]Years in current position 15 [532] 9 [123] 5 [39] 1Years in IT management 14 [328] 16 [823] 14 [5.522] 1a Based on the Ontario Hospital Association 2010 ( www.oha.ca ); all other data were obtained from the panelists. P value associated with Chi-square test for categorical variables and ANOVA test for continuous variables.
http://www.oha.ca/http://www.oha.ca/

8/12/2019 1-s2.0-S1386505611001985-main

5/13


Phase 1 Brainstorming

5-10 key IT management issues solicited from CIOs / IT directorsThree panels of respondents from rural, community, and academic hospitalsA total of 207 IT management issues identified from all three panelsFinal combined list of 36 unique IT management issues after removing duplicates andmerging similar issues

Phase 2 Narrowing Down

The three panels separately pared the list downPanelists selected at least 10 IT management issues most deserving of their attentionand resourcesIssues identified by at least 1/3 of each panel were retainedTotal of 18 IT management issues retained in the rural panel; 20 in the community

panel; and 17 in the academic panel

Phase 3 Ranking

The three panels separately produced a ranked list of IT management issues retained

on his / her panelDegree of consensus calculated using Kendalls coefficient of concordance (W)Results shared with panelists who were asked to rank the key IT management issuesagain until reaching consensusModerate level of consensus reached after two rounds (W = 0.41 for the rural panel;W = 0.54 for the community panel; W = 0.43 for the academic panel)

Fig. 1 Delphi survey phases.

associated with the effective use of IT in hospitals. As men-tioned earlier, responses in phase 1 were received from 30participants in the three panels of hospitals (13 in rural hos-pitals, 9 in community hospitals, and 8 in academic centers).Following the brainstorming phase, twoof theauthors workedon collating responses, removing duplicates, and grouping similar answers.

In the secondphase, narrowing down , therandomly orderedconsolidated list of issues was circulated to participants inthe three panels for corrections, additions andvalidation. Fol-lowing Delbecq et al. [26] and Okoli and Pawlowski [38], theparticipants in each of the threepanels were asked to select atleast 10 IT management issues from the list that they consid-ered most deserving of their attention and resources over thenext three years. Participants in the three panels pared the listdown separately, andthe IT management issuesselectedby atleast one third of the respondents in each panel were retainedfor the next phase. A total of 29 panelists participated in thesecondphase of theDelphi surveys; responsewas not receivedfrom one of the participants in the panel of rural hospitals. Itis usually recommended to move to the last phase of the Del-phi when the pared list in thenarrowing down phase includesabout 20 items [32]. In this study, the number of IT manage-ment issues thatwere retainedin each of thethree panelswas:18 in the rural hospitals group; 20 in the community hospitalsgroup; and 17 in the academic hospitals group.

In the third phase, ranking , the experts in each of the threepanels were asked to separately rank, in order of importance,the IT management issues that were retained from the pre-ceding phase. Two rounds of ranking were necessary in orderto reach a moderate level of consensus among participants

in each of the three panels. We calculated the Kendall coef-cient of concordance ( W ) to determine the level of consensusbetween thepanelists in each group. In therst round of rank-ing, the IT management issues that were retained from phase2 were presented in random order to the panelists in the threegroups. In the second round of ranking, the IT managementissueswerepresented to thepanelists in order of theiraveragereceived ranks in the previous round within each group. Dur-ing this phase, three outlier responses from the rural panelwere excluded from the nal analysis given the effect theywere producing on the analysis of results.

In light of the sensitivity of the Delphi survey approachto attrition rates [31], it is important to monitor and reportthe number of panelists included in each round of the survey[32]. Table 2 presents the number of participants who wereincluded throughout the three phases of the Delphi surveys.As mentioned earlier, a total of 30 participants responded tothe initial phase. The attrition rate was minimal in phase 2 of the Delphi with only one participant from the rural hospitalsgroup not completing the survey. The attrition rate was alsominimal in the last phase with only one participant from thecommunity hospitals group not completing the second roundof the ranking exercise.

3. Results

3.1. Phase 1: brainstorming

Given theexploratory nature of this study and the limited evi-dence in this area, the goal of this phase was to develop a

8/12/2019 1-s2.0-S1386505611001985-main

6/13


Table 2 Number of participants in the three panels.

Agreed to participatein the Delphi study

Participated inphase 1

Participated inphase 2

Participated inphase 3 a

3a 3bRural/small hospitals panel 15 13 12 12 12Community hospitals panel 17 9 9 9 8Academic hospitals panel 9 8 8 8 8

Total 41 30 29 29 28a Two roundsof ranking of IT management issues (3a and 3b) were conducted in order to reach a good level of consensus between participants

in the three panels.

comprehensive list of IT management issues that covers thespectrumof issues faced byCIOs/ITdirectors in different typesof hospitals. The number of issues provided by each respon-dent ranged between three and ten issues. In total, 207 issueswere generated by the three panels. Eighty eight issues wereidentiedin the rural hospitalspanel; 61 issuesin thecommu-nity hospitals panel; and 58 issues in the academic hospitals

panel.In order to consolidate the list of IT management issues

generated in the rst phase of the Delphi, two of theauthors examined the responses, removed duplicate answers,and combined similar issues. For example, lack of fund-ing, funding requirements to keep pace with technology,adequate funding, funding for IT, funding for peopleand equipment, funding, and sufcient funding for theimplementation of IT strategic plan were combined underHaving sufcient funds. Similarly, attracting, developing,and retaining IT professionals, attracting qualied IT pro-fessionals, stafng shortages for IT, nding staff who arewell trained, attracting and retaining IT professionals, dif-

cult to nd HL7 expertise, aging workforce and many staff near retirement, lack of trained IT staff, ability to attractand retain highly qualied IT staff, and ability to hire high-endIT professionals such as project managers andarchitectswere combinedunder Recruiting anddeveloping IT staff withtheappropriate skillset. Differences in theconsolidationpro-cess were reconciled and the nal list included a total of 36 ITmanagement issues; a brief description of the complete list of issues is presented in Appendix A .

Eleven IT management issues were overlapping betweenthe three panels in phase 1 (see Table 3 ), which highlightstheir importance and relevance irrespective of the type of hospitals considered. Thus, despite the differences between

the three panels, there were common concerns and issuesthat are similar to what have been observed in the IS litera-ture. Althoughsome other issues were sometimesoverlapping between two of the panels at a time, the distribution andfrequency of the identied IT management issues differedacross the three groups. Certain issues were more commonly,or sometimes exclusively, reported by one of the panels.For example, managing external technological threats (e.g.,viruses) and malware (i.e. malicious software), unrealisticexpectations for IT to solve all problems, and prioritizing short-term xes over strategic execution were only reportedby the panelists in the rural group. On the other hand, man-aging demands and expectations for IT services, leveraging

business intelligence tools to report on performance and

outcomes, and deploying foundational technology serviceswere only reported by the panels in the community and aca-demic groups.

3.2. Phase 2: narrowing down

In the second phase of the Delphi survey, the consolidatedlist of randomly ordered issues identied in phase 1 was sentback to the respondents who were asked to select at least 10IT management issues most deserving of their attention andresources over the next three years. As mentioned earlier, theIT management issues thatwere identied byat leastone thirdof the participants in each of the three groups were retained.As a result, 18 issueswereretained in the rural panel;20 issuesin the community panel; and 17 issues in the academic panel.Although slight differences were observed between the threelists, there was overlap on nine issues between the three pan-els (Table 3 ). The top issues retained by the largest percent of respondents in this phase included: managing demand andexpectations for IT services (76% of all participants); having sufcient funds (69% of all participants); recognizing IT as akey stakeholder in major hospital decisions (65% of all partici-pants); and implementation of an EMR (i.e. shift from existing paper based health records and transitioning to a local sys-tem) (62% of all participants). Interestingly, two of these topissues were not originally identied by all three panels inphase 1. Specically, managing demand and expectations forIT serviceswas only reportedby thecommunityand academicpanels in phase1, andrecognizingIT as a key stakeholderwasoriginally identied by the rural panel only.

Out of the nine IT management issues that were retainedby all three panels in phase 2, six were also common to thethree groups in the preceding phase ( Table 3 ), which furtheremphasizes the relevance and importance of these issues irre-spective of the type of hospitals considered. The remaining common three issues in phase 2 (i.e. recognizing IT as a keystakeholder in major hospital decisions, managing demandsand expectations for IT services, and IT competing with otherclinical priorities for scarce resources) were only identiedby either one or two panels in phase 1. This indicates thatalthough not all panels may have originally identied theseissues, IT experts recognized their relevance in the secondphase of the Delphi survey and retained them among the listof most important challenges. These three issues are directlyrelated to the preoccupation of IT executives and the strate-gic positioning of IT in the organization, whereas the rst sixissues (also common in phase 1) are more organizational in

8/12/2019 1-s2.0-S1386505611001985-main

7/13


Table 3 Common IT management issues in phase 1 and in phase 2.

Common IT management issues reported in phase 1 by all three panelsPrioritizing and implementing IT initiatives with other health organizationsImplementation of an EHRAligning strategy across the hospitals, LHIN, and the provinceStaying current with the rapid changes in technologyHaving sufcient funds

Recruiting and developing IT staff with the appropriate skill set (i.e. PM, HL7)Adequate and reliable networking between health care providersAdoption of IT by clinicians and staff Time and cost involved in training IT staff and usersProviding access to appropriate health information as per privacy legislation requirementsAvailability of provincial standards and best practices for technology

Common IT management issues retained in phase 2 by all three panelsPrioritizing and implementing IT initiatives with other health organizationsImplementation of an EHRAligning strategy across the hospitals, LHIN, and the provinceStaying current with the rapid changes in technologyHaving sufcient fundsRecruiting and developing IT staff with the appropriate skill set (i.e. PM, HL7)Recognizing IT as a key stakeholder in major hospital decisionsManaging demand and expectations for IT servicesIT competing with other clinical priorities for scarce resources

nature (e.g., implementation of an EMR and staying currentwith the rapid changes in technology).

3.3. Phase 3: ranking

In the nal phase of the Delphi study, the participants in eachof the three panels were asked to rank in order of impor-tance the IT management issues retained in phase 2. For thispurpose, a separate list of randomly ordered IT managementissues was sent to each respective panel based on the resultsof the preceding phase.

In the rst round of the ranking phase, the level of agree-ment was fair for all three panels: W =0.22 for the ruralhospitals panel; W = 0.36 for the community hospitals panel;and W = 0.36 for the academic hospitals panel. Subsequently,a second round of ranking was conducted in all groups, whichproduced a higher level of consensus among the participantsin each panel. The new list of rankedIT management issues ispresented in Table 4 . In this second round, the level of agree-mentwas moderate forall three panels: W = 0.41 forthe ruralpanel; W = 0.54 for the community panel; and W =0.43 for theacademic panel.

The ranking exercise conrms that there are similaritiesbetween the three panels on certain IT management issuesthat were retained by all three panels initially. Besides having sufcient funds, three of the top IT management issues wererelated to the strategic positioning of IT within the hospitaland represent intimate preoccupations of CIOs/IT directorsin hospitals. These include: managing demands and expec-tations for IT services in light of the multiple internal andexternal stakeholders demands and growing users expecta-tions; IT competing with other clinical priorities for scarceresources in light of the xed hospital budget and increas-ing spending in IT; and recognizing IT as a key stakeholderin major hospital decisions in light of the pressure on IT todeliver solutions. Nevertheless, the three types of hospitalshave particularities and different congurations that result

in different types of preoccupations and challenges. This hasbeen reected in the nal ranking results whereas certainIT management issues were unique to each of the panels(Table 4 ).

4. Discussion

Although the three types of hospitals considered in this studydiffer in terms of their characteristics and environments, aconsiderable number of IT management issues were consis-tent across the panels, which highlights their relevance inthe context of public hospitals in general. Some of the topcommon issues that were identied by the three panels wererelated to the strategic positioning of IT within the hospital(e.g., managing demands and expectations for IT services, ITcompeting with other clinical priorities for scarce resources,andrecognizing IT as a key stakeholder in major hospital deci-sions).

Nevertheless, in lightof the particularities and unique con-gurations of hospitals in each of the three panels, therewere differences in preoccupations and challenges betweenthem. This further contributes to the value of our ndingsand underscores the importance of including three panels.Interestingly, community hospitals seem to have some pre-occupations that are common with both rural and academichospitals as indicated by our results. Given the fact that com-munityhospitalsare in general larger thanrural hospitals, andas such have more resources and partnerships with academichospitals, they face similar issues as the latter in relation toengagingclinical leadership in IT projects, leveragingbusinessintelligence tools to report on performance and outcomes,leveraging portfolio and project management, and ability torecognize the value of IT as partner in improving patientcare. In fact, engaging clinical leadership in IT projects, whichranked sixth and seventh in the academic and communitypanels, respectively, underscores the need for strong clinical

8/12/2019 1-s2.0-S1386505611001985-main

8/13


Table 4 Final ranking of key IT management issues in phase 3.

Ranks IT management issues

O R C A1 3 2 2 Having sufcient funds2 1 1 3 Managing demands and expectations for IT services3 7 3 1 IT competing with other clinical priorities for scarce resources4 4 6 5 Recognizing IT as a key stakeholder in major hospital decisions5 9 14 4 Aligning strategy across the hospitals, LHIN, and the province6 6 8 9 Implementing an EHR7 6 20 12 Prioritizing and implementing IT initiatives with other health organizations8 15 16 10 Recruiting and developing IT staff with the appropriate skill set9 13 19 16 Staying current with the rapid changes in technology

2 15 Merging and integrating many different systems within the hospital5 4 Unrealistic expectations for IT to solve all problems8 5 Keeping infrastructure current evergreening

10 9 Maximizing resource potential to manage IT with minimal staff 11 Availability of provincial standards and best practices for technology12 Enforcing security policies through appropriate security measures13 Managing external technological threats such as malware and viruses14 11 Adoption of IT by clinicians and staff 16 Time and cost involved in training IT staff and users

7 6 Engaging clinical leadership in IT projects10 14 Leveraging BI tools to report on performance and outcomes11 Prioritizing short-term xes over strategic execution12 7 Ability to recognize the value of IT in improving care13 Ensuring cost containment17 Dependence on online systems in the absence of redundant solutions18 15 Leveraging portfolio and project management

8 Establishing capabilities for p rocess redesign13 Immature software industry17 Deploying foundational technology services (e.g., RFID, kiosks)

O =overall rank; R = rural panel; C =community panel; A =academic panel.

informatics teams in order to support process reengineering

and improvements enabled by technology. The emerging andcritical position of Chief Medical Information Ofcer (CMIO)in large centers may also play a positive role in this respect bycreating a bridge between the medical, administrative andIT community in a hospital, and providing strong leadershipfor clinicians involvement in IT projects [39,40]. As such, itmay also enhance the ability to recognize the value of IT inimproving care, which ranked seventh and twelfth in the aca-demic and community panels, respectively. Future researchshould explore various mechanismsand approaches thatmayfacilitate clinical engagement in IT projects. Another interest-ing nding in the academic and community panels relatesto leveraging business intelligence tools to report on perfor-

mance, andportfolio andproject management methodologiesto adequately manage the increasing number and size of ITprojects in larger hospitals. Further research in these areaswould allow a better assessment of the best practices in hos-pitals and an evaluation of the effectiveness of the variousapproaches used in these settings.

Ontheotherhand,community hospitalshavemore limitedresources and capabilities than academic hospitals, attractdifferent types of health professionals (no afliation withteaching medical schools and health sciences), and caterfor different types of patients (less complex cases). Thus,they share some characteristics with rural hospitals and facesimilar issuesas the latter in relation to the unrealistic expec-

tations of IT, maximizing resource potential to manage IT

with minimal staff, merging and integrating different sys-

tems, and keeping the infrastructure current. These ndingsunderscore fruitful areas for further exploration. For exam-ple, rural managers showed greater concern over merging andintegrating different systems within the hospital compared tothe other two panels. This may be the result of more legacysystems present in rural hospitals, or it may be associatedwith the limited IT resources that areavailable (e.g., maximiz-ing resource potential to manage IT with minimal staff wasretained by the rural panel as one of the critical IT manage-ment issues). Along the same line, the ranking of the issuekeeping infrastructure evergreening by the rural and com-munity panels (it was not retained by the academic panel)indicateschallenges faced in these settings in keeping upwith

the rapid changes in infrastructure and increasing the breadthand depth of computer-based applications. Unrealistic expec-tations for IT to solve all problems, which ranked fth andfourth in the rural and community panels, respectively, repre-sents an interesting nding that is worth further explorationin order to betterunderstandits underlying causesand how toaddress it.

Several IT management issues generated in this Delphistudy have been previously discussed in the medical infor-matics literature although not systematically and rigorouslyassessed in the context of hospitals. For example, the lack of sufcientresources [27,28]thathasbeendiscussed in previousresearch was also identied in this study by the three pan-

els. Other issues related to the adoption of IT by clinicians

8/12/2019 1-s2.0-S1386505611001985-main

9/13


[29], privacy and security concerns [27,28], the volatility of the technology market and the fast pace of evolution of ITsolutions [29,30] were also reported by specic panels in thisstudy although not consistently reported by all three groups.

When comparing the IT management issues generated inthis study with those reported in prior Delphi studies pub-lished in the IS literature, we notice considerable overlapdespite some ne differences associated with the specicityof the health care environment. For example, in the two mostrecent studies on key issues for IT executives [24,25], the topten IT management issues included: IT and business align-ment, building business skills in IT, attracting and retaining new IT professionals, and business agility and speed to mar-ket. These issues were similar to what was reported in thisstudy in relation to aligning strategyacross the hospitals, localhealth network and province, recruiting and developing ITstaff with the appropriate skill set, and staying current withthe rapid changes in technology. Yet, this study presentsaddi-tionalcontributionand valueto what iscurrentlyknown in theIS literature. Indeed, it generates industry-specic results thatshed light on thepreoccupations andconcernsof IT managersin public hospitals, and are representative and informative of what is happening in the health care sector.

In the past, there has been a lack of comprehensive under-standing and rigorous assessment of the challenges andopportunities faced by CIOs/IT directors in healthcare organi-zations, which hindered the appropriate investment of effortsandresources in neededareas. Although this study was basedon a limited number of self-selected participants, it employeda systematic approach that hasnot been used inprior researchin this area, and followed the recognized criteria that con-tribute to its rigor. First, a detailed description of the expertsrecruitment and selection process was provided, which high-lighted the breadth of perspectives (i.e. three panels fromdifferent settings) that ensure the validity of the results [41]andreduce potential distortion of data [26]. Second, thesurveyfollowed the recommended steps in the three phases of theDelphi method [32], and provided clear and detailed instruc-tions to experts in the data collection process. Last, data werethoroughly analyzed in each phase, and the study was con-cluded when a moderate level of consensus were reached ineach of the three panels.

In addition, the diverse sample that has been considered(as indicated by the three panels of experts) further increasesthe generalizability of ndings and their relevance to vari-ous types of hospitals in industrialized countries. Althoughthe list of IT management issues emerged in the context of Canadian hospitals, it is fairly comprehensive and may begeneralizable to similar environments. There is no evidencethat the key issues identied in the context of this studyare unique to Canadian hospitals; they may be applicable inother industrialized countries as was observed in IS studieswhere similarities in the economic development of countrieswas paralleled bysimilaritiesin the identiedIT managementissues identied (e.g., [20,42]. In this line of thought, a recentcomparison of electronic medical record systems adoption inseven industrialized countries (Canada, U.S., U.K., Germany,Netherlands, Australia, and New Zealand) revealed relativelylow, yet comparable, rates of diffusion of these systems inhospital settings [2]. This may further be associated with

similarities in the challenges and issues faced by hospitalsin these countries. Future research may replicate this studyin other countries in order to gain a better understanding of similaritiesanddifferencesin theIT management issues facedby hospitals. In addition, as a next step, a large scale survey of CIOs/IT directors in hospitals should be developed in order togain a more comprehensive understanding of the importanceandsignicance of thevarious IT management issues in thesesettings.

This studypresents a signicant contributionto the eld of medical informatics by being therst to identify andprioritizecritical IT management issues in public hospitals. Althoughsome of the prioritized IT management issues may not besurprising to CIOs/IT directors in hospitals, the rigor and thor-oughness in this study support the use of the ndings as abenchmark and diagnostic tool for strategic decision mak-ing related to IT. For example, CIOs/IT directors may usethe prioritized lists internally to negotiate with their coun-terparts (e.g., nance and human resources) for additionalbudget or resources. In fact, in a hospital setting where theprovision of services and quality care are driven by evidence-based practice, management is beingchallenged to usesimilarevidence-based approaches. In this case, CIOs/IT directorsmay inform and support their decision making and internalnegotiations with evidencegenerated in this study in theformof the prioritized lists of IT management issues. This hasbeenechoed by discussions with some of the respondents follow-ingthe conclusion of the studywho indicatedthatthe ndingspresent strong evidence that validate their experience. Theyindicated that the evidence generated in this study may beused to support the development and execution of a hospitalstrategic plan and show that a CIO/IT directors needs are inline with what his/her peers are experiencing in other hospi-tals.

The broadening of our understanding of the challengesfaced by IT executives in hospitals would support a moresystematic evaluation of these issues over time, and allowmanagement, educational, and research resources to beinvested in the appropriate areas. At the management level,the list of prioritized issues provides relevant informa-tion which may be used to nd appropriate strategies andapproaches for addressing them. It may also be used byCIOs/IT directors as guidelines to understand where theirhospitals stand compared to other hospitals with similarenvironments and characteristics. Panelists have particularlyshown interest in obtaining access to the output of thisresearch given the comprehensive and grounded nature of the list of IT management issues. The list may also be usedby software providers and IT consultants to provide ser-vices that meet the needs of hospitals when opportunitiesarise.

At the educational level, new opportunities may be identi-ed for training of IT professionals such as developing projectmanagement skills. At the policy level, it is critical to havea thorough understanding of the areas that are critical andrequire urgent attention in hospitals over the upcoming yearsin order to address them andsupport the national IT develop-ment agenda. The comparable views regarding some of theseissues across different types of hospitals build a strong caseas to their relevance and importance in hospitals in general.

8/12/2019 1-s2.0-S1386505611001985-main

10/13


Summary pointsWhat was already known on the topic

Emerging information technologies are increasinglyused to addresssome of themajorchallengesin healthcare organizations.

IT executives in the health care sector are fac-ing various challenges which unless recognized andaddressed, may hinder progress in this area.

IT management issueshave beensystematically inves-tigated in the Information Systems literature since theearly 1980s while the medical informatics literaturehas lagged behind with respect to this research topic.

Only scattered evidenceexistswithregardto themajorIT management issues in health care organizations.

What this study added to our knowledge

Top IT management issues identied by thethreepan-els of experts were related to the strategic positioning of IT and the IT function within hospitals.

In light of the unique congurations of rural, com-munity and academic hospitals, differences wereobserved in terms of topIT management issues acrossthe three groups.

The ndingsvalidated the experience of CIOs/IT direc-tors andprovided evidencein theform of a benchmarkanddiagnostic tool that may be used to support strate-gic decision making related to IT.

Management, educational, and research resourcesmay be invested to address the IT management issuesin the relevant areas.

Last, at the research level, it is important to monitor the evo-lution of IT management issuesover time, and investigate thedifferent ways to addressthem inorder to supportthe progressof IT development, use, and impacts in hospitals.

5. Conclusion

This study presented the ndings of a Delphi study that wasconducted withCIOsandIT directors in rural, community, and

academic hospitals in the most populated Canadian provinceand one of the largest health care jurisdictions in the coun-try. More specically, we followed a systematic and rigorousapproach working toward the development of an authorita-tive list of IT management issues in public hospitals. Sincethe three lists of IT issues generated in this study were basedon the input of experienced IT executives, they have strong foundation and face validity. Nevertheless, as a next step ourDelphi approach must be replicated in other countries andhealth care settings for generalizability purposes and in orderto compare similarities and differences in IT managementissues between different environments.

Authors contributions

All authors (Dr. Mirou Jaana, Dr. Haitham Tamim, Dr. GuyPar, and Ms. Mari Teitelbaum) contributed substantially tothe manuscript entitled Key IT Management Issues in Hospi-tals: Results of a Delphi Study in Canada.

Conict of interest statement

The manuscript entitled Key IT Management Issues in Hospi-tals: Results of a Delphi Studyin Canada, whichwas preparedby Dr. Mirou Jaana, Dr. Haitham Tamim, Dr. Guy Par, and Ms.Mari Teitelbaum, does not present any conict of interest.

Acknowledgments

The Telfer School of Management at the University of Ottawaand the Canada Research Chairs Program are gratefullyacknowledgedfor providing nancialsupport for thisresearchproject.We also thank the30 dedicatedexpertswho madethisresearch possible.

Appendix A. List of IT management issuesgenerated in phase 1

IT management issues Brief description

Having sufcient funds Capital and operating funds are required to deliver the technology being requested of hospital IT departments. Lack of nancial resources fromthe hospital and government funding (limit in increases in health carespending) needed to complete mandated requests is a challenge for theimplementation and sustainability of IT

Managing demand and expectationsfor IT services

Conicting demands from multiple internal and external stakeholders,demands from an increasingly computer-literate user base, and themaintenance required to keep the systems available are signicant.Growing users expectations of technology and its ability to transformhealth care

8/12/2019 1-s2.0-S1386505611001985-main

11/13



IT competing with other clinicalpriorities for scarce resources

With a xed hospital budget, it is difcult for IT projects to requestadditional budget and justify spending that might come at the expense of other clinical projects

Recognizing IT as a key stakeholderin major hospital decisions

There are projects within hospitals today that do not have a technologycomponent or impact. IT is expected to deliver solutions after decisionsare made, whereas IT should be considered a key stakeholder in majorhospital decisions

Aligning strategy across thehospitals, Local HealthInformation Network (LHIN), andthe province

With hospitals, LHIN and the province all doing strategic planning, thestrategies must be aligned to ensure consistent direction and avoidconict between initiatives

Implementation of an EMR Providers and patients recognize the need for an EMR to support clinicalinformation sharing, but also the challenge associated with the processof implementation (shift from existing paper health record) andcompletion of an EMR

Prioritizing and implementing ITinitiatives with other healthorganizations

Leaders recognize the need to work with others in the community ontechnology projects but struggle to manage multiple projects, and alignpriorities with differing strategic objectives

Recruiting and developing IT staff with the appropriate skill set (i.e.PM, HL7)

Shortage in qualied IT personnel with health care specic skills (e.g.,HL7 and project management), aging workforce, and challenge inattracting and retaining these professionals present threats to theeffective implementation and use of IT

Staying current with the rapidchanges in technology

The rate at which new technology is being introduced is continuouslyincreasing, which necessitates that IT staff remain current with industrytrends and have a good understanding of possible replacements

Merging and integrating the manydifferent systems within thehospital

The evolution of IT within the hospital has resulted in a fragmentedenvironment, making it difcult to merge and integrate these systems asworkows and providers have become dependent on them

Unrealistic expectations for IT tosolve all problems

Many users (e.g., administrative staff, medical staff) expect technology tosolve pre-existing problems without considering the process or availableresources

Keeping infrastructure current(evergreening)

Hospital technology infrastructure must be replaced regularly in light of the rapid changes and increasing breadth and depth of computer-basedapplications in order to ensure a high-quality and stable environment

Maximizing resource potential tomanage IT with minimal staff

Ensuring staff are allocated appropriately to maximize their potential willhelp keep IT costs down

Availability of provincial standardsand best practices for technology

Each hospital is developing its own solutions with inconsistencies insystems standards and applications that limit province wide integration.Provincial standards and best practices would ensure that knowledge andexperience are shared across sites

Enforcing security policies throughappropriate security measures

The enforcement of hospital security policies necessitates theintroduction of technology security solutions that are difcult toimplement and often have low adoption by users

Managing external technologicalthreats such as malware andviruses

With the hospital systems interfacing with the external environment,protection from threats such as malware and viruses is a problem evenwith up-to-date defensive software

Adoption of IT by clinicians and staff Fear of change may impede progress, and technology solutions will onlybe successful if adopted by clinicians and staff who are engaged in theprocess; it is harder to manage people issues than technology issues

Time and cost involved in training ITstaff and users

As the number and complexity of applications within a hospital increase,the time and cost involved in staff training on system/process changesincrease, which is problematic given limited resources

Engaging clinical leadership in ITprojects

Clinicians support for and engagement in IT projects is a critical successfactor to facilitate process reengineering and improvements enabled bytechnology; need for strong clinical informatics team

8/12/2019 1-s2.0-S1386505611001985-main

12/13



Leveraging business intelligence (BI)tools to report on performanceand outcomes

As hospitals gather information, it should be accessed using businessintelligence tools to begin reporting on hospital performance andoutcomes (e.g., quality and safety metrics)

Prioritizing short-term xes overstrategic execution

More time spent on xing problems than doing the right thing withsenior management focus on reghting rather than long termsolutions, leading to reactive rather than proactive operations

Ability to recognize the value of IT inimproving care

Perception of IT as help desk rather than partner for improving patientcare; difculty in recognizing the value of IT in improving care

Ensuring cost containment Increasing pressure for cost containment given the limited increase inhealth care spending; technology costs can spiral out of control and mustbe closely managed

Dependence on online systems inthe absence of redundantsolutions

As clinicians become dependent on online systems, redundancy isrequired to ensure high availability, but is costly and often notimplemented

Leveraging portfolio and projectmanagement

With the number and size of IT projects increasing, portfolio and projectmanagement methodologies are necessary to manage the work withinthe department and across the hospital

Establishing capabilities for processredesign

To increase technology project success, redesign of the processesimpacted by the new technology should be considered

Immature software industry The immature health care software industry has resulted in many smallsoftware providers that do not prioritize integration

Deploying foundational technologyservices (e.g. RFID, kiosks)

Foundational (utility) technology services that support hospitaloperations (e.g., RFID for asset tracking) are required

Access and collection of timely andappropriate information byfront-line staff

Technology can deliver critical information to providers only if it iscollected and accessible to them in a timely and appropriate way

Adequate and reliable networking between health care providers

Reported inadequate and incomplete computer networking linkagesbetween health care providers prohibits the delivery of criticalinformation in a timely and secure way, and unreliable wirelessenvironments

Developing a credible and reliableworking relationship with hospitalstaff

Building trust and a good working relationship between hospital and ITstaff (e.g., responsiveness, reliable operations, user engagement) isnecessary for project success

Developing user friendly systems The development and implementation of user friendly technologysolutions and point of care systems (e.g., order entry, clinical charting) isrequired to support user adoption and maximize time with patients

Effect of mobile devices and socialmedia on patient privacy

As information becomes increasingly accessible through mobile devicesand social media, patient privacy is put at risk

Establishing the right ITinfrastructure

With ever increasing technology load and changes, it is challenging tocreate and implement the appropriate IT architecture (network,equipment and software) to support the hospitals need in a scalable andstable way

Providing access to appropriatehealth information as per privacylegislation requirements

Adherence to complex and evolving privacy legislation that denesaccess to personal health information is challenging, while ensuring providers have all the information they need for the best possible carewithin and across organizations

Regaining public support in thewake of the e-health scandal

The e-health scandal of misspent funds on health technology in Ontariohas eroded public support for e-health projects

r e f e r e n c e s

[1] B. Chaudhry, J. Wang, S. Wu, et al., Systematic review: impactof health information technology on quality, efciency, andcosts of medical care, Ann. Intern. Med. 144 (2006) E12E22.

[2] A.K. Jha, D. Doolan, D. Grandt, T. Scott, D. Bates, The use of health information technology in seven nations, Int. J. Med.Inform. 77 (2008) 848854.

[3] S. Koch, Home telehealth current state and future trends,Int. J. Med. Inform. 75 (2006) 565576.

8/12/2019 1-s2.0-S1386505611001985-main

13/13


[4] M. Jaana, G. Par, Home telemonitoring of patients withdiabetes: a systematic assessment of observed effects, J.Eval. Clin. Pract. 13 (2) (2006) 242253.

[5] G. Par, M. Jaana, C. Sicotte, Systematic review of hometelemonitoring for chronic diseases: the evidence base, JAMIA 14 (3) (2007) 269277.

[6] M. Jaana, G. Par, C. Sicotte, Home telemonitoring forrespiratory conditions: a systematic review, Am. J. Manag.Care 15 (5) (2009) 313320.

[7] D.W. Bates, R.S. Evans, H. Murff, P.D. Stetson, L. Pizziferri, G.Hripcsak, Detecting adverse events using informationtechnology, JAMIA 10 (2) (2003) 115128.

[8] R. Kaushal, K.G. Shojania, D.W. Bates, Effects of computerized physician order entry and clinical decisionsupport systems on medication safety, Arch. Intern. Med.163 (2003) 14091416.

[9] Canada Health Infoway, Electronic Health Records 2015:Canadas Next Generation of Health Care at a Glance.Available from: http://www2.infoway-inforoute.ca/Documents/Vision Summary EN.pdf (Internet;cited 2011 January 14).

[10] E. Coiera, Building a national health IT system from themiddle out, JAMIA 16 (3) (2009) 271273.

[11] J. Anderson, S. Tang, C. Blue, Health care reform and theparadox of efciency: writing in culture, Int. J. Health Serv.37 (2) (2007) 291320.

[12] Canadian Institute for Health Information (CIHI), HealthCare in Canada, 2007, Available athttp://secure.cihi.ca/cihiweb/products/hcic2007 e.pd f (Internet; cited 2011 January 14).

[13] Canadian Medical Association (CMA), InformationTechnology and Health Care in Canada: 2007 Status Report,2007, Available at http://prismadmin.cma.ca/multimedia/CMA/Content Images/Inside cma/HIT/2007 status report/IT-handbook-e.pdf (Internet; cited 2011 January14).

[14] J.C. Brancheau, J.C. Wetherbe, Key issues in informationsystems management, MIS Quart. 11 (1) (1987) 2345.

[15] F. Niederman, J.C. Brancheau, J.C. Wetherbe, Informationsystems management issues for the 1990, MIS Quart. 15 (4)(1991) 475500.

[16] J.C. Brancheau, B.D. Janz, J.C. Wetherbe, Key issues ininformation systems management: 199495 SIM Delphiresults, MIS Quart. 20 (2) (1996) 225242.

[17] S. Dekleva, J. Zupancic, Key issues in information systemsmanagement: a Delphi study in Slovenia, Inform. Manage.31 (1996) 111.

[18] C.E. Pollard, S.C. Hayne, A comparative analysis of information systems issues facing Canadian business, in:Proceedings of the 29th Annual Hawaii InternationalConference on System Science, 1996, pp. 6877.

[19] H.-L. Yang, Key information management issues in Taiwanand the U.S., Inform. Manage. 30 (1996) 251267.

[20] D. Pimchangthong, M. Plaisent, P. Bernard, Key issues ininformation systems management: a comparative study of academics and practitioners in Thailand, JGITM 6 (4) (2003)2744.

[21] J. Luftman, E.R. McLean, Key issues for IT executives, MISQuart. Exec. 3 (2) (2004) 89104.

[22] J. Luftman, Key issues for IT executives, MIS Quart. Exec. 4(2) (2005) 269285.

[23] P. Inedo, Key information systems management issues inEstonia for the 2000 and a comparative analysis, JGITM 9 (2)(2006) 2244.

[24] J. Luftman, R. Kempaiah, E.H. Rigoni, Key issues for ITexecutives 2008, MIS Quart. Exec. 8 (3) (2009) 151159.

[25] J. Luftman, T. Ben-Zvi, Key issues for IT executives 2009:difcult economys impact on IT, MIS Quart. Exec. 9 (1) (2010)4959.

[26] A. Delbecq, A. Van de Ven, D. Gustafson, Group Techniquesfor Program Planning, Scott Foresman and Company,Glenview, IL, 1975.

[27] S. Szydlowski, C. Smith, Perspectives from nurse leaders andChief Information Ofcers on health information technologyimplementation, Hosp. Top. 87 (1) (2009) 39.

[28] C.C. Diamond, C. Shirky, Health information technology: afew years of magical thinking? Health Affairs 27 (5) (2008)383390.

[29] E.H. Shortliffe, T. Shortliffe, Strategic action in healthinformation technology: why the obvious has taken so long,Health Affairs 24 (5) (2005) 12221233.

[30] D.A. Guise, Health information systems challenges: theHeidelberg conference and the future, Int. J. Med. Inform. 69(2003) 105114.

[31] H.M. Donohoe, R.D. Needham, Moving best practice forward:Delphi characteristics, advantages, potential problems, andsolutions, Int. J. Tourism Res. 11 (5) (2009) 415437.

[32] R.C. Schmidt, Managing Delphi surveys using nonparametric statistical techniques, Decis. Sci. 28 (3) (1997)763774.

[33] R. Schmidt, K. Lyytinen, M. Keil, P. Cule, Identifying softwareproject risks: an international Delphi study, JMIS 17 (4) (2001)536.

[34] M. Jaana, M.M. Ward, G. Par, C. Sicotte, Antecedents of clinical information technology sophistication in hospitals,Health Care Manage. Rev. 31 (4) (2006) 289299.

[35] G. Pare, M. Jaana, C. Sicotte, Exploring health informationtechnology innovativeness and its antecedents in Canadianhospitals, Methods Inf. Med. 49 (2010) 2836.

[36] Ontario Hospital Association (OHA). Available atwww.oha.ca (Internet; cited 2010 January 21).

[37] Ontario Hospital Association (OHA) and the Government of Ontario, Hospital Report: Acute Care, 2007, Available athttp://secure.cihi.ca/cihiweb/products/OHA ED07 chpt5 e.p df (Internet; cited 2011 January 14).

[38] C. Okoli, S.D. Pawlowski, The Delphi method as a researchtool: an example, design considerations and application,Inform. Manage. 42 (2004) 1529.

[39] J. Catapano, Elevating the role of medical informationalofcer, Physician Exec. 34 (3) (2008) 3437.

[40] L.A. Runy, The changing role of the CMIO, Hosp. HealthNetw. 82 (2) (2008) 3742.

[41] H.A. Linstone, M. Turoff (Eds.), Delphi Method: Techniquesand Applications, U.S., 2002. Available athttp://is.njit.edu/pubs/delphibook/index.html .

[42] P.C. Palvia, S.C.J. Palvia, J.E. Whitworth, Global technology: ameta analysis of key issues, Inform. Manage. 39 (2002)403414.
http://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004http://www2.infoway-inforoute.ca/Documents/Vision_Summary_EN.pdfhttp://secure.cihi.ca/cihiweb/products/hcic2007_e.pdfhttp://prismadmin.cma.ca/multimedia/CMA/Content_Images/Inside_cma/HIT/2007_status_report/IT-handbook-e.pdfhttp://prismadmin.cma.ca/multimedia/CMA/Content_Images/Inside_cma/HIT/2007_status_report/IT-handbook-e.pdfhttp://www2.infoway-inforoute.ca/Documents/Vision_Summary_EN.pdfhttp://prismadmin.cma.ca/multimedia/CMA/Content_Images/Inside_cma/HIT/2007_status_report/IT-handbook-e.pdfhttp://www.oha.ca/http://secure.cihi.ca/cihiweb/products/OHA_ED07_chpt5_e.pdfhttp://is.njit.edu/pubs/delphibook/index.htmlhttp://is.njit.edu/pubs/delphibook/index.htmlhttp://secure.cihi.ca/cihiweb/products/OHA_ED07_chpt5_e.pdfhttp://www.oha.ca/http://prismadmin.cma.ca/multimedia/CMA/Content_Images/Inside_cma/HIT/2007_status_report/IT-handbook-e.pdfhttp://prismadmin.cma.ca/multimedia/CMA/Content_Images/Inside_cma/HIT/2007_status_report/IT-handbook-e.pdfhttp://secure.cihi.ca/cihiweb/products/hcic2007_e.pdfhttp://www2.infoway-inforoute.ca/Documents/Vision_Summary_EN.pdfhttp://localhost/var/www/apps/conversion/tmp/scratch_6/dx.doi.org/10.1016/j.ijmedinf.2011.07.004