1

Perceptions and experiences on using computerized clinical decision support systems to implement recommendations:

a qualitative evidence synthesis

Maertens Jan, Persoons Elisabeth, Stevens Nick, Vandenbempt Inge, KU Leuven

Supervisor: Vermandere Mieke, PhD, KU Leuven, ACGH, EBMPracticeNet vzw

Co-supervisor: Van de Velde Stijn, Norwegian Institute of Public Health, KU Leuven

Master of Family Medicine

Master’s Thesis Family Medicine 2018 – 2019

2

This Master’s Thesis is an exam document that was not corrected for possibly assessed errors after the defence. Without written permission of the authors and the promotors, copying, adopting, using or realizing this edition or parts of this edition is forbidden. For requests or information regarding the adopting and/or using and/or realization of parts of this publication, please turn to the university where the authors are subscribed.

Prior written permission of the promotors is also required for the application of the described (original) methods, products, circuits and programs for industrial or commercial purpose and for the submission of this publication for participation in scientific contests.

Deze masterproef is een examendocument dat niet werd gecorrigeerd voor eventueel vastgestelde fouten. Zonder voorafgaande schriftelijke toestemming van zowel de promotor(en) als de auteur(s) is overnemen, kopiëren, gebruiken of realiseren van deze uitgave of gedeelten ervan verboden. Voor aanvragen tot of informatie i.v.m. het overnemen en/of gebruik en/of realisatie van gedeelten uit deze publicatie, wendt u tot de universiteit waaraan de auteur is ingeschreven.

Voorafgaande schriftelijke toestemming van de promotor(en) is eveneens vereist voor het aanwenden van de in dit afstudeerwerk beschreven (originele) methoden, producten, schakelingen en programma’s voor industrieel of commercieel nut en voor de inzending van deze publicatie ter deelname aan wetenschappelijke prijzen of wedstrijden.

3

Acknowledgements

This master’s thesis has been a real journey for the four of us: a journey to discover qualitative research and possibilities and limitations of decision supports, but also a journey to Norway.

We all took off, two years ago, with great interest in and curiosity for this special project. Getting to know the basic principles of qualitative research was one of the first steps. We were invited to go to Oslo, the home base of Stijn Van de Velde, and were taught enthusiastically about qualitative research by Claire Glenton, Simon Lewin, Signe Flottorp and Karin Hannes and the team of the Norwegian Institute of Public Health. We learned how to assess CERQual levels of evidence and developed a heart for research. None of this would have been possible without the patience and support of our mentors. Therefore, we would like to thank these wonderful people for their unconditional support. We are proud to have been part of the GUIDES project.

A special thank you to Mieke Vermandere and Stijn Van de Velde. During this journey, every step was supported and adjusted with great patience and care.

Last but not least, we would like to thank our families, partners and friends for the support, the grammar checks, the listening ear and the understanding.

4

Perceptions and experiences on using computerized clinical decision support systems to implement recommendations: a qualitative evidence synthesis Maertens Jan, Persoons Elisabeth, Stevens Nick, Vandenbempt Inge, Van de Velde Stijn, Vermandere Mieke

Abstract

Background: A clinical decision support (CDS) is a technology that uses patient-specific data to provide relevant medical knowledge at the point of care.

The objective of this study is to synthesize qualitative research evidence on the perceptions and experiences regarding the use of CDS to support the implementation of recommendations. This study was conducted in close cooperation with the authors of the GUIDES framework, a tool developed to assist professionals when implementing CDS, to further advance the existing framework.

Methods: We conducted a qualitative evidence synthesis using studies identified through a previous literature search, reference lists of relevant studies, contacting authors of potentially relevant articles included in a previous meta-aggregation study and suggested articles by Google Scholar and Mendeley. Study titles and abstracts were screened against inclusion criteria. Full articles were assessed for eligibility and sampled based on data richness and relevance. Findings were constructed through a framework analysis using the existing GUIDES framework. Levels of confidence were assigned using the GRADE-CERQual approach.

Results: We identified 219 studies, out of these; we included 21 studies in our final sample. 31 findings were constructed and organized using the GUIDES framework. Important findings include that the CDS be easy to use, time efficient and that information about the system is adequate.

Conclusion: Findings identified in this study can be used to improve the existing GUIDES framework, in order to guide developers to successfully implement CDS in clinical practice.

5

TABLE OF CONTENTS

ACKNOWLEDGMENTS……………………………………………………………………………………………………………3

ABSTRACT……………………………………………………………………………………………………………………………..4

ARTICLE

1. BACKGROUND……………………..........................................................................................6

2. OBJECTIVES....................................................................................................................7

3. METHODS ......................................................................................................................7

4. RESULTS ......................................................................................................................10

5. DISCUSSION................................................................................................................. 22

6. CONCLUSIONS .............................................................................................................26

REFERENCES.............................................................................................................................26

APPENDIX

Table 1 – trial sibilings indentified in the guides project…………………….…………………….30

Table 2 – included but not sampled studies…………………………………………………..…………31

Table 3 – sampled studies………………………………………..………………………………………………33

Table 4 – characteristics of included studies…………………………………………………………….35

CerQual evidence profile.………………………………………..……………….………………………...…..41

Example letter………………………………………………………………………………………………………….55

6

Perceptions and experiences on using computerized clinical decision support systems to implement recommendations: a qualitative evidence synthesis

1. Background

Making medicine evidence-based is a way to improve quality of care. The knowledge regarding evidence-based medicine continues to improve. However, the implementation of evidence in everyday practice remains challenging. To meet this challenge, there has been a growth in the quality and development of clinical decision support systems in the past decades(1).

Alongside the development of electronic health records and other medical computer systems, computerized clinical decision support systems (CDS) have occupied a special place in the implementation of guidelines and therefore evidence-based medicine in clinical practice. A CDS is an information technology designed to aid clinicians and patients in making clinical decisions, based on patient-specific data(2).

CDS have been proven to be effective by a multitude of studies. Murphy provides us with a literature review(3). CDS can be effective in the use of preventive services(4), to improve clinical outcomes(5), and to increase the uptake of practice guidelines (6).

However, some studies found no significant improvement in process outcome(7,8) or mortality outcomes(9). Therefore, the investments that have been made may not always result in value for money.

Reasons for the variability in effectiveness of CDS are listed by Kawamoto(10) and Roshanov (11). They found a paradoxal effect of reminders at the point of care: sometimes, guidelines were less frequently followed when an electronic reminder was available. Alert fatigue and distractiveness of the system are possible explanations for this undesirable effect.

The GUIDES-framework(12) was made to improve the impact of CDS through optimized implementation, based on high-quality and trustworthy decision support content. The authors executed a literature review of frameworks, systematic reviews, process evaluations and qualitative evidence pertaining to factors for successful CDS implementation. With the resulting factors, they built the GUIDES checklist: a tool to assist professionals when implementing guidelines with computerized decision support. Their four pillars of a successful CDS are enabling the context, an appropriate content, an effective system and an effective implementation. Furthermore, sixteen minor factors are identified as building blocks of the resulting checklist.

7

2. Objectives

The overall aim of this review is to explore factors affecting the use of computerized decision support.

The review has the following objectives:

- To identify, appraise and synthesize qualitative research evidence on the barriers facilitators to the use of CDS;

- To compare the resulting factors with GUIDES checklist.

Qualitative research was explained by Shank (2006) as ‘a systematic empirical inquiry into meaning’(13). It is proven useful to explore how people experience situations, to find barriers and facilitators in their behavior, and why interventions work, or don’t work(14,15). In the field of healthcare, analysis and synthesis of qualitative evidence can identify contextual factors and the impact of outcomes on patients and caregivers(16,17). Since we were interested in opinions of stakeholders regarding factors influencing the use of CDS, a qualitative evidence synthesis was our method of choice.

3. Methods 3.1 Criteria for considering studies for this review

3.1.1 Type of studies This is a systematic review of primary studies that use qualitative methods for data collection (i.e. focus groups, interviews, observation) and for data analysis (i.e. thematic analysis, framework analysis, grounded theory). Two types of qualitative studies were included: qualitative trial sibling studies and unrelated qualitative studies. A qualitative trial sibling study is a study that is directly related to an intervention evaluated in a trial. An unrelated qualitative study is a study that has no association with a trial of the effectiveness of an intervention, but explores a comparable intervention in a similar context and with broadly comparable participants. This differentiation was made in order to being able to examine potential differences between the outcomes of the two types of qualitative studies. While this comparison is not a part of the current objective, it is important to understand the methodological choices that we made.

3.1.2 Type of participants We included studies that focus on the views and experiences of any stakeholder affected by the CDS. This includes patients, healthcare providers, quality improvers, health facility managers, public health officers and CDS professionals. Studies where the population is limited to the use of CDS by students only were excluded. We considered studies conducted in any healthcare setting. Papers where the CDS is evaluated in simulated settings were excluded.

8

3.1.3 Types of phenomena of interest We included studies that assess stakeholders’ views and experiences of CDS that is applied to support the implementation of recommendations. We considered CDS with any objective (e.g. diagnosis, treatment, test ordering, screening) for any health condition. We defined CDS as an information technology to aid clinicians and patients in making healthcare decisions, based on patient-specific data. The data can be produced both by healthcare professionals and by patients. CDS can be Internet-based, installed on a local personal computer or a networked electronic health record, or function on a handheld device and comes in many types and functions.

We included both computer generated decision support that is displayed on screen or provided on paper. We considered information generated through CDS that is directed at healthcare professionals or targeted at both professionals and patients. We excluded papers to assess people’s views and expectations before the CDS has been implemented. We excluded papers where compliance with the advice of CDS is mandatory. Papers about sending reminder messages for attendance at upcoming healthcare appointments were excluded, as well as those about medication prescribing modules or medication interaction systems.

3.2 Search methods for identification of studies 3.2.1 Trial sibling studies We started with a collection of six trial-sibling studies that were identified in a literature search in the GUIDES project (table 1)(12). We also contacted the authors of a meta-regression analysis on success features of CDS that identified 162 trials.(11) We asked the authors if any qualitative trial-sibling study had taken place. The appendix provides an example of a contact e-mail. After two weeks, a reminder email was sent. In the case that no response was obtained after the first reminder, we verified the main text and the reference list of the trial report to identify any related qualitative study. We also identified the trial in Google Scholar and screened the studies that cited these trials. Furthermore, during our search we added articles suggested to us by Mendeley.

3.2.2 Unrelated studies To identify the unrelated studies, we made use of the studies identified in three existing qualitative evidence syntheses, which were assessed to be meaningful during the GUIDES project. (18-20) We made a list of all the qualitative studies identified in these reviews. Furthermore, during our search we added articles suggested to us by Mendeley.

3.3 Data collection and analysis

3.3.1 Selection of studies Step 1: Assessment of abstracts and titles according to the inclusion criteria We collated all studies identified through the search strategy into one database. After removing duplicate studies, two independent reviewers screened the titles and abstracts for each record for its potential inclusion eligibility.

9

Step 2: Assessment full text according to the inclusion criteria Thereafter, we retrieved the full text of all of the abstracts and titles that have been assessed as potentially eligible. Two reviewers screened the full text for eligibility according to the predefined criteria.

Step 3: Sampling of studies to be included in the review Qualitative evidence synthesis does not require an exhaustive sample and large numbers of studies can impair the quality of the analysis.(21) We sampled the studies in such a way that they explore a comparable intervention in a similar context and with broadly comparable participants. We excluded studies with low richness of data. To assess the data richness, we used a 1 tot 5 scale with 1 corresponding to very few or thin qualitative data (for example, from an open-ended survey question); 3 being an average qualitative article in a peer-reviewed health services journal; and 5 being very rich data (for example, from an ethnographic study). We excluded all articles that scored a 1 or 2 for data richness. 3.3.2 Appraisal of study quality Quality appraisal of individual studies was undertaken with the Critical Appraisal Skills Programme tool for qualitative studies, which we inserted within Google Forms.(22) Each study has been appraised by two reviewers. Unresolved disagreements were discussed with the promotor.

3.3.3 Data extraction and management Step 1: Selection of a framework to assess the perceptions and experiences We used the GUIDES framework to capture the views and experiences of the CDS stakeholders.

Step 2: Development of the data extraction form We tailored an existing GUIDES data extraction form to the purpose of this qualitative review. This form was created within Google Forms and exports the data to an excel spreadsheet. The form contained the factors included in the GUIDES framework combined with a description of the study characteristics were extracted: intervention characteristics, implementation, and funding, recipients of the intervention, country, focus of the qualitative research questions, qualitative methods used for data collection and data analysis, interval between intervention implementation and qualitative data collection.

Step 3: Data extraction (coding) from the full texts Two reviewers independently extracted data for each study. Participant quotations were not treated as data by themselves but served as illustrations of authors’ descriptions of a phenomenon.

3.4 Data synthesis

We conducted a framework analysis arranging the extracted data according to how it best fit in the existing GUIDES framework. This was done in pairs, after which all four authors scrutinized the arranged data. Data synthesis was a group effort of four authors, where each focused on a part of the arranged data. After structuring the data as such we coded it using a thematic analysis approach. Identified themes and corresponding data were double checked by the three others for possible links to their part of the data.

10

3.5 Appraisal of the confidence in the review findings

The confidence in the findings from the qualitative evidence synthesis was assessed with CERQual. 23,24). This approach is based on the following concepts:

• Methodological limitations of included studies: the extent to which there are problems in the design or conduct of the primary studies that contributed evidence to a review finding. We used the appraisal generated through the CASP questionnaire to identify potential limitations.

• Relevance of the included studies to the review question: the extent to which the body of evidence from the primary studies supporting a review finding is applicable to the context (perspective or population, phenomenon of interest, setting) specified in the review question.

• Coherence of the review finding: the extent to which the review finding is well grounded in data from the contributing primary studies and provides a convincing explanation for the patterns found in these data.

• Adequacy of the data contributing to a review finding: an overall determination of the degree of richness and quantity of data supporting a review finding.

This step resulted in an assessment of our confidence for each individual review finding. This confidence can either be rated as high, moderate, low or very low. We summarised in a table the review findings with their corresponding level of confidence, and an explanation for the CERQual rating. We appraised the confidence individually and organised a group meeting to discuss the individual judgements.

4. Results

4.1 Description of the studies

4.1.1 Results of the search We identified a total of 31 sibling and 188 unrelated studies. After screening titles, abstracts and full texts, we sampled 21 unrelated en 20 sibling studies according to the predefined criteria and included 12 unrelated and 9 sibling studies for our synthesis (figure 1). For a list of included but not sampled studies, see table 2. For a list of the sampled studies, see table 3.

11

4.1.2 Description of the included studies 4.1.2.1 Context, type of CDS and participants With the exception of one study, which took place in Kenya, all the included studies represent a Western point of view, predominantly U.S.A. and the U.K. We were able to include a broad scope of CDS stakeholders, with the majority of the included studies having been conducted in a primary care setting. As we collated all the data in one database, it was not possible to distinguish between the different experiences of different type of health workers. All studies were conducted during or after the implementation. The types of CDS commented vary. They range from preventive aid modules to therapeutic decision aids to managing chronic diseases. For a summary of the characteristics of the included studies, see table 4.

Searchstrategy:•  Literaturesearch(GUIDES-project)•  MetaRegressionanalysisRoshanov,etal.

•  Contactedtheauthorsofthetrialsincluded•  Searchedthereferencelistsofthetrialsincluded•  SearchedGoogleScholarforcitedreferences

•  IdenEfiedrelevantunrelatedstudiesfrom3QES(Moxeyetal.,Milleretal.,Alietal.)

•  SuggestedarEclesonline(Mendeley)

188unrelatedstudies+31trial-siblingstudies

21unrelatedstudies+18trial-siblingstudies

12unrelatedstudies+9trial-siblingstudies

Screening7tlesandabstracts

Fulltextscreening

Samplingbasedondatarichness

Figure 1 - Search strategy

12

4.1.2.2 Qualitative methods In the majority of the studies, semi-structured interviews were used to assess the views and experiences of the participants. Data were mostly analyzed using thematic analysis.

4.2 Findings

Domain 1: CDS context

1.1 CDS can achieve the defined quality objectives

Finding 1: If users think the CDS is helpful, and thus adding value to daily practice, they are more

willing to use it. (Low confidence)

Clinicians often stated that when decision support improved their day-to-day clinical practice in some way, they were more inclined to use the CDS.(23-25)

A strong theme was the helpfulness of the CDS to follow the guidelines. Some studies say that the CDS gave better access to the guidelines, and thus stimulated their use.(23,26) Clinicians in the study of Weber (25) said the CDS helped persuading patients to agree with the advice recommended in guidelines. The CDS was also said to reduce barriers in therapeutic decision-making, such as lack of familiarity with guidelines, inertia to previous practice and patient refusal.(24) Rousseau (27) says that if the system overemphasises aspects of guidelines that have been given low priority to by clinicians, it is perceived as not helpful and will therefore be used less. Some other studies suggest that CDS can be used to make clinicians more aware of their way of practice. Clinicians say that the CDS gives them insight in their working procedures (26,28) and gives a template for an ideal management scenario.(23) Some GP’s say the CDS can help to avoid routinely overlooking important aspects of the health encounter.(23)

Also, some authors state that the CDS can help detect risk information. (26) If the users feel that the CDS doesn’t give enough new information, the program will probably be used less.(28) Krall lists some criteria for a useful alert and says that participants complained vociferously about being told things they already knew.(30) In some articles, users said they don’t need decision support. An explanation given by some authors was that experts believed their opinion is more valuable than guidelines. (31,23) Another reason not to use CDS was that users think they already follow the guidelines. (24)

Participants of the study of Goud (26) said the CDS can be helpful to predict outcomes, like mortality and length of stay.

Finally, there were studies saying that CDS can give clinicians more confidence in their decision making by confirming decisions they’ve already made. (24,32)

13

1.2 The quality of the patient data is adequate

Finding 2: Users think that good quality of patient data is necessary for appropriate functioning of

the decision support. (moderate confidence)

The data show that clinicians value the use of correct data for good functioning of the CDS.(33,25)

Some authors found that clinicians don’t always trust the data entered by colleagues. (34,25) Data- entry clerks who make mistakes, clinicians still using paper documentation, wrongful diagnoses or failing to code relevant data were listed as reasons for poorly coded data.(35,36,27,37)

Users also stated the importance of the way the system uses the information at hand. Clinicians having the impression that the decision support system is not taking into account relevant coded information, and hence giving wrong guidance, can act as a barrier to use. (33,30, 34,23) Wrongly interpreting pending test results as negative or still giving reminders to run the test were explicitly listed as examples.(35,30)

“She’s [already] had her cholesterol done and then it says ‘cholesterol evaluation is recommended’ ... They’re telling us to suck eggs repeatedly, and I don’t like it.” (Peiris, 2011)

1.3 Stakeholders and users accept CDS

Finding 3. Clinicians are concerned about the influence of the CDS on the patient-doctor relationship and the communication with the patients. (moderate confidence)

A recurrent theme is the influence of the CDS on the patient-doctor relationship and the communication with the patients. Doctors are afraid that the CDS would adversely affect the face time with the patient (31), that the use of a computer would interrupt the communication with the patient (38-40), or that patients would express negative reactions about the use of a computerized aid (38).

“It just takes a lot of time and makes you focus too much on your computer and the patient just does not like that. I can see the patient thinking... while I’m only staring at that stupid screen”. (Lugtenberg, 2015)

On the other hand, some think the CDS can facilitate the communication with the patient and bring up discussions about health-related topics (43).

“Being able to show patients their risk and how to minimise it was considered beneficial: Clear and concise when risk factors come up on the screen. Getting the risk factor to show patients and help them understand they can make a difference by modifying behaviours.” (Wilson, 2007)

14

Finding 4. Some users feel the CDS negatively impacts the autonomy of the clinician. (moderate confidence)

Several articles describe how users say they feel watched and controlled by using the CDS, although other participants don’t mind the presence of a third party. (23,34,28,30)

Another issue was the perceived autonomy of the caregiver. CDS was reported by several authors to have a negative influence on the perceived autonomy of the users (28,35).

"If the patient needs sleep I can decide whether to order a drug or not." (Agostini, 2007)

Finding 5. Limited skills and confidence in information technology can be a barrier in the use of a

CDS. (moderate confidence)

Since the CDS is computer based, a certain level of information technology skills is obligatory. Several authors found that using the computer in general might be a barrier to the use (29,28,32,38). Some clinicians state they have trouble learning to work with the CDS, being a different application than the one they are used to working with. (32)

A general distrusting of computers was reported by some authors. (38,27)

‘‘It doesn’t matter how useful it might be, because it has a computer attached to it makes it unusable. That applies to computerized systems full-stop.’’ (Short, 2003)

Finding 6. If users feel like they don’t need decision support, the CDS may be used less. (moderate

confidence)

In some articles, users said they don’t need decision support. An explanation given by authors was that experts believed their opinion to be more valuable than guidelines. (31,23) Another reason not to use CDS was that users think they already follow the guidelines. (24,27) One author describes how users (falsely) think that the CDS makes mistakes because the users are unaware of the guidelines used by the CDS. (35)

"I mean I don't find or look at them...because I'm usually relatively comfortable with my respiratory management shall we say, I do very few respiratory referrals etc." (McDermott, 2014)

1.4 CDS can be added to the existing workload, workflows and systems

Finding 7: Decision support users indicated that the time and effort required from them are important factors in the successful adoption of a CDS. (High confidence)

Several studies stated that time is an important resource for decision support users. (39,34,38,23) Participants indicated that when decision support is efficient (30) and time saving (29,27), it is more likely to be well received. The opposite is true for programs that require substantial time and effort investments. Many studies cited time and effort demands as an important barrier to the use of the decision support system. (29, 37, 39, 34, 38, 40, 24, 43, 31, 27)

15

“But in the end, it needs to be a tool that is easy to use. And you can quickly use it to help you make certain decisions in the limited amount of time you have.” (Lomotan, 2011)

Finding 8: Users stated that whether decision support fits in the existing workflow is important. (moderate confidence)

Workflow, the way that clinical work is organized, is an important aspect in the implementation of a decision support system. (36) Decision support that does not fit in the established workflow is unlikely to be adopted by targeted users (37, 31, 40), unless workflow is changed to accommodate the decision support system (39).

“[I take notes] on paper. The [Interval History forms] that we were using before the electronic system came about. We still have the paper forms there because the nurses record vital signs on those paper forms.” (Lomotan, 2010)

Domain 2: CDS content

2.1 The content provides trustworthy evidence-based information

Finding 9: User distrust of the underlying evidence can be a barrier to CDS use. (moderate

confidence)

User disagreement with or distrust of the content of the underlying guideline is a barrier to decision support use. For example, when users perceive the guidance as being too stringent or the prescribed assessment procedures too shallow. (26)

Users suspect pharmaceutical industry involvement can lead to misuse of confidential information (39,36) or the perceived pushing of certain brands and quantities of drugs (27), leading them to users to distrust the underlying guideline and therefore the decision support program.

“Well, then it makes me wonder: do they own any stock options? Yeah, I know it sounds a bit silly. But it makes me wonder which pharmaceutical company is backing this?” (Lugtenberg, 2015)

Finding 10: Users are generally more inclined to trust the decision support system when recommendations are evidence based. (low confidence)

One study reported that nurses trusted the decision support system because the system’s underlying protocol was evidence based (34). Another study reported that patients were more willing to accept guideline recommendations than professional opinion. (26) One study found that when expert opinion plays a major role in the development of recommendations, physicians feel justified to use their own expertise and not the decision support system. (31)

“[The guideline] is based on expert opinion, and that’s very clearly stated. So, I think that, keeping that in mind, we have expertise, too, so I think that our expert opinion counts as well.” (Lomotan, 2011)

One study however found that physicians feel evidence needs to be validated by expert opinion before incorporating new evidence into practice. (23) Another study found that perceived

16

shortcomings of evidence based medicine in relation to new treatments or concerns regarding the currentness of the evidence were a barrier to implementation of the decision support. (27)

“One of the problems that I have with evidence based medicine is that sometimes you go down certain lines because the evidence is best for certain things, but it may be that the evidence is only best for certain things because they are older and they’ve been around longer and the evidence is more robust, but because they have been around longer they may well not be the best.” (Rousseau, 2003)

2.2 The decision support is relevant and accurate

Finding 11: Users stressed the importance of relevant and accurate reminders. (moderate

confidence)

Multiple studies mentioned the importance of having accurate and relevant guidance. (28, 36, 33) When the guideline content is not applicable in a certain setting or for a specific patient (36, 31, 40, 27) or when the guidance itself is too simplistic, incomplete or incorrect the content can function as a barrier. (28)

“And so should I get an IgE and a RAST test or maybe send you to Allergy [clinic] to get skin prick testing done, and see if you qualify for immune therapy or [omalizumab] therapy? So those are the kinds of tools that specialists would need, which is not something that pediatricians would need. Because which pediatrician is gonna start thinking about [omalizumab] for an asthmatic in their office? They’re not gonna do that. It’s actually not even their job to do that.” (Lomotan, 2011)

Finding 12: When guidance is individualised to the user, it might be better received. (low confidence)

A few clinicians stated that a lack of flexibility in terms of being able to adapt the content of the decision support to a specific user was perceived as a barrier. (39) This could be improved by filtering alerts according to criteria such as specialty, degree and past performance or by allowing users to set preferences about how and when they prefer to receive the decision support guidance. (30)

(About the lack of adjustability to personal preferences) “The customization options are still rather limited. You should be able to turn off specific types of advices, for instance the ‘give up-smoking-alerts’ rather than all life style advices at ones” (Lugtenberg, 2015)

Finding 13: Users expressed concern about receiving decision support advice for matters unrelated to the reason for encounter. (moderate confidence)

Users stated that discrepancies between the patient’s reason for visiting and the content of the decision support was a reason not to use the program. (39) Sometimes a particular alert may be seen as inappropriate considering the presenting or emerging problems of the visit. (30, 27, 29)

“For example, patients coming for acute health problems, such as high fever, may not feel comfortable discussing the computer-reported psychosocial issues.” (Ahmad, 2010).

17

Finding 14: Repeatedly receiving the same guidance may lead to frustration among users. (very

low confidence)

Some healthcare workers complained about repetition and going through the same process multiple times. (41,30) On the other hand, some users seem to appreciate these same reminders and might choose to continue them. (30)

“To tell you the truth it was a bit boring in the end because all you were doing is going over the same thing time and time again. That was a bit monotonous, to tell you the truth.” (Weiss, 2009)

Finding 15: Language incompatibility can be a barrier to CDS use. (Low confidence)

In one study, users encountered difficulties when patients speak a different language than the one the decision support is presented in. Having the guidance available in additional languages could remedy this. (24)

"If it’s just in English it's not going to be useful specifically for us...um for our patient population Urdu or Mirpuri" (McDermott, 2014)

Finding 16: Some users feel decision support is more useful for inexperienced healthcare providers. (Low confidence)

Certain users stated that decision support may be more useful for inexperienced healthcare providers. (43,24,42)

"New colleagues or new prescribers might be needing to look at it more." (McDermott, 2014)

2.3 The decision support provides an appropriate call to action

Finding 17: Users stated that if there is an adequate call to action, the CDS is more helpful. (moderate confidence)

The decision support needs to be clear on the action that is required and on the clinical importance and urgency of this action. Some clinicians suggested that they are more inclined to follow the recommended action if there is an adequate call to action. (28,26)

Other clinicians indicated that if the meaning of the call to action is unclear to them, the perceived value of the system will be lowered. Patients suggest that an adequate call to action helps them to seek medical assistance. (43)

2.4 The amount of decision support is manageable for the target user

Finding 18: Alerts triggering too often or for unimportant issues can be a barrier to CDS use. (Moderate confidence)

One study found that a high intensity of alerts was mentioned to be a barrier (39), another study found that alerts for unimportant acted as a barrier for users, because they diminish the impact of more serious alerts. (33)

18

Clinicians said that they would like to individualize the guidance to ensure that the number and quality of the alerts are acceptable. (30,27)

“... did you check kidney function, liver function...? At a certain point you’ll get overloaded with information that is actually quite straightforward.... 25 yellow [an alert] out of the last 50 patients....”. (Lugtenberg, 2015)

Finding 19: Users pointed out that they have concerns about the time it took to read an advice. (Moderate confidence)

Three studies reported that users were concerned about the time they need to read the decision support advice. (39,24,28)

“So it was sort of a nice idea but it's just that sort of real pressure on time, thinking you know, I can't go through all of this and it put me off" (McDermott, 2014)

Domain 3: CDS system

3.1 The system is easy to use

Finding 20: Users suggested that the system should be easy to use, with a minimum of training. (High confidence)

Five studies described that users liked the system when it was easy to use (37, 33, 24, 41, 43). One study described that when the system is not easy to use, and the doctors need time to explore the software outside of a consultation, they were generally reluctant to experiment with the system. (27)

Users pointed outed that they disliked that the system can’t interact with other computerized information systems and that they can’t do most of their tasks within the same application.

Three studies found that users would like the CDS to be linked to other computerized information systems. (39,34,43) One study showed that users appreciate the ability to link the system to other knowledge resources across the internet from within the application. (30)

One study showed that users would like the decision support to be connected to the patient’s medical record. (27)

“The alert screen should directly be linked to follow-up actions that need to be done! So, if you are to prescribe a statin, it should go directly to that screen. If you have to register blood pressure, you should be able to register it right there”. (Lugtenberg, 2015)

3.2 The decision support is well delivered

Finding 21: Users pointed out that the display of the decision support should be adequate. (Moderate confidence)

19

Five studies described that the display of the CDS can be a source of frustration when using the system. Out of those five, three studies described that users would like new alerts to be highlighted. (33, 24, 30)

Users in one study stated that the size and placement of buttons and other controls should be designed to achieve speed and minimize error. (30) One other study described that the decision support information should be available in a layout that works for all sizes of computer screens. (43)

"I think they just didn't attract your attention away from what you were doing to notice them, so if they were somehow made to stand out more, or moved across to a different part of the screen this would help me" (McDermott, 2014)

Finding 22: Users pointed out that the requirement to justify the (non-)compliance to the CDS can be a barrier to the use. (Moderate confidence)

Some participants disliked having to justify their action. One reason for this was the time it took to follow documentation procedures. Another reason reported was that there were too many situations where users are required to justify their reasons for an action. (33, 40)

‘‘... And previous work that other people have done on asking people to put in justifications as to why they do something have resulted in a great deal of recording of ASDFG [the typing of meaningless letters].’’ (Expert in computerized clinical decision support) (Avery, 2007)

Finding 23: Users reported the importance of finding a balance between avoiding unnecessary interruption and increasing awareness by intrusive CDS. (Low confidence)

Some clinicians reported that intrusive guidance, such as alerts, must be used very selectively. Intrusive guidance restricts or interrupts clinical action in some way, as opposed to passive guidance, which does not. They suggested that when timely attention to a CDS is crucial an intrusive alert could increase awareness and possibly compliance, as compared to a passive alert.

There is a delicate balance to be found between the desire to avoid unnecessary interruption and the requirement that important alerts must be paid attention to. (30)

“And don't make things red unless you need to attend to them right now."(Krall, 2002)

3.3 The system delivers the decision support to the right target person

Finding 24: Physicians pointed out that the alerts could be presented to someone else in the clinical workflow. (Moderate confidence)

Two studies described that doctors would like that not only doctors, but also nurses or other health care providers should be able to make use of CDS. (30,27)

Two studies suggested that doctors would like to delegate certain tasks of the CDS to a different staff member. (29,37)

“Well, I think my own personal view is that nurses are very good at working to protocols and pathways of care and all the rest of it and they’re comfortable with going from A to B, whereas . . .

20

our skill is perhaps in kind of thinking in a roundabout way and jumping through a few of those pathways through whatever it is, experience or whatever, so you don’t ask people the 10 or 15 questions to get from question 1 to question 14 you know, you go straight from one to 14 by intuition almost” (General practitioner, interview study) (Rousseau, 2003)

3.4 The decision support is available at the right time

Finding 25: Users claimed that the alerts from the decision support must come at the right time. (Moderate confidence)

Users tend to dislike the CDS when guidance triggers too early.(27,30) When guidance triggers too early, or before seeing the patient, users feel this disrupts thought processes. Some users would like to be able to postpone certain alerts to remedy this.(30)

On the other hand, users also dislike guidance that comes too late in the decision making process. Some doctors, for example, described that the system was triggered by the entry of a morbidity code at the end of the consultation, and that this was too late to be used.(27) Other physicians, for example, used the system after the patient had left, rather than during the consultation, too late for the CDS to make an impact.(42,30)

“They’re off [patient has left], I turn back and go back into this and then select the problem title, and then I say right well I’ve looked at ischaemic heart disease and then this comes up—and there [system has activated] and the patient’s already gone by this stage” (General practitioner, interview study) (Rousseau, 2003)

Domain 4: CDS implementation

4.1 Information to users about the CDS system and its functions is appropriate

Finding 26: Users generally perceive a lack of knowledge about the CDS’s purpose and functioning as a barrier to use the system. (High confidence)

Several papers claim that awareness about the CDS’s purpose seems to be necessary for the clinicians’ willingness to use the system and to even notice the CDS. (29,24,25)

A lot of participants argue that a lack of knowledge regarding the functioning of the CDS is a barrier. (39, 40, 25, 30) The users express a need for more training and up-front education to improve their knowledge of system use. However, there is no clear consensus among the authors of the studies about how this training should be organised (for example: bedside training, workshops, training by colleagues etc.). (37, 39, 33, 25, 40, 41, 32, 27, 36)

In general, clinicians say that it is important to be involved in the implementation process and follow-up of the CDS, in order to understand the system and its purpose, but also to improve the use of the CDS. (32, 24)

“The APACHE III is like a sledge hammer waiting to fall on a patient’s head. We weren’t involved with the decision to use it, and we have physicians who think they know how to use it and don’t. This could

21

be dangerous to patients and family members. They trained the nurses but not the physicians. And if anyone is going to train me around its use, it better be an experienced physician who has used the system. I don’t understand why we needed it, what it can do for us, so I have no intention to use it. I want to be clear: If resources were dedicated to training physicians, I’d participate and maybe use it. I’m open to the concept of having additional objective information to use when making difficult decisions in the ICU, but if you don’t invest in training and demonstrate how it will benefit my practice and patients, why should I use it?” (Weber, 2009)

4.2 Other barriers and facilitators to compliance with the decision support advice are assessed/addressed

Finding 27: Clinicians consider organizational constraints to be a barrier for the implementation.

(moderate confidence)

Clinicians report in one paper that they do not follow the guidelines’ decision-making rules due to a lack of facilities and resources. (26) They mention for example the lack of capacity to perform some therapeutic sessions. Other participants argue that a lack of management priority is a barrier. (26,29)

“It [exercise therapy] is currently full due to a lack of accommodation. The physiotherapist says he just wants five patients in his group, because otherwise the hall is too small for sports activities.” (Goud, 2010)

Finding 28: Participants report financial aspects as an incentive to use the CDS. (Moderate

confidence)

Clinicians mention that attempting change could be unrewarding, but that financial incentives could encourage them. (27) Participants in other papers report that a lack of reimbursement is a persistent barrier to implement CDS.(29,26)

“You can have postgraduate education until the cows come home, it doesn’t change attitudes. The only thing that I know that works is actually setting a target system, with financial carrots or financial sticks” (General practitioner, interview study) (Rousseau, 2003)

4.3 Implementation is stepwise and the improvements in the CDS system are continuous

We didn’t find any data in our included studies to support this section of the GUIDES tool.

4.4 Governance of the CDS implementation is appropriate

Finding 29: Finding: Participants want the computers to be in the exam room and to be up and

running. (Moderate confidence)

Users say that computers in de exam room may improve the likelihood that the CDS will appear at a helpful time. (30) Other users mention the importance of a fast and non-distracting access during clinic. (31,27)

“The kids play with the computers, and they sound like they’re taking off [noise of intermittent cooling fan]. You know, like some rocket explosion or something like that. Even when you’re not using them. So, it’s not uncommon for them to not be working. (Fellow) “ (Lomotan, 2010)

22

Finding 30: Users find a standardized process to incorporate new content in the CDS a facilitator.

(Low confidence)

Participants consider integrating new clinical knowledge through a systematically environmental scanning whereby board members provide oversight to be a facilitator to the use of a CDS.(36)

Finding 31: Users find the lack of exchange of data between practices to be a barrier to the use of a CDS. (Low confidence)

Participants mention that the inability to see patient data from other practices, limits the use of a CDS because the CDS does not have sufficient information to fully judge a patient’s entire clinical state in this case. (36)

5. Discussion

This article offers an analysis of 21 qualitative studies reviewing the experiences of patients, healthcare providers, quality improvers, health facility managers, public health officers and CDS professionals. We used the GUIDES framework as a base for our findings, and we analyzed whether the data, collected by an intensive search of the qualitative literature, was compatible with the GUIDES framework.

5.1 Discussion of main findings

Three findings were scored as high confidence through our CERQual assessments: time and effort required being important factors to consider; the requirement that CDS systems be easy to use, with a minimum of training; information about the CDS and training need to be adequate. We feel these findings reflect some of the key points of this evidence synthesis and are important for CDS developers and healthcare policy makers to consider. Findings with moderate, low and even very low confidence could also be important to consider, but will not be discussed in detail here.

First, CDS users want their information fast and easy. Ease of use, integration in clinical workflow and time required to operate the CDS are important factors. User-friendliness is a very important aspect for CDS developers to consider. A program that requires substantial input of time and effort, that is disruptive to clinical workflow and difficult to navigate, is destined to fail.

In accordance with this, having to justify non-compliance to certain recommendations seems to be a barrier according to our results, however other (quantitative) evidence reviews have identified this as a success factor(11, 47). While it may improve the objective outcomes, as perhaps physicians comply with the recommendation to avoid the extra work of having to justify their actions, this may lead to frustration and the registration of a great deal of meaningless letters instead of actual reasons. From a utilitarian point of view this may then be seen as an effective implementation strategy, but perhaps a rather cynical one.

23

Other points of debate in the CDS literature include whether or not decision support should provide guidance automatically or on an on-demand basis (4, 11, 48) and whether guidance should be provided within the electronic health record (EHR) or in a separate program.(11) Data from this review do not allow us to draw conclusions on the automatic vs on-demand question, but do suggest that complete integration of the CDS with the EHR is a success factor. This fits the philosophy of creating a very user friendly CDS. Roshanov(11) concludes that this integration of CDS and EHR is strongly associated with failure and hypothesizes that integrating CDS within the EHR leads to alert fatigue(46) and therefore the ignoring of prompts. In our review users underscored the importance of using interruptive DS sparingly and warned against a high intensity of alerts. Alert fatigue is therefore certainly an important factor to consider, perhaps even more so for integrated CDS.

Second, CDS users need to be aware of the importance of the program and how to use it. We found that CDS must fulfil a perceived need and be experienced as helpful. When clinicians feel the CDS adds little value they are not inclined to use it. This can occur, for example, when they are unaware of their knowledge gaps, when they consider themselves already in-line with the guideline recommendations, or when they disagree with or distrust the underlying guidance. Furthermore users state that information about the system, both before and after implementation is important. These findings are in line with what others have found on this subject(11, 19) and underscore the importance of exploring the needs and perceptions of clinicians before starting development of a CDS, as well as providing adequate education about the system both before and after implementation.

5.2 Comparison to GUIDES framework

Our review used the structure of the GUIDES framework, with the differentiation between facilitators and barriers in context, content, system and implementation. Most of our findings had significant overlap with the framework. However, several findings were not, or only partially mentioned in the GUIDES framework. Here we list certain points that might deserve a more pronounced mention in the checklist.

We found that some clinicians still have a general distrust of computers and others clinicians don’t have the necessary skills to work with a new computer system. This is an important finding because CDS developers should be aware that, even if they take into account all the suggestions of the GUIDES framework, a part of the working clinicians will not use it, only because they are not able to work with computerized programs or because they have a general distrust of computers.

Also, clinicians are afraid of possible involvement of the pharmaceutical industry in the CDS, and the misuse of confidential information or the pushing of certain brands and quantities of drugs. (finding 9).

Another important factor is that clinicians do not always trust the data entered by their colleagues (finding 2). This stands in contrast with another finding that suggest that clinicians would like to delegate certain tasks to other people in the clinical workflow (finding 24).

24

Discrepancies between the patient’s reason for visiting and the content of the decision support were listed as barriers in using the program (finding 13).

In contrast to the GUIDES network, we found no qualitative data that confirmed that the implementation should be stepwise and the improvements in the CDS system should be continuous.

5.3 Confidence in the findings

Confidence in our findings ranges from very low to high based on the GRADE-CERQual approach(44). The main reason for downgrading the methodology component of the assessment was concerns about the recruitment strategy of the included studies. Other important reasons for downgrading methodology included concerns about data analysis, researcher reflexivity and research design. Relevance concerns were seldom present due a varied sample that closely fits the scope of our research question. When present, concerns were usually due to a limited geographical spread. We downgraded findings in terms of coherence when contradictory data was present or when underlying data did not clearly represent the finding, as is the case in some of the more heavily interpreted points. Concerns about adequacy were due to either limited data richness or quantity of the underlying studies.

5.4 Overall completeness and applicability of evidence

We identified relevant studies through various methods, but did not perform an exhaustive literature search. We included 21 studies in the final sample, based on data richness and overall applicability. This approach was chosen because qualitative evidence synthesis does not require an exhaustive sample. Moreover, a large number of studies can impair the quality of the analysis.(21)

The final sample included many different actors of the healthcare landscape. Predominantly physicians and nurses, but also physiotherapists, physician assistants, practice managers, supporting staff, etc. Studies were conducted in different settings: primary, secondary and tertiary care centers; from rural healthcare to intensive care. We feel this broad scope of stakeholders is one of the strengths of our study. We chose to exclude studies conducted exclusively with students or studies conducted in a simulated environment, as these do not fully represent day-to-day practice.

We also chose to exclude studies regarding the use of medication prescribing module type of CDS and focused on CDS designed for the implementation of guidelines, as this fits the objective of our study better. The subjects of the CDS in the included studies are varied: asthma, insulin therapy, preventive care, cardiac revalidation, etc.

Geographically, our sample is mostly limited to Europe, Northern America and Australia, with one study conducted in Kenia. This may be due to the way we conducted our search strategy or because of potentially poor uptake of CDS outside of these regions. We could not find any studies regarding the degree of CDS uptake around the world. The impact of this geographical spread on our findings is uncertain, however it could make our results less applicable to other regions.

25

5.5 Reflexivity within the author team

The four main authors of this review are GP trainees. We believe this background did not have a significant impact on how we collected or interpreted the data, but the relative inexperience of our team might have, as this is our first qualitative evidence synthesis. We were however supported by a team researchers with experience in qualitative research, helping us ensure methodological quality. Furthermore, we ensured every phase of data collection, as well as study and finding quality appraisal was done by two researchers independently to reduce bias.

Working closely together with the main authors of the GUIDES framework has undoubtedly influenced us in the interpretation of our findings and was the reason for choosing a framework synthesis approach. This method was chosen because the secondary objective of our study was to improve upon the existing framework. Admittedly there are downsides to using this method, such as the tendency of this approach to suppress interpretive creativity and a tendency to neglect evidence that presents a fundamental challenge to the existing model.(16)

5.6 Implications for future practice

Before implementing decision support to address a certain topic, target users’ ideas regarding the subject need to be explored. When target users reject the necessity of the intervention, or when they are unaware of its potential merits, the program has but a slim chance to succeed.

Potential concerns identified through initial exploration need to be addressed. Clinicians must be made aware of why the intervention is necessary.

Adequate training must be provided both before and after implementation to ensure uptake of the decision support.

Developers should design CDS to be as user friendly as possible: requiring minimal time investment, an intuitive design and seamless integration in clinical workflow.

5.7 Implications for future research

Future research could focus on validating concerns identified in this review and other studies through surveys and questionnaires or pilot projects with a CDS. Points of particular interest are whether or not CDS should be integrated within the EHR or be a stand-alone program, whether guidance should trigger automatically or on an on-demand basis and the extent to which intrusive guidance should be used.

On the subject of methodology, it could be interesting to explore whether there is a difference between the trial sibling studies and the unrelated studies used in this review. As this was not the main focus of our study, we did not address this question.

26

6. Conclusion We have conducted a qualitative evidence synthesis of the barriers and facilitators computerized decision support users experience. We have summarized the data into 31 findings with levels of confidence according to the GRADE-CERQual assessment.

CDS developers and policy makers can use these findings to guide future development and implementation of decision support. Our findings will be included in the GUIDES framework.

References

1. Wright A, Sittig DF. A Four-Phase Model of the Evolution of Clinical Decision Support Architecture. Int J Med Inform. 2008 October;77(10): 641-649

2. Haynes RB, Wilczynski NL, Team CDSSR. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision-maker-researcher partnership systematic review. Implement Sci. 2010;5:12. doi:10.1186/1748-5908-5-12.

3. Murphy EV, Clinical Decision Support: Effectiveness in Improving Quality Processes and Clinical Outcomes and Factors That May Influence Success. Yale journal of biology and medicine 87 (2014), pp 187-197.

4. Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, et al. Enabling Health Care Decisionmaking Through Clinical Decision Support and Knowledge Management. Evid Rep Technol Assess (Full Rep). 2012;(203):1-784.

5. Kucher N, Koo S, Quiroz R, Cooper JM, Pa- terno MD, Soukonnikov B, et al. Electronic Alerts to Prevent Venous Thromboembolism in Hospitalized Patients. N Engl J Med. 2005;352(10):969-77. �

6. Haut ER, Lau BD, Kraenzlin FS, Hobson DB, Kraus PS, Carolan HT, et al. Improved Prophylaxis and Decreased Rates of Preventable Harm With the Use of a Mandatory Computerized Clinical Decision Support Tool for Prophylaxis for Venous Throm- boembolism in Trauma. Arch Surg. 2012;47(10):901-7.

7. Burack RC, Gimotty PA, George R, Simon MS, Dews P, Moncrease A. The Effect of Pa- �tient and Physician Reminders on the Use of Screening Mammography in a Health Mainte- nance Organization. Cancer. 1994;78(8):1708- 21.

8. Fiks AG, Hunter KS, Localio AR, Grund- meier RW, Bryant-Stephens T, Luberti AA, et al. Impact of Electronic Health Record- Based Alerts on Influenza Vaccination in Children with Asthma. Pediatrics. 2009;124(1):159-69.

9. Moja L, Koren HK, Lytras T, Bertizzolo L, Brandt L, et al. Effectiveness of Computerized Decision Support Systems Linked to Electronic Health Records: A Systematic Review and Meta-Analysis. American Journal of Public Health. 2014;104(12)

10. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems : a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765.

11. Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM et al. Features of effective computerized clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. doi:10.1136/bmj.f657.

12. Van de Velde S, Roshanov P, Kortteisto T, Kunnamo I, Aertgeerts B, Vandvik PO et al. Tailoring implementation strategies for evidence-based recommendations using computerized clinical decision support systems: protocol for the development of the GUIDES tools. Implement Sci. 2016;11(1):29. doi:10.1186/s13012-016-0393-7.Shank G D (2006).

27

Qualitative Research: A Personal Skills Approach (2nd edition). New Jersey. Merrill Prentice Hall.

13. Shank G D (2006). Qualitative Research: A Personal Skills Approach (2nd edition). New Jersey. Merrill Prentice Hall.

14. Roen K, Arai L, Roberts H, Popay J. Extending systematic reviews to include evidence on implementation: Methodological work on a review of community-based initiatives to prevent injuries. Social Science & Medicine 2006;63:4(1060-1071)

15. Heyvaert M, Hannes K, & Onghena P. Conducting systematic and comprehensive literature reviews: The Mixed Methods Research Synthesis approach. (2015). In: Mixed method research series, N. V. Ivankova & V. Plano-Clark.

16. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Integrative approaches to qualitative and quantitative evidence. London, UK: Health Development Agency; 2004.

17. Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Systematic Reviews (2017); 6:61

18. Miller A, Moon B, Anders S, Walden R, Brown S, Montella D. Integrating computerized clinical decision support systems into clinical work: A meta-synthesis of qualitative research. Int J Med Inform. 2015. doi:10.1016/j.ijmedinf.2015.09.005.

19. Moxey A, Robertson J, Newby D, Hains I, Williamson M, Pearson S-A. Computerized clinical decision support for prescribing: provision does not guarantee uptake. Journal of the American Medical Informatics Association : JAMIA. 2010;17(1):25-33.

20. Ali MK, Shah S, Tandon N. Review of electronic decision-support tools for diabetes care: a viable option for low- and middle-income countries? Journal of diabetes science and technology. 2011;5(3):553-70.

21. Odendaal W, Goudge J, Griffiths F, Tomlinson M, Leon N, Daniels K. Healthcare workers' perceptions and experiences on using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2015;2015(11). doi:10.1002/14651858.CD011942.

22. Qualitative Appraisal Checklist for Qualitative Research. Critical Appraisal Skills Programme (CASP).2006.http://www.caspuk.net/wpcontent/uploads/2011/11/CASP˙Qualitative˙Appraisal˙Checklist˙14oct10.pdf.

23. Peiris, D., Usherwood, T., Weeramanthri, T., Cass, A., & Patel, A. (2011). New tools for an old trade: a socio-technical appraisal of how electronic decision support is used by primary care practitioners. Sociology of Health {&} Illness, 33(7), 1002–1018.

24. McDermott, L., Yardley, L., Little, P., van Staa, T., Dregan, A., McCann, G., … Gulliford, M. (2014). Process evaluation of a point-of-care cluster randomised trial using a computer-delivered intervention to reduce antibiotic prescribing in primary care. BMC Health Services Research, 14, 594.

25. Weber, S. (2007). A qualitative analysis of how advanced practice nurses use clinical decision support systems. Journal of the American Academy of Nurse Practitioners, 19(12), 652–667.

26. Goud, R., van Engen-Verheul, M., de Keizer, N. F., Bal, R., Hasman, A., Hellemans, I. M. I. M. I. M., … Edmondson, A. (2010). The effect of computerized decision support on barriers to guideline implementation: a qualitative study in outpatient cardiac rehabilitation. International Journal of Medical Informatics, 79(6), 430–437

27. Rousseau, N., McColl, E., Newton, J., Grimshaw, J., & Eccles, M. (2003). Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. BMJ, 326(7384), 314.

28. Agostini, J. V., Concato, J., & Inouye, S. K. (2008). Improving sedative-hypnotic prescribing in older hospitalized patients: Provider-perceived benefits and barriers of a computer-based reminder. Journal of General Internal Medicine, 23(1 SUPPL.), 32–36.

28

29. Ahmad, F., Skinner, H. A., Stewart, D. E., & Levinson, W. (2010). Perspectives of family physicians on computer-assisted health-risk assessments. Journal of Medical Internet Research, 12(2).

30. Krall, M. a, & Sittig, D. F. (2002). Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proceedings / AMIA ... Annual Symposium. AMIA Symposium, 400–404.

31. Lomotan, E. A., Hoeksema, L. J., Edmonds, D. E., Ramirez-Garnica, G., Shiffman, R. N., & Horwitz, L. I. (2012). Evaluating the use of a computerized clinical decision support system for asthma by pediatric pulmonologists. International Journal of Medical Informatics, 81(3), 157–165.

32. Weber, S., Crago, E. A., Sherwood, P. R., & Smith, T. (2009). Practitioner approaches to the integration of clinical decision support system technology in critical care. The Journal of Nursing Administration, 39(11), 465–9.

33. Avery, A. J., Savelyich, B. S. P., Sheikh, A., Morris, C. J., Bowler, I., & Teasdale, S. (2007). Improving general practice computer systems for patient safety: qualitative study of key stakeholders. Quality and Safety in Health Care, 16(1), 28–33.

34. Campion, T. R., Waitman, L. R., Lorenzi, N. M., May, A. K., & Gadd, C. S. (2011). Barriers and facilitators to the use of computer-based intensive insulin therapy. International Journal of Medical Informatics, 80(12), 863–871.

35. Noormohammad, S. F., Mamlin, B. W., Biondich, P. G., McKown, B., Kimaiyo, S. N., & Were, M. C. (2010). Changing course to make clinical decision support work in an HIV clinic in Kenya. International Journal of Medical Informatics, 79(3), 204–210.

36. Ash, J. S., Sittig, D. F., Wright, a., McMullen, C., Shapiro, M., Bunce, a., & Middleton, B. (2011). Clinical decision support in small community practice settings: a case study. Journal of the American Medical Informatics Association, 18(6), 879–882.

37. Fiks, A. G., DuRivage, N., Mayne, S. L., Finch, S., Ross, M. E., Giacomini, K., … Grundmeier, R. W. (2016). Adoption of a Portal for the Primary Care Management of Pediatric Asthma: A Mixed-Methods Implementation Study. Journal of Medical Internet Research, 18(6), e172.

38. Short, D., Frischer, M., & Bashford, J. (2004). Barriers to the adoption of computerised decision support systems in general practice consultations: A qualitative study of GPs’ perspectives. International Journal of Medical Informatics.

39. Lugtenberg, M., Weenink, J.-W., van der Weijden, T., Westert, G. P., & Kool, R. B. (2015). Implementation of multiple-domain covering computerized decision support systems in primary care: a focus group study on perceived barriers. BMC Medical Informatics and Decision Making, 15(1), 82.

40. Patterson, E. S., Nguyen, A. D., Halloran, J. P., & Asch, S. M. (2004). Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc, 11(1), 50–59.

41. Weiss, M. C., Montgomery, A. A., Fahey, T., & Peters, T. J. (2004). Decision analysis for newly diagnosed hypertensive patients: A qualitative investigation. Patient Education and Counseling, 53(2), 197–203. article.

42. Dowding, D., Mitchell, N., Randell, R., Foster, R., Lattimer, V., & Thompson, C. (2009). Nurses’ use of computerised clinical decision support systems: a case site analysis. Journal of Clinical Nursing, 18(8), 1159–67.

43. Wilson, A., Duszynski, A., Turnbull, D., & Beilby, J. (2007). Investigating patients’ and general practitioners’ views of computerised decision support software for the assessment and management of cardiovascular risk. Inform Prim Care, 15(1), 33–44.

44. Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M et al. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895. doi:10.1371/journal.pmed.1001895.

45. Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J et al. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and

29

child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013;10:CD010414. doi:10.1002/14651858.CD010414.pub2.

46. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce “alert fatigue” while still minimizing the risk of litigation. Health Aff (Millwood) 2011;30:2310-7.�

47. Scott GPT, Shah P, Wyatt JC, Makubate B, Cross FW. Making electronic prescribing alerts more effective: scenario-based experimental study in junior doctors. J Am Med Inform Assoc 2011;18:789-98.�

48. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano M, Devereaux P, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1223-38.�

30

Table 1 - Trial siblings identified in the GUIDES project

Study Title

Tai, S. (1999)1 Evaluation of general practice computer templates – Lessons from a pilot randomised controlled trial

Ackerman S. L. (2013)2 One size does not fit all: evaluating an intervention to reduce antibiotic prescribing for acute bronchitis

Lugtenberg M. (2015)3 Implementation of multiple-domain covering computerized decision support systems in primary care: a focus group study on perceived barriers

Goud R. (2010)4 The effect of computerized decision support on barriers to guideline implementation: A qualitative study in outpatient cardiac rehabilitation

McDermott L. (2014)5 Process evaluation of a point-of-care cluster randomised trial using a computer-delivered intervention to reduce antibiotic prescribing in primary care

Martens J.D. (2006)6 Design and Evaluation of a Computer Reminder System to Improve Prescribing Behaviour of GPs

1 Tai SS, Nazareth I, Donegan C, Haines A. Evaluation of general practice computer templates. Lessons from a pilot randomized controlled trial. Methods Inf Med. 1999 Sep;38(3):177-81 2 Ackerman SL, Gonzales R, Stahl MS, Metlay J. One size does not fit all: evaluating an intervention to reduce antibiotic prescribing for acute bronchitis. BMC Health Services Research 2013, 13:462 3 Lugtenberg, M., Weenink, J.-W., van der Weijden, T., Westert, G. P., & Kool, R. B. (2015). Implementation of multiple-domain covering computerized decision support systems in primary care: a focus group study on perceived barriers. BMC Medical Informatics and Decision Making, 15(1), 82. 4 Goud, R., van Engen-Verheul, M., de Keizer, N. F., Bal, R., Hasman, A., Hellemans, I. M. I. M. I. M., … Edmondson, A. (2010). The effect of computerized decision support on barriers to guideline implementation: a qualitative study in outpatient cardiac rehabilitation. International Journal of Medical Informatics, 79(6), 430–437 5 McDermott, L., Yardley, L., Little, P., van Staa, T., Dregan, A., McCann, G., … Gulliford, M. (2014). Process evaluation of a point-of-care cluster randomised trial using a computer-delivered intervention to reduce antibiotic prescribing in primary care. BMC Health Services Research, 14, 594. 6 Martens JD, Van der Aa A, Panis B, Van der Weijden T, Winkens RAG, Severens JL. Design and evaluation of a computer reminder system to improve prescribing behaviour of GPs. Stud Health Technol Inform. 2006;124:617– 23.

31

Table 2 - Included but not sampled studies

Study Title

Lindgren, H. (2008)1 Decision support system supporting clinical reasoning process - an evaluation study in dementia care.

Baysari, M. (2011)2 The influence of computerized decision support on prescribing during ward-rounds: are the decision-makers targeted?

Krall, M. A. (2001)3 Subjective assessment of usefulness and appropriate presentation mode of alerts and reminders in the outpatient setting.

Lin, N. D. (2006)4 Identifying barriers to hypertension guideline adherence using clinician feedback at the point of care.

Militello, L. (2004)5 Clinical Reminders: Why Don’t they use them?

Peleg, M. (2009)6 Using multi-perspective methodologies to study users’ interactions with the prototype front end of a guideline-based decision support system for diabetic foot care.

Ralston, J. D. (2004)7 Patients’ experience with a diabetes support programme based on an interactive electronic medical record: Qualitative study.

Saleem, J. J. (2009)8 Provider perceptions of colorectal cancer screening clinical decision support at three benchmark institutions.

Trivedi, M. H. (2009)9 Barriers to implementation of a computerized decision support system for depression: an observational report on lessons learned in “real world”; clinical settings.

Ackerman, S. L. (2013)10 One size does not fit all: evaluating an intervention to reduce antibiotic prescribing for acute bronchitis.

Jenssen, B. P. (2016)11 Clinical Decision Support Tool for Parental Tobacco Treatment in Primary Care.

Martens, J. D. (2006)12 Design and evaluation of a computer reminder system to improve prescribing behaviour of GPs.

De Clercq, P. A. (2003)13 A consumer health record for supporting the patient-centered management of chronic diseases.

Mayne, S. (2012)14 The implementation and acceptability of an HPV vaccination decision support system directed at both clinicians and families.

Tai, S. S. (1999)15 Evaluation of general practice computer templates. Lessons from a pilot randomised controlled trial.

Tierney, W. M. D. (2005)16 Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial.

Henderson E.J. (2013)17 The utility of an online diagnostic decision support system (Isabel) in general practice: a process evaluation

32

1 Lindgren H Decision support system supporting clinical reasoning process – an evaluation study in dementia care. Stud Health Technol Inform. (2008);136:315-20. 2 Baysari, M. T., Westbrook, J. I., Richardson, K. L., Day, R. O., Wright, A., Phansalkar, S., … Vulto, A. (2011). The influence of computerized decision support on prescribing during ward-rounds: are the decision-makers targeted? Journal of the American Medical Informatics Association : JAMIA, 18(6), 754–759. 3 Krall, M. a, & Sittig, D. F. (2002). Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proceedings / AMIA ... Annual Symposium. AMIA Symposium, 400–404. 4 Lin ND, Martins SB, Chan AS, Coleman RW, Bosworth HB, et al. Identifying barriers to hypertension guideline adherence using clinician feedback at the point of care. AMIA Annu Symp Proc(2008):494-8 5 Militello L, Patterson ES, Tripp-Reimer T, et al., Clinical reminders: why don’t they use them, in: Human Factors and Ergonomics Society Annual Meeting Proceedings, vol. 48, 2004, pp. 1651–1655. 6 Peleg M, SHachak A, Wang D, Karnieli E. Using multi-perspective methodologies to study users' interactions with the prototype front end of a guideline-based decision support system for diabetic foot care. Int J Med Inform 2009 Jul.78(7):482-93 7 Ralston JD, Revere D, Robins LS, Goldberg HI. Patients’ experiences with a diabetes support programme based on an interactive electronic medical record: qualitative study. BMJ (2004) May 15;326(7449):1159. 8 Saleem JJ, Militello LG, Arbuckle N, Flanagan M, Haggstrom DA, et al. Provider perceptions of colorectal cancer screening clinical decision support at three benchmark institutions. AMIA Annu Symp Proc. 2009 Nov 14;2009:558-62 9 Trivedi MH, Daly EJ, Kern JK, Grannemann BD? Sunderajan P, Claassen CA. Barriers to implementation of a computerized decision support system for depression: an observational report on lessons learned in “real world” clinical settings. BMC Med Inform Decis Mak. 2009 Jan 21;9:6 10 Ackerman SL, Gonzales R, Stahl MS, Metlay J. One size does not fit all: evaluating an intervention to reduce antibiotic prescribing for acute bronchitis. BMC Health Services Research 2013, 13:462 11 Jenssen, B. P., Bryant-Stephens, T., Leone, F. T., Grundmeier, R. W., & Fiks, A. G. (2016). Clinical Decision Support Tool for Parental Tobacco Treatment in Primary Care. Pediatrics, 137(5). 12 Martens JD, Van der Aa A, Panis B, Van der Weijden T, Winkens RAG, Severens JL. Design and evaluation of a computer reminder system to improve prescribing behaviour of GPs. Stud Health Technol Inform. 2006;124:617-23. 13 De Clercq, P. A., Hasman, A., & Wolffenbuttel, B. H. R. (2003). A consumer health record for supporting the patient-centered management of chronic diseases. Medical Informatics and the Internet in Medicine, 28(2), 117–127. 14 Mayne, S., Karavite, D., Grundmeier, R. W., Localio, R., Feemster, K., DeBartolo, E., … Fiks, A. G. (2012). The implementation and acceptability of an HPV vaccination decision support system directed at both clinicians and families. AMIA ... Annual Symposium Proceedings / AMIA Symposium. AMIA Symposium, 2012, 616–624. 15 Tai SS, Nazareth I, Donegan C, Haines A. Evaluation of general practice computer templates. Lessons from a pilot randomized controlled trial. Methods Inf Med. 1999 Sep;38(3):177-81 16 Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, et al. Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res 2005;40:477-97. 17 Henderson, E. J., & Rubin, G. P. (2013). The utility of an online diagnostic decision support system (Isabel) in general practice: a process evaluation. JRSM Short Reports, 4(5), 31.

33

Table 3 - Sampled studies

Study Title

Ahmad (2010)1 Perspectives of Family Physicians on Computer-assisted Health-risk Assessments

Fiks (2016)2 Adoption of a Portal for the Primary Care Management of Pediatric Asthma: A Mixed-Methods Implementation Study

Lugtenberg (2015)3 Implementation of multiple-domain covering computerized decision support systems in primary care: a focus group study on perceived barriers

Agostini (2008)4 Improving Sedative-Hypnotic Prescribing in Older Hospitalized Patients: Provider-Perceived Benefits and Barriers of a Computer-Based Reminder

Goud (2010)5 The effect of computerized decision support on barriers to guideline implementation: A qualitative study in outpatient cardiac rehabilitation

Mc Dermott (2014)6 Process evaluation of a point-of-care cluster randomized trial using a computer-delivered intervention to reduce antibiotic prescribing in primary care

Rousseau (2003)7 Practice based, longitudinal, qualitative interview study of computerized evidence based guidelines in primary care

Weiss (2004)8 Decision analysis for newly diagnosed hypertensive patients: a qualitative investigation

Patterson (2004)9 Human Factors Barriers to the Effective Use of Ten HIV Clinical Reminders

Ash (2011) 10 Clinical decision support in small community practice settings: a case study

Avery (2007)11 Improving general practice computer systems for patient safety: qualitative study of key stakeholders

Campion (2011)12 Barriers and facilitators to the use of computer-based intensive insulin therapy

Dowding (2009)13 Nurses’ use of computerized clinical decision support systems: a case site analysis

Krall (2002)14 Clinicians’ Assessment of Outpatient Electronic Medical Record Alert and Reminder Usability and Usefulness Requirements

Lomotan (2012)15 Evaluating the use of a computerized clinical decision support system for asthma by pediatric pulmonologists

Noormohammad (2010)16 Changing course to make clinical decision support work in an HIV clinic in Kenya

Peiris D (2011)17

New tools for an old trade: a socio-technical appraisal of how electronic decision support is used by primary care practitioners

Short (2004)18 Barriers to the adoption of computerized decision support systems in general practice consultations: a qualitative study of GPs’ perspectives

Weber (2007)19 A qualitative analysis of how advanced practice nurses use clinical decision support systems

Weber (2009)20 Practitioner Approaches to the Integration of Clinical Decision Support System Technology in Critical Care

Wilson (2007)21 Investigating patients’ and general practitioners’ views of computerized decision support software for the assessment and management of cardiovascular risk

1 Ahmad, F., Skinner, H. A., Stewart, D. E., & Levinson, W. (2010). Perspectives of family physicians on computer-assisted health-risk assessments. Journal of Medical Internet Research, 12(2). 2 Fiks, A. G., DuRivage, N., Mayne, S. L., Finch, S., Ross, M. E., Giacomini, K., … Grundmeier, R. W. (2016). Adoption of a Portal for the Primary Care Management of Pediatric Asthma: A Mixed-Methods Implementation Study. Journal of Medical Internet Research, 18(6), e172.

34

3 Lugtenberg, M., Weenink, J.-W., van der Weijden, T., Westert, G. P., & Kool, R. B. (2015). Implementation of multiple-domain covering computerized decision support systems in primary care: a focus group study on perceived barriers. BMC Medical Informatics and Decision Making, 15(1), 82. 4 Agostini, J. V., Concato, J., & Inouye, S. K. (2008). Improving sedative-hypnotic prescribing in older hospitalized patients: Provider-perceived benefits and barriers of a computer-based reminder. Journal of General Internal Medicine, 23(1 SUPPL.), 32–36. 5 Goud, R., van Engen-Verheul, M., de Keizer, N. F., Bal, R., Hasman, A., Hellemans, I. M. I. M. I. M., … Edmondson, A. (2010). The effect of computerized decision support on barriers to guideline implementation: a qualitative study in outpatient cardiac rehabilitation. International Journal of Medical Informatics, 79(6), 430–437 6 McDermott, L., Yardley, L., Little, P., van Staa, T., Dregan, A., McCann, G., … Gulliford, M. (2014). Process evaluation of a point-of-care cluster randomised trial using a computer-delivered intervention to reduce antibiotic prescribing in primary care. BMC Health Services Research, 14, 594. 7 Rousseau, N., McColl, E., Newton, J., Grimshaw, J., & Eccles, M. (2003). Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. BMJ, 326(7384), 314. 8 Weiss, M. C., Montgomery, A. A., Fahey, T., & Peters, T. J. (2004). Decision analysis for newly diagnosed hypertensive patients: A qualitative investigation. Patient Education and Counseling, 53(2), 197–203. 9 Patterson, E. S., Nguyen, A. D., Halloran, J. P., & Asch, S. M. (2004). Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc, 11(1), 50–59. 10 Ash, J. S., Sittig, D. F., Wright, a., McMullen, C., Shapiro, M., Bunce, a., & Middleton, B. (2011). Clinical decision support in small community practice settings: a case study. Journal of the American Medical Informatics Association, 18(6), 879–882. 11 Avery, A. J., Savelyich, B. S. P., Sheikh, A., Morris, C. J., Bowler, I., & Teasdale, S. (2007). Improving general practice computer systems for patient safety: qualitative study of key stakeholders. Quality and Safety in Health Care, 16(1), 28–33. 12 Campion, T. R., Waitman, L. R., Lorenzi, N. M., May, A. K., & Gadd, C. S. (2011). Barriers and facilitators to the use of computer-based intensive insulin therapy. International Journal of Medical Informatics, 80(12), 863–871. 13 Dowding, D., Mitchell, N., Randell, R., Foster, R., Lattimer, V., & Thompson, C. (2009). Nurses’ use of computerised clinical decision support systems: a case site analysis. Journal of Clinical Nursing, 18(8), 1159–67. 14 Krall, M. a, & Sittig, D. F. (2002). Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proceedings / AMIA ... Annual Symposium. AMIA Symposium, 400–404. 15 Lomotan, E. A., Hoeksema, L. J., Edmonds, D. E., Ramirez-Garnica, G., Shiffman, R. N., & Horwitz, L. I. (2012). Evaluating the use of a computerized clinical decision support system for asthma by pediatric pulmonologists. International Journal of Medical Informatics, 81(3), 157–165. 16 Noormohammad, S. F., Mamlin, B. W., Biondich, P. G., McKown, B., Kimaiyo, S. N., & Were, M. C. (2010). Changing course to make clinical decision support work in an HIV clinic in Kenya. International Journal of Medical Informatics, 79(3), 204–210. 17 Peiris, D., Usherwood, T., Weeramanthri, T., Cass, A., & Patel, A. (2011). New tools for an old trade: a socio-technical appraisal of how electronic decision support is used by primary care practitioners. Sociology of Health {&} Illness, 33(7), 1002–1018. 18 Short, D., Frischer, M., & Bashford, J. (2004). Barriers to the adoption of computerised decision support systems in general practice consultations: A qualitative study of GPs’ perspectives. International Journal of Medical Informatics. 19 Weber, S. (2007). A qualitative analysis of how advanced practice nurses use clinical decision support systems. Journal of the American Academy of Nurse Practitioners, 19(12), 652–667. 20 Weber, S., Crago, E. A., Sherwood, P. R., & Smith, T. (2009). Practitioner approaches to the integration of clinical decision support system technology in critical care. The Journal of Nursing Administration, 39(11), 465–9. 21 Wilson, A., Duszynski, A., Turnbull, D., & Beilby, J. (2007). Investigating patients’ and general practitioners’ views of computerised decision support software for the assessment and management of cardiovascular risk. Inform Prim Care, 15(1), 33–44.

35

Table 4 - Characteristics of included studies

Clinical setting Location Focus Sibling/unrelat

ed study

Methods Before/during/af

ter implementation

Ahmad F. 10 general practitioners (7 females and three males) in an urban, multi-doctor, hospital-affiliated family practice clinic.

Canada Enhance understanding about computer-assisted health-risk assessments

Sibling Semi-structured interview guide with open-end questions analysed using analytic induction and constant comparison

After

Fiks A.G. 10 clinician focus groups and 22 semi structured parent interviews in a pediatric primary care setting

U.S.A. and Canada

Evaluate the feasibility of using a patient portal for pediatric asthma in primary care, its impact on management, and barriers and facilitators of implementation success

Sibling Semi-structured interviews and focus groups, interpreted in the context of the conceptual model.

Not clear

Lugtenberg M. 24 primary care practitioners (general practitioners, general practitioners in training and practice nurses)

The Netherlands

Identify the perceived barriers to using large-scale implemented CDSSs covering multiple disease areas in primary care

Sibling Focus group sessions using a semi-structured literature-based topic list, analyzed using thematic content analysis

After

Agostini J.V. 36 house staff physicians at a university hospital.

U.S.A. Identify provider-perceived benefits or barriers of a computer-based reminder regarding appropriate

Sibling Semi-structured interviews analyzed using grounded theory methodology

After

36

use of SH medications

Goud R. (2010) 29 professionals (21 rehabilitation nurses, 7 physiotherapists, and 1 rehabilitation doctor) from 21 outpatient clinics

The Netherlands

Assess the effect of CDSSs on cognitive, organizational, and environmental factors that hamper guideline implementation

Sibling In-depth, semi-structured interviews analyzed using the conceptual framework from Cabana et al.

After

McDermott L. 20 general practitioners and 4 members of staff

U.K. Process evaluation for a cluster randomized trial of a computer-delivered, point-of-care intervention to reduce antibiotic prescribing in primary care

Sibling Semi-structure telephone interviews analyzed using inductive thematic analysis

After

Rousseau N. 5 general practices in northeast England. 13 respondents (two practice managers, three nurses, and eight general practitioners)

England Understand the factors influencing the adoption of a computerized clinical decision support system for two chronic diseases in general practice

Sibling Semi-structured interviews analyzed using thematic analysis

Before and at different times during the intervention period

Weis M.C. 15 newly diagnosed hypertensive patients in a general practice

U.K. Explore patients’ views on the usefulness of a decision analytic decision aid

Sibling Semi-structured interviews, analyzed using the framework method of analysis

During

Patterson E.S. Physicians, pharmacists, nurses, and case managers from 8

U.S.A. Identify human factors barriers to the use of CR to improve adherence to

Sibling Semi-structured interviews analyzed using process

After

37

outpatient clinics

guidelines for human immunodeficiency virus

tracing analysis

Ash J.S. A wide variety of clinicians, including allopathic and osteopathic physicians, medical assistants, nurses, nurse practitioners, and physician assistants from independent clinics in semirural and rural settings

U.S.A. Identify barriers and facilitators to clinical decision support (CDS) implementation in a community setting

Unrelated Semi-structured interviews, analysed using grounded theory approach and open coding.

After

Avery A.J. 31 participants including clinicians, computer system and drug database suppliers, academics with interests in health informatics and members of governmental, and professional and patient representative bodies.

U.K. Identify ways in which the use of general practice computer systems could be improved to enhance safety in primary care.

Unrelated Semi-structured interviews, thematic synthesis until data saturation.

After

Campion T.R. Nurses and other clinicians at two intensive care units in the same hospital.

U.S.A. Illuminate barriers and facilitators to use of Computerized clinical decision support systems (CDSSs) for intensive insulin therapy (IIT).

Unrelated Observation of nurse workflow and conducting unstructured interviews with nurses and other clinicians analyzed using grounded theory.

After

38

Dowding D. Four different case sites, all within NHS Trusts in England, identified using a national survey. Case sites included nurses working in an anticoagulation team; nurses working in a spinal assessment clinic; nurses working in a Walk-in center; nurses working in a hospital based respiratory clinic.

U.K. Explore how nurses use CDSS in practice and some of the factors that appear to influence that use. (CDSS to support decisions for patient anticoagulation management; CDSS to support spinal assessment processes; CDSS to support patient assessment in a walk-in center; CDSS to support the home management of patients with COPD)

Unrelated Non-participant observation, brief interviews with nurses after CDSS use and in-depth semi- structured interviews with nurses, analysed using thematic content analysis.

After

Krall M.A. Three focus groups, with 4 to 7 primary care clinicians (internal medicine, family medicine, pediatrics physicians, physician assistants and nurse practitioners).

U.S.A. Barriers and facilitators in the use of EpicCare, an outpatient electronic medical record system.

Unrelated Focus group interviews with observation through one-way mirror analysed using open coding.

After

Lomotan E.A. Nine pediatric pulmonologists.

U.S.A. Use of the electronic health record, key obstacles to their use of the CDS system during patient care, and their

Unrelated Review of electronic data, direct observation, and interviews with all nine pediatric pulmonologis

After

39

thoughts on the usefulness of a guideline-based CDS system in a pediatric subspecialty setting.

ts in the clinic, analysed using grounded theory.

Noormohamm

ad S.F.

Clinicians in H.I.V. clinic in Kenya.

Kenya Real-world challenges to successful implementation of CDSS for CD4-reminders in a resource-poor setting.

Unrelated Multi-method qualitative design, including clinician feedback and interviews, workflow analysis and assessment of providers’ knowledge.

During

Peiris D. 21 GPs working in eight private, teaching general practices and three AMS.

Australia GP’s views on a novel CDS tool being developed for cardiovascular disease management.

Unrelated In-depth interview evaluation and a survey analysed using thematic content analysis.

During

Short D. 15 GPs from 9 practices.

U.K. Perceived barriers to using both integrated and non-integrated systems in primary care

Unrelated Interviews analysed using constant comparison and thematic synthesis.

During

Weber S. (2007)

23 advanced practice nurses from 16 critical care units in six large urban medical centers.

Midwest U.S.A.

Identifying relevant themes and patterns of CDSS use in critical care practices.

Unrelated Single-structured in-depth interviews with open-ended audiotaped questions, analysed using constant comparative method, synthesized using

After

40

grounded theory.

Weber S.

(2009)

33 intensive care clinicians (nurses and physicians).

U.S.A. Experiences to the use of CDSS to predict outcomes in critical care, reasons to use or not to use CDSS.

Unrelated In-depth structured interviews, analysed using constant comparative technique, synthesised using grounded theory.

After

Wilson A. 5 GPs and 9 patients

Australia Usability and acceptability of a CDSS for management of cardiovascular disease.

Unrelated Semi-structured questionnaire, thematically coded.

During

41

Cerqual evidence profile Domain 1: CDS context. 1.1 CDS can achieve the defined quality objectives

Finding 1: If users think the CDS is helpful, and thus adding value to daily practice, they are more willing to use it.

Assessment for each CERQual component

Methodological limitations Minor concerns The majority of the contributing studies are methodologically strong. Only one underlying study has significant methodological flaws.

Relevance No concerns All of the contributing studies are relevant to the research topic.

Coherence Major concerns A significant amount of data interpretation has led to this finding.

Adequacy Minor concerns The finding is supported by a sufficient amount of data from nine studies.

Overall CERQual assessment Low confidence In part due to minor concerns about

methodology and adequacy; however mostly due to a rather large amount of interpretation, leading to major coherence concerns.

1.2 The quality of the patient data is adequate

Finding 2: Users think that good quality of patient data is necessary for appropriate functioning of the decision support. Methodological limitations Minor concerns

There are a sufficient number of core studies supporting the finding that are methodologically sound. While some of the included studies have methodological flaws, we feel their contributions are less critical to the main finding and more indirectly support it.

Relevance No concerns All of the included studies are relevant to the research topic.

Coherence Moderate concerns While there are a number of directly supporting studies, a degree of interpretation has been used for other contributing studies.

Adequacy Minor concerns The finding is supported by a sufficient amount of data from nine studies.

42

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence because of moderate concerns regarding coherence, minor concerns regarding methodological limitations and adequacy.

1.3 Stakeholders and users accept CDS

Finding 3. Clinicians are concerned about the influence of the CDS on the patient-doctor relationship and the communication with the patients. Assessment for each CERQual component Methodological limitations Minor concerns

Most important data come from methodologically sound studies.

Relevance No concerns All of the included studies are relevant to the research topic.

Coherence Minor concerns There is one study (Wilson 2007) whose findings contrast with the other articles.

Adequacy Moderate concerns The finding is based on only five studies.

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate adequacy concerns and minor concerns regarding methodology and coherence.

Finding 4. Some users feel the CDS negatively impacts the autonomy of the clinician

Assessment for each CERQual component

Methodological limitations Moderate concerns Two of the contributing studies have insufficient methodological quality

Relevance Minor concerns No European studies. Good diversity in type of CDS and pathology.

Coherence Moderate concerns Contradictory data.

Adequacy Moderate concerns Only six contributing studies

Overall CERQual assessment

Moderate confidence

There is moderate confidence in this finding as not all the contributing studies support the finding and contradictory data are present. Moreover, some of the studies have insufficient methodologies.

43

Finding 5. Limited skills and confidence in information technology can be a barrier in the use of a CDS.

Finding 6. If users feel like they don’t need decision support, the CDS may be used less.

Assessment for each CERQual component Methodological limitations Minor concerns

One of the supporting articles is of low methodological quality. The others compensate sufficiently for this however.

Relevance No concerns All of the included studies are relevant to the research topic.

Coherence No concerns There are no contradictory data or ambiguity.

Adequacy Moderate concerns There are only five supporting articles

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate concerns about adequacy and minor concerns about methodology. No concerns regarding coherence and relevance.

1.4 CDS can be added to the existing workload, workflows and systems

Finding 7: Decision support users indicated that the time and effort required from them are important factors in the successful adoption of a CDS. Assessment for each CERQual component Methodological limitations Minor concerns.

The majority of the studies are methodologically strong. critical points in the finding are supported by enough methodologically sound studies such that the methodological flaws in the other studies don't impact the finding in a significant way.

Assessment for each CERQual component Methodological limitations Minor concerns

There are some methodologically flawed studies, but nothing that would significantly impact confidence in the finding.

Relevance No concerns All of the included studies are relevant to the research topic.

Coherence No concerns. There are no contradictory data or ambiguity.

Adequacy Moderate concerns There are only five supporting articles.

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate concerns about adequacy and minor concerns about methodology. No concerns regarding coherence and relevance.

44

Relevance No concerns All studies are relevant to the research question.

Coherence No concerns There are no contradictory data or ambiguity.

Adequacy No concerns The finding is supported by a substantial amount of rich data from 12 studies.

Overall CERQual assessment High confidence. This finding was graded as high confidence

due to minor methodological concerns. No concerns regarding coherence, relevance or adequacy

Finding 8: Users stated that whether decision support fits in the existing workflow is important Assessment for each CERQual component Methodological limitations Moderate

There are a significant number of studies with methodological flaws.

Relevance

No concerns All studies are relevant to the research question.

Coherence No concerns There are no contradictory data or ambiguity.

Adequacy Moderate concerns There are only five supporting studies.

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate concerns about methodology and moderate concerns about adequacy. No concerns regarding relevance or coherence.

Domain 2: CDS content 2.1 The content provides trustworthy evidence-based information Finding 9: User distrust of the underlying evidence can be a barrier to CDS use. Assessment for each CERQual component Methodological limitations Minor concerns

The majority of contributing studies are methodologically strong.

Relevance Minor concerns Minor relevance concerns. Only European studies contributing. The study population show multiple disciplines and therefore fits the research question.

Coherence Minor concerns Minor coherence concerns. Variation among the reasons why clinicians wouldn't trust the data used by the CDS.

Adequacy Moderate concerns Only four contributing studies.

45

Overall CERQual assessment Moderate confidence Moderate confidence in the finding: good

methodological value and relevance of the contributing studies, but only four studies supporting the finding.

Finding 10: Users are generally more inclined to trust the decision support system when recommendations are evidence based Assessment for each CERQual component Methodological limitations Minor concerns

Most underlying studies are methodologically strong.

Relevance No concerns Good geographical spreading, multidisciplinary population

Coherence Moderate concerns Contradictory findings regarding the weight of expert opinion versus guidelines in the development of the CDS.

Adequacy Major concerns Only a limited number of studies (four) contributing to the finding.

Overall CERQual assessment Low confidence This finding was graded as low confidence in

the finding due to a limited number of studies and the presence of contradictory data.

2.2 The decision support is relevant and accurate

Finding 11: Users stressed the importance of relevant and accurate reminders Assessment for each CERQual component

Methodological limitations Moderate concerns The studies contributing to this finding are of variable methodological quality.

Relevance No concerns All of the contributing studies are relevant to the research topic.

Coherence Minor concerns: relevance and accuracy both important for users

Adequacy Minor concerns There is a sufficient number of studies (six) supporting the finding.

Overall CERQual assessment Moderate confidence

This finding is based on a sufficient number of studies. However, not all of them are methodologically strong.

Finding 12: When guidance is individualised to the user, it might be better received. Assessment for each CERQual component Methodological limitations Moderate concerns

Moderate methodological concerns: one of the two contributing studies is of insufficient methodological quality

46

Relevance No concerns All studies are relevant to the research question.

Coherence No concerns There are no contradictory data or ambiguity.

Adequacy Major concerns Major adequacy issues: insufficient number of studies (only two) supporting the finding, weak data richness.

Overall CERQual assessment Low confidence Only two supporting studies, one of which is

of questionable methodological quality.

Finding 13: Users expressed concern about receiving decision support advice for matters unrelated to the reason for encounter. Assessment for each CERQual component Methodological limitations Minor methodological concerns

One of the supporting studies of poor methodological quality

Relevance No relevance concerns All studies are relevant to the research question.

Coherence No coherence concerns Coherent findings, no concerns regarding coherence

Adequacy Moderate adequacy concerns Moderate data richness due to only four contributing studies

Overall CERQual assessment Moderate confidence This finding is graded as moderate confidence

due to the limited number of contributing and the presence of one methodologically poor study.

Finding 14: Repeatedly receiving the same guidance may lead to frustration among users. Assessment for each CERQual component Methodological limitations Major concerns

One out of two studies had insufficient methodological quality

Relevance

No concerns All studies are relevant to the research question.

Coherence Moderate concerns Parts of the data don’t support the finding.

Adequacy Major concerns Poor data richness from only two studies, major concerns regarding adequacy.

Overall CERQual assessment Very low confidence This finding is graded as very low confidence

since it is only supported by two studies, from which one has insufficient methodologies. Moreover, the limited data are partly contradictory to the finding.

47

Finding 15: Language incompatibility can be a barrier to CDS use.

Assessment for each CERQual component Methodological limitations Minor concerns

The study supporting the finding is methodologically strong.

Relevance Minor issues Only one study with GP participants from the UK.

Coherence No concerns No contradictory data

Adequacy Major concerns Very low data richness since only one study supports the finding.

Overal CERQual assessment Low confidence This finding is graded as low confidence due

to very poor data richness. The only contributing study, however, is methodologically well executed.

Finding 16: Some users feel decision support is more useful for inexperienced healthcare providers Assessment for each CERQual component Methodological limitations Moderate concerns

Two out of three studies have methodological flaws.

Relevance No relevance issues All studies are relevant to the research question.

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Major concerns Only three contributing studies.

Overal CERQual assessment Low confidence This finding was graded as low confidence

due to major adequacy concerns and moderate methodological concerns.

2.3 The decision support provides an appropriate call to action

Finding 17: Users stated that if there is an adequate call to action, the CDS is more helpful. Assessment for each CERQual component Methodological limitations Moderate concerns

One out of three studies have methodological flaws.

Relevance Moderate concerns Because of a rather small study population, there are moderate concerns about relevance.

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Moderate concerns Only three contributing studies

48

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate adequacy concerns and moderate methodological en relevance concerns.

2.4 The amount of decision support is manageable for the target user

Finding 18: Alerts triggering too often or for unimportant issues can be a barrier to CDS use.

Assessment for each CERQual component Methodological limitations Minor concerns

The studies supporting the finding is methodologically strong.

Relevance No concerns All studies are relevant to the research question.

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Moderate concerns Only three contributing studies

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate adequacy concerns and moderate methodological concerns.

Finding 19: Users pointed out that they have concerns about the time it took to read an advice. Assessment for each CERQual component Methodological limitations Minor concerns

The studies supporting the finding is methodologically strong.

Relevance Minor concerns All studies are relevant to the research question.

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Moderate concerns Only three studies are supporting the finding.

Overall CERQual assessment Moderate confidence This finding was graded as moderate confidence

due to minor methodological, coherence and relevancy concerns but some concerns regarding adequacy considering rather thin data support the finding.

49

Domain 3: CDS system 3.1 The system is easy to use Finding 20: Users suggested that the system should be easy to use, with a minimum of training. Assessment for each CERQual component Methodological limitations Minor concerns

The studies supporting the finding is methodologically strong.

Relevance No concerns All studies are relevant to the research question

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Minor concerns There are a sufficient number of studies supporting the finding: thick data from nine studies.

Overall CERQual assessment High confidence This finding was graded as high confidence

due to minor methodological en adequacy concerns. No concerns regarding relevance and coherence.

3.2 The decision support is well delivered

Finding 21: Users pointed out that the display of the decision support should be adequate. Assessment for each CERQual component Methodological limitations Moderate concerns

One out of four studies had major methodological problems, another one had moderate methodological problems.

Relevance No concerns All studies are relevant to the research question

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Moderate concerns Insufficient number of studies supporting the finding, weak data richness from only four studies.

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate methodological and adequacy concerns. No concerns regarding relevance and coherence.

50

Finding 22: Users pointed out that the requirement to justify the (non-)compliance to the CDS can be a barrier to the use.

Finding 23: Users reported the importance of finding a balance between avoiding unnecessary interruption and increasing awareness by intrusive CDS.

Assessment for each CERQual component Methodological limitations Minor concerns

There a minor concerns regarding the methodology as the finding is based on two methodically strong studies according to the CASP criteria.

Relevance Moderate concerns Relevance is partial because it is possible that we only get the perspective from a subset of the population.

Coherence Minor concerns There are no contradictory or ambiguous data.

Adequacy Moderate concerns The finding is supported by only two studies, but there is a clear explanation of the reasons behind the review finding.

Overall CERQual assessment Moderate confidence Due to the concerns about relevance and

adequacy.

Assessment for each CERQual component Methodological limitations Major concerns

The finding is based on only one study with methodologically major concerns because of an unclear recruitment strategy and commercial funding.

Relevance Major concerns There are only sixteen participants in the study.

Coherence Minor concerns There are no contradictory or ambiguous data.

Adequacy Major concerns Because of the small study population and thin data from only one study, there are major concerns about the adequacy.

Overall CERQual assessment Low confidence Due to the major concerns regarding

methodology, relevance and adequacy.

51

3.3 The system delivers the decision support to the right target person

Finding 24 : Physicians pointed out that the alerts could be presented to someone else in the clinical workflow Assessment for each CERQual component Methodological limitations Moderate concerns

The finding is based on four studies, one study with methodologically major concerns, two studies with moderate concerns and one study with no methodological concerns.

Relevance No concerns All studies are relevant to the research question

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Moderate concerns insufficient number of studies supporting the finding, weak data richness from only four studies.

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate methodological and adequacy concerns. No concerns regarding relevance and coherence

3.4 The decision support is available at the right time

Finding 25: Users claimed that the alerts from the decision support must come at the right time Assessment for each CERQual component Methodological limitations Moderate concerns

One out of three studies have methodological flaws.

Relevance No concerns All studies are relevant to the research question

Coherence No concerns No contradictory data, no ambiguous data.

Adequacy Major concerns Thin data from only three studies.

Overall CERQual assessment Moderate confidence This finding was graded as moderate

confidence due to moderate methodological concerns. No concerns regarding relevance, coherence and adequacy.

52

Domain 4: CDS implementation 4.1 Information to users about the CDS system and its functions is appropriate Finding 26: Users generally perceive a lack of knowledge about the CDS’s purpose and functioning as a barrier to use the system. Assessment for each CERQual component Methodological limitations Minor concerns

There are no significant limitations concerning methodology of the studies supporting this finding.

Relevance No concerns All of the included studies are relevant to the research topic.

Coherence Minor concerns There are no contradictory or ambiguous data.

Adequacy Minor concerns A rather poor explanation is given about the reasons behind the preferences, but the finding is based on enough studies to trust the adequacy. Rich data from eleven studies.

Overall CERQual assessment High confidence Due to the absence of concerns about each

CERQual component. 4.2 Other barriers and facilitators to compliance with the decision support advice are assessed/addressed

Finding 27: Clinicians consider organizational constraints to be a barrier for the implementation.

Assessment for each CERQual component Methodological limitations Minor concerns

There are no significant limitations concerning methodology of the studies supporting this finding.

Relevance Moderate concerns Because of a rather small study population, there are moderate concerns about relevance.

Coherence Moderate concerns The underlying body of evidence may be more specific than the review finding. There is some transformation of data.

Adequacy Moderate concerns The finding is based on rather thin data from only two studies, but it is sufficient for supporting the finding so there are only moderate concerns regarding adequacy.

Overall CERQual assessment Moderate confidence Due to a rather small study population and

some transformation of data.

53

Finding 28: Participants report financial aspects as an incentive to use the CDS. Assessment for each CERQual component Methodological limitations Minor concerns

There are no significant limitations concerning methodology of the studies supporting this finding.

Relevance No concerns All of the included studies are relevant to the research topic.

Coherence Minor concerns There are no contradictory or ambiguous data.

Adequacy Moderate concerns The finding is supported by rich data, but only contributed by three studies.

Overall CERQual assessment Moderate confidence Due to a rather small study population. 4.3 Implementation is stepwise and the improvements in the CDS system are continuous

/ 4.4 Governance of the CDS implementation is appropriate

Finding 29: Finding: Participants want the computers to be in the exam room and to be up and running.

Assessment for each CERQual component Methodological limitations Moderate concerns

Two methodological strong studies and one weak one support the review finding. The lack of recruitment strategy and commercial funding in the Krall study however does not seem to influence the trust we can put in the review finding, so there are only moderate concerns regarding methodology.

Relevance Moderate concerns Relevance is partial as two out of three studies supporting the finding investigate CDS concerning asthma, which could influence the role of the CDS.

Coherence Minor concerns There are no contradictory or ambiguous data.

Adequacy Moderate concerns The finding is supported by rich data, but only contributed by three studies.

Overall CERQual assessment Moderate confidence Due to the moderate concerns about the

relevance and adequacy.

54

Finding 30: Users find a standardized process to incorporate new content in the CDS a facilitator. Assessment for each CERQual component Methodological limitations Moderate concerns

The finding is based on only one study with an unclear recruitment strategy and data analysis.

Relevance Moderate concerns Relevance is partial as only the perspective of a small subset of the population is included.

Coherence Minor concerns There is some transformation of data.

Adequacy Major concerns The finding is only supported by thin data from one study.

Overall CERQual assessment Low confidence Due to the thin data with methodologically

concerns supporting this finding. Finding 31: Users find the lack of exchange of data between practices to be a barrier to the use of a CDS. Assessment for each CERQual component Methodological limitations Moderate concerns

The finding is based on only one study with an unclear recruitment strategy and data analysis.

Relevance Moderate concerns Relevance is partial as only the perspective of a small subset of the population is included.

Coherence Minor concerns There is some transformation of data.

Adequacy Major concerns The finding is only supported by thin data from one study.

Overall CERQual assessment Low confidence

Due to the thin data with methodologically concerns supporting this finding.

55

Example letter:

Dear Dr Dexter,

I am writing on behalf of our collaborative team regarding the following published study:

Dexter PR, Wolinsky FD, Gramelspacher GP, Zhou XH, Eckert GJ, Waisburd M, et al. Effectiveness of

computer-generated reminders for increasing discussions about advance directives and completion of advance directive forms. A randomized, controlled trial. Ann Intern Med. 1998;128(2):102–110.

We are currently undertaking a synthesis of qualitative research studies on the views and experiences regarding the use of computerized clinical decision support systems. Examples of qualitative studies include interviews, focus groups or observations.

We would greatly appreciate if you could inform us if any qualitative research has been done during or after your trial to explore processes, contextual factors or intervention characteristics that may have influenced the trial results.

If you have any further questions regarding our requests, please feel free to contact me.

Thank you very much for considering this request.

Yours sincerely

Geachte Heer/Mevrouw De Opleidingspecifieke Ethische Begeleidingscommissie van de opleiding "Master in de huisartsgeneeskunde (Leuven e.a.)" heeft uw voorstel tot Masterproef "Perceptions and experiences on using computerised clinical decision support systems to implement recommendations: a qualitative evidence synthesis" onderzocht en gunstig geadviseerd. Dit betekent dat de commissie van oordeel is dat de studie, zoals beschreven in het protocol, wetenschappelijk relevant en ethisch verantwoord is. Dit gunstig advies van de commissie houdt niet in dat zij de verantwoordelijkheid voor de geplande studie op zich neemt. U blijft hiervoor zelf verantwoordelijk. Indien u van plan bent uw masterproef te publiceren kan deze e-mail dienen als bewijs van goedkeuring.

Dear Mr/Ms The Supervisory Committee on Medical ethics of the "Master in de huisartsgeneeskunde (Leuven e.a.)" programme has reviewed your master's thesis project proposal "Perceptions and experiences on using computerised clinical decision support systems to implement recommendations: a qualitative evidence synthesis" and advises in its favour. This means that the committee has acknowledged that your project, as described in the protocol, is scientifically relevant and in line with prevailing ethical standards. This favourable advice does not entail the committee's responsibility for the planned project, however. You remain solely responsible. If you intend to publish your master's thesis, this e-mail may be used as proof of the committee's consent.

Met vriendelijke groeten

Opleidingsspecifieke begeleidingscommissie van de opleiding Master in de huisartsgeneeskunde (Leuven e.a.)