UvA-DARE is a service provided by the library of the University of Amsterdam (http://dare.uva.nl)

UvA-DARE (Digital Academic Repository)

Improving quality of intensive careOptimizing audit & feedback with actionable indicators and an action implementation toolboxBlom, M.J.

Link to publication

LicenseOther

Citation for published version (APA):Blom, M. J. (2019). Improving quality of intensive care: Optimizing audit & feedback with actionable indicatorsand an action implementation toolbox.

General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s),other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).

Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, statingyour reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Askthe Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam,The Netherlands. You will be contacted as soon as possible.

Download date: 20 Jan 2021

Chapter 9

General discussion

160 | Chapter 9

The overall aim of this thesis was twofold: i) to assist ICU professionals in continuously improving quality of (parts of their) care for critically ill patients and ii) to gain knowledge on how the effectiveness of audit and feedback (A&F) could be increased. To improve quality of care we used A&F which involves a summary of some aspect of clinical performance e.g. using quality indicators (part of audit) over a specific period of time, and subsequent provision of that summary (feedback) to ICU teams [1]. Hence, having relevant, reliable and feasible audit measures (i.e. quality indicators) is a prerequisite for providing feedback that triggers quality improvement activities and consequently improves quality of care [2, 3]. In this final chapter we will present a summary of our main findings, discuss these results in the context of lessons learned regarding both the quality indicator (part of audit) part as well as the feedback part and provide suggestions for future research.

Main findings

In Part I of this thesis we focused on the quality indicator (i.e. part of audit) component of A&F. Together with experts from the field, we developed in Chapter 2 and 3 quality indicators for appropriate antibiotic use and adequate pain management at the ICU following a modified RAND procedure. For antibiotic use this procedure resulted in three process indicators, one structure indicator and one quantity metric (Chapter 2): percentage of patients for whom at least two sets of blood cultures before start of empirical systemic therapy were performed (indicator 1); percentage of patients treated with vancomycin or aminoglycosides in whom therapeutic drug monitoring was performed (indicator 2); percentage of patients treated with selective digestive or oropharyngeal decontamination in whom surveillance cultures were obtained (indicator 3); number of face-to-face meetings between ICU and microbiology staff in which local resistance rates are discussed (indicator 4); and quantitative antibiotic use at the ICU expressed in days of therapy (DOT; indicator 5). In addition, we described in Chapter 2 the development of an action implementation toolbox which can be used to support stewardship actions to increase performance on antibiotic use utilizing the checklist of Flottorp et al. [4]. The final toolbox contains a list of 24 unique possible barriers that might lead to poor performance on the selected indicators, and a list of 37 unique improvement strategies to overcome these barriers. For pain management we developed four indicators (Chapter 3) and assessed their room for improvement, feasibility of data collection, and reliability of the indicators. In 69.2% (interquartile range [IQR], 58.7 to 84.9)

General discussion | 161

9

of the patient-shifts pain was measured at least once (indicator 1); in 56.7% (IQR, 49.6 to 73.5) pain scores were acceptable (indicator 2); in 11.7% (IQR, 5.6 to 26.4) pain assessments with unacceptable scores were remeasured within 1 h (indicator 3); and in 10.9% (IQR, 5.1 to 20.1) unacceptable scores were normalized within 1 h (indicator 4). This means substantial variation and ample room for improvement in pain management across Dutch ICUs exists. We found data collection feasible because data were available for most (~80%) of the admissions. We considered all indicators reliable as they produced consistent performance scores. In Chapter 4 we described the establishment of the action implementation toolbox for pain management using the Systems Engineering Initiative for Patient Safety (SEIPS) model [5]. The toolbox includes expert-based actions complemented with evidence from the literature, resulting in a practice-based list of 26 unique goal-oriented actions and supporting material which enhances its employment.

Part II of this thesis focused on the feedback component of A&F and which efforts we made to increase its effectiveness. Following Control Theory [6], we hypothesized that our developed action implementation toolboxes may support ICU professionals to turn intentions to improve into behaviour change by suggesting improvement actions in case they do not know what they can do to improve (lack of knowledge) and materials in case they do not know how to do it (lack of skills; lack of resources). To assess the effectiveness of the pain management toolbox, we conducted a head-to-head cluster-randomized controlled trial (RCT) in 21 ICUs that received a web-based A&F intervention with or without action implementation toolbox. Chapter 5 describes the study protocol for this cluster RCT with details on the intervention and its rationale. Chapter 6 presents the results of the cluster RCT. We found a moderate absolute improvement in adequate pain management of 4.8% (95% confidence interval [CI], 4.2 to 5.5) after A&F in the group without toolbox, and 14.8% (95% CI, 14.0 to 15.5) in the group with A&F and also the toolbox. ICUs with toolbox achieved larger improvements than those without (p=0.049). Improvement in both groups was mostly due to an increase in measuring pain each shift, and on repeating pain measurements within 1 h after an unacceptable pain score was observed. The mixed-methods process evaluation of the cluster RCT (Chapter 7) showed that ICUs supported by suggested actions from the action implementation toolbox developed more improvement intentions and across a wider range of practice determinants compared to those without the toolbox (p=0.037). ICUs without toolbox tended to remain longer in earlier processes

162 | Chapter 9

of the feedback cycle such as data verification and exploring possible solutions. Regardless of offering the toolbox, all ICUs still experienced barriers relating to the feedback (low controllability; accuracy) and organizational context (competing priorities; resources; cost). In Chapter 8 four organizational factors and their association with adequate pain management were assessed with data from thirteen ICUs. We found that case-mix adjusted nurse to patient ratios of over 0.70 were significantly associated with adequate pain management (odds ratio [OR] 1.14 [95% CI, 1.07 to 1.21] for nurse to patient ratios of 0.70 to 0.80, and OR 1.16 [95% CI, 1.08 to 1.24] for nurse to patient ratios over 0.80). Bed occupancy rate and intensivist presence were not associated with adequate pain management.

Discussion and recommendations – Quality indicators (part of audit)

It is important that quality indicators are accepted and acknowledged by their end users. If measures with little relevance or validity are used to judge quality of care, this often results in several adverse reactions ranging from resistance to gaming [7, 8]. Furthermore, it can divert attention from genuine improvement towards superficial improvement [7]. To enhance their credibility and acceptance, quality indicators should be evidence-based and involvement of representative stakeholders during the development of quality indicators should be encouraged. To ensure credibility of indicators, the development process should ensure that the final indicator adheres to some fundamental a priori characteristics: relevance, feasibility, reliability, actionability, unambiguously, room for improvement and validity [9-12]. During the development of the indicators for antibiotic use and pain management we did incorporate a multi-disciplinary panel to enhance credibility and acceptance (Chapter 2 and 3). However, despite doing so, the intensive care professionals criticized the pain management indicators when we applied them to clinical practice. First, some ICUs complained about data incompleteness which resulted in a lack of trust in the indicators [13, 14]. Second, regarding measuring pain at least once per shift in each patient the ICU professionals argued this was not realistic for patients whose length of stay was less than about two hours during their shift as other care related tasks have more priority. Third, a 100% target for the indicator acceptable pain scores was not realistic according to the ICU professionals because pain scores measured in the first shift of patients’ ICU admission are only partly influenced by pain management in the ICU. Patients from the operating

General discussion | 163

9

room or from the emergency room may arrive at the ICU with high pain scores. Unfortunately this point was not raised during the development process of the pain management indicators and became clear once the indicators were applied into practice. Although one could argue that professionals of the ICU should discuss this issue with their colleagues from the operating theater and emergency room it reduces the actionability of the ICU professionals. To motivate the ICU teams we could ‘repair’ this issue by neglecting the first patient shift of the ICU admission when calculating the indicator score. Fourth, during the development of the indicators remeasuring and normalizing unacceptable pain within 1 h the expert panel already thoroughly discussed which time-threshold to use. The one hour threshold, although considered to be challenging, was chosen because timely pain relief is essential to prevent systemic and physiologic adverse effects [15] and analgesic agents used at the ICU have a quick onset [16, 17]. When we applied these indicators to clinical practice the ICU professionals again argued against the threshold of 1 h, not only because they considered it unrealistic, but also because of timeliness of the measurements. If pain was assessed within 1 h but the score was recorded somewhat later in their electronic health record (EHR) they were not able to change the time to the exact moment of measurement, resulting in a lower performance score on the corresponding indicators. We considered 2 h and 3 h as alternative thresholds, but the sensitivity analyses (results not shown) on these more liberal thresholds still showed large room for improvement. However, ICUs might have judged improvement on the indicators more achievable and actionable when we chose to use one of the more liberal thresholds. In addition, ICU professionals in practice doubted the relevance of remeasuring unacceptable pain within 1 h if they were convinced they treated the patient with an unacceptable pain score correctly and immediately. But, as Skrobik [18] described, without assessment the individual variability in pain perception and response to analgesia cannot be evaluated. ICU professionals in our study practice however mentioned that in these situations pain ís measured again but not recorded in the patient file when pain is resolved as they expected. Another point of criticism was related to normalizing unacceptable pain within 1 h; it could result in the excessive sedation of patients and consecutively in adverse effects such as prolonged mechanical ventilation [19]. However, we expect the clinician will take into account that favorable pharmacokinetic properties not necessarily translate into clinical advantages in the ICU setting [20]. In future an additional indicator on sedation, taking the effect of sedation use without adverse effects and adequate pain relief into account, could be a good extension. Despite all

164 | Chapter 9

concerns mentioned above we showed in Chapter 3 that there is considerable variation between ICUs and ample room for improvement on pain management indicating improvement activities are required.

While the attention towards the administrative burden of registries is rising [21, 22] feasibility of reusing routinely collected data to calculate the indicators [11, 12] becomes increasingly important. During the development of the indicators for antibiotic use and pain management the feasibility of data collection was an important selection criterion and experts rated data collection feasible for the resulting indicators (Chapter 2 and 3). However, during the evaluation of the pain management indicators in daily practice the feasibility turned out to be underestimated (Chapter 3). Judgement of feasibility of data collection might be enhanced when the reliability of routinely measured data is better assessed by taking into account data completeness and timeliness. On beforehand it was said the pain data were available, but in practice not all performed pain measurements were recorded in the EHR especially when pain was treated and remeasurement showed that pain was resolved. Data completeness was mainly reflected by the indicator evaluating if pain was measured each shift, but a measurement each shift was insufficient regarding remeasuring and normalizing pain within one hour. The timeliness of recording pain measurements in the EHR is especially important for determining whether pain was remeasured and normalized pain within one hour. Lastly, it is essential to determine whether the data needed to calculate the indicators can be extracted from the EHR without additional administrative or financial burden. In this thesis we showed 21 of 83 ICUs were able to extract the pain measurement data from their EHR (Chapter 3). ICUs that were not able to extract data reported that the data were not electronically available or they lacked financial or human resources to realize the needed data extraction from their EHR.

The discussion points raised in this section on challenges on developing and measuring quality indicators show that one has to be aware that quality indicators are essential in the audit of care quality but just a part of total quality management. A set of quality indicators sheds a light on a part of the total care provided and therefore indicator development should be a continuous process that needs reconsiderations resulting in indicator revisions over time to preserve support of all stakeholders.

General discussion | 165

9

Discussion and recommendations - Feedback

Previous research of the NICE registry on feedback was empirically based and consisted of an activating performance feedback strategy including local, multidisciplinary quality improvement teams, monthly feedback reports and educational outreach visits [23]. This multifaceted feedback strategy did not result in improved outcomes [24, 25]. A qualitative evaluation showed that ICUs experienced barriers to achieving improvement such as a lack of normative standards and benchmarks, inadequate case-mix adjustment or stratification, lack of knowledge on how to improve, and insufficient allocated time and staff [26]. A Cochrane review of 140 A&F studies [3] showed that feedback is effective, but with only a median 4.3% absolute improvement (interquartile range 0.5 to 16%). Meta-analyses indicated A&F may be more effective when baseline performance is low, it is delivered by a supervisor or colleague, it is provided more than once, it is delivered in both verbal and written formats, and when it includes explicit targets and an action plan [3, 27, 28]. However, it remained unclear how these elements could be best operationalized [29]. To address these barriers and enablers, we incorporated several strategies into the quality dashboard including explicit benchmark information, patient subgroup analyses, lists of patients who had not achieved the indicator target, and an action implementation toolbox containing potential barriers and suggested actions to improve practice (Chapter 2 and 4).

The most recent work of the NICE registry on feedback as described in this thesis (Chapter 5, 6 and 7) was based on extant theory. Conceptualizing audit and feedback within a theoretical framework offers a way forward. We previously performed an online laboratory experiment guided by Control Theory to collect ICU professionals’ perceptions about their clinical performance and improvement intentions before and after receiving first-time feedback [30]. We found the professionals to have good improvement intentions which were further improved by providing feedback. The limited effects typically found by audit and feedback studies are likely predominantly caused by barriers to translation of intentions into actual change in clinical practice. The toolbox we developed and implemented was also based on Control Theory and should help ICU professionals to turn their intention into action and enhance the likelihood that actions will be completed; i.e. closing the intention-behaviour gap (Chapter 5). However, as the cluster RCT revealed marginal improvement on pain management in the group with toolbox as compared to the group

166 | Chapter 9

without toolbox (Chapter 6) we can argue whether the toolbox’s development or appliance could be improved if we look at it in retrospect. We developed the action implementation toolboxes using Flottorp et al.’s checklist for determinants of practice [4] and the Systems Engineering Initiative for Patient Safety (SEIPS) model [5] to identify potential barriers in the care structures and processes that could lead to poor performance on each of the indicators (Chapter 3 and 4). Next, for each barrier we determined expert, literature and guideline-based examples of goal-oriented actions that may improve performance and collected supporting materials that could facilitate the implementation of those actions. Although this was a thorough process, including the level of evidence in selecting the possible improvement actions and material could have improved the credibility of the toolbox [31]. Furthermore, presenting this level of evidence next to the suggested actions within the dashboard might have enhanced ICU professionals’ decision making process on choosing which action would be most effective in improving their quality [32]. Some improvements on the toolbox for future use are discussed from here. First, we offered all possible actions to all ICUs. Possibly, we could have increased the application of the toolbox by improving the overview of relevant actions by instructing the ICUs to deselect all actions which were already implemented in their organization at the start of the study. Second, it could be beneficial to include more actions in the toolbox that specifically improve pain outcomes rather than processes. From our cluster RCT (Chapter 6 and 7) we learned that ICU professionals tended to focus on processes that were included in the indicators (e.g. they focused more on measuring pain every shift than on decreasing pain) which may only indirectly result in improved patient outcomes [33]. The developed set of antibiotic indicators does not include outcome indicators and the present toolbox might therefore be sufficient in supporting ICUs to improve their performance. Thirdly, we found in the cluster RCT and process evaluation that actions are sometimes selected but not completed due to barriers for example at the cultural or organizational level (Chapter 6 and 7). It is questionable whether A&F can ever solve these types of barriers. ICU professionals expressed an additional interest in learning best practices from high performers (Chapter 7). A dynamic toolbox or additional meeting where ICU professionals share experiences on how they overcome cultural or organizational barriers could complement our static list of suggested improvement actions and might be a way forward.

The pain management quality indicators and action implementation toolbox were provided to the ICUs on a web-based dashboard (Chapter 4 and 5). The

General discussion | 167

9

dashboard needs to be accessed proactively. Some critics say that computerized decision support (CDS) systems may be more effective as these deliver guideline-based recommendations at the time and place where clinical decisions are made [34]. However, currently not all EHRs are suitable for integrating CDS systems and reminders or alerts of the CDS system often interrupt workflow and can result in alert fatigue [35]. CDS alerts are therefore ineffective at some medical domains such as sepsis where feedback might be effective [36]. CDS systems give patient-specific alerts at the point of care which can be overridden for good reasons (justified overriding) [37]. These intentional reasons for non-adherence are not always appropriate [38, 39]. As performance dashboards include (i) the provision of summary data on quality indicators against benchmarks and (ii) the use of data visualization techniques to provide feedback to leaders, ICU teams or individual health care professionals, these dashboards can inform them how they perform compared to their peers and thereby interpret the justification to deviate from the guidelines [40]. We therefore believe that CDS systems and A&F systems complement each other.

Future perspectives

We made several efforts to contribute to the ongoing need for evaluating the strengths and shortcomings of the care an ICU provides in order to improve quality of care. During all the steps we took we have learned some lessons but also new questions and implementation ideas have originated.

First, in accordance with the AIRE (appraisal of indicators through research and evaluation) criteria [11] the completeness, timeliness and accuracy of routinely collected data can be assessed more extensive than we did for the pain management indicators. This is especially important in the light of the rising attention to the administrative burden of quality registries. We will utilize this extensive assessment to other medical domains within the ICU such as antibiotic use, mechanical ventilation and blood use.

Second, we will aim to increase the effectiveness of the action implementation toolbox. This might be realized by continuously expanding the toolbox with new evidence based suggestions and materials for improvement. In addition, enabling ICU professionals to share best practices via a professional community based toolbox will inform ICUs with room for improvement how they can learn from best performers.

168 | Chapter 9

Third, to assure dashboard use and to monitor progress with the action plans we called the ICUs every 4-6 weeks during the cluster RCT. This supported the ICUs to use the dashboard, but these calls are also resource intensive and disrupt the ICU professionals in their daily work. An alternative method to promote the access and use of the dashboard may be a tailored and motivational email that is sent to the ICU professionals [41, 42]. To determine whether this will be an effective method in intensive care medicine we can conduct a study within the NICE registry where the efficacy of these emails compared to phone calls is assessed.

Fourth, as A&F provided via the dashboard needs to be accessed proactively and CDS systems produce alerts at the point of care an opportunity is to investigate to what extent these two methods can enhance each other.

Conclusion

We conclude that an electronic A&F intervention based on actionable indicators and extended with an action implementation toolbox has and improved effectiveness. Quality indicators are essential in the audit of care quality and continuous quality improvement. As medical knowledge as well as technical data infrastructures (e.g EHRs) evolve, indicator development should be a continuous process that needs reconsiderations. This will result in indicator revisions over time preserving high data quality and support of all stakeholders. The toolbox led to improvement in ICU pain management and helped the ICU professionals to initiate actions. Therefore, to optimize future A&F interventions’ effectiveness we recommend to provide health care professionals suggestions of actions for improvement which address barriers for practice change.

General discussion | 169

9

References

[1] Brehaut JC, Eva KW. Building theories of knowledge translation interventions: use the entire menu of constructs. Implement Sci 2012;7:114.

[2] Mainz J. Developing evidence-based clinical indicators: a state of the art methods primer. Int J Qual Health Care 2003;15 Suppl 1:i5-11.

[3] Ivers NM, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;6(6):CD000259.

[4] Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci 2013;8:35.

[5] Carayon P, Schoofs Hundt A, Karsh BT, Gurses AP, Alvarado CJ, Smith M, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006;15 Suppl 1:i50-8.

[6] Carver CS, Scheier MF. Control theory: a useful conceptual framework for personality-social, clinical, and health psychology. Psychol Bull 1982;92(1):111-35.

[7] Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet 2004;363(9415):1147-54.

[8] Lester HE, Hannon KL, Campbell SM. Identifying unintended consequences of quality indicators: a qualitative study. BMJ Qual Saf 2011;20(12):1057-61.

[9] Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care 2002;11(4):358-64.

[10] Center for Health Policy/Center for Primary Care and Outcomes Research & Battelle Memorial Institute. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement. Rockville, MD: Agency for Healthcare Research and Quality; 2011.

[11] de Koning J, Smulders A, Klazinga NS. Appraisal of Indicators through Research and Evaluation (AIRE). Amsterdam: Academisch Medisch Centrum Universiteit van Amsterdam, afdeling Sociale Geneeskunde; 2007.

[12] Kelley E, Hurst J. Health Care Quality Indicators Project: Conceptual Framework Paper. Paris: OECD Publishing; 2006.

[13] Arts DG, De Keizer NF, Scheffer GJ. Defining and improving data quality in medical registries: a literature review, case study, and generic framework. J Am Med Inform Assoc 2002;9(6):600-11.

170 | Chapter 9

[14] van der Veer SN, de Keizer NF, Ravelli AC, Tenkink S, Jager KJ. Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform 2010;79(5):305-23.

[15] Lindenbaum L, Milia DJ. Pain management in the ICU. Surg Clin North Am 2012;92(6):1621-36.

[16] Joffe AM, Hallman M, Gelinas C, Herr DL, Puntillo K. Evaluation and treatment of pain in critically ill adults. Semin Respir Crit Care Med 2013;34(2):189-200.

[17] Hall LG, Oyen LJ, Murray MJ. Analgesic agents. Pharmacology and application in critical care. Crit Care Clin 2001;17(4):899-923, viii.

[18] Skrobik Y. Pain may be inevitable; inadequate management is not. Crit Care 2008;12(2):142.

[19] Kollef MH, Levy NT, Ahrens TS, Schaiff R, Prentice D, Sherman G. The use of continuous i.v. sedation is associated with prolongation of mechanical ventilation. Chest 1998;114(2):541-8.

[20] Erstad BL, Puntillo K, Gilbert HC, Grap MJ, Li D, Medina J, et al. Pain management principles in the critically ill. Chest 2009;135(4):1075-86.

[21] Rijksoverheid. (Ont)Regel de Zorg, https://www.ordz.nl/; 2019 [accessed May 15, 2019].

[22] Zegers M, Gerritse G, Welker G. Registratielast: laat de teugels vieren. Medisch Contact 2018;41:32-3.

[23] van der Veer SN, de Vos ML, Jager KJ, van der Voort PH, Peek N, Westert GP, et al. Evaluating the effectiveness of a tailored multifaceted performance feedback intervention to improve the quality of care: protocol for a cluster randomized trial in intensive care. Implement Sci 2011;6:119.

[24] de Vos ML, van der Veer SN, Wouterse B, Graafmans WC, Peek N, de Keizer NF, et al. A multifaceted feedback strategy alone does not improve the adherence to organizational guideline-based standards: a cluster randomized trial in intensive care. Implement Sci 2015;10:95.

[25] van der Veer SN, de Vos ML, van der Voort PH, Peek N, Abu-Hanna A, Westert GP, et al. Effect of a Multifaceted Performance Feedback Strategy on Length of Stay Compared With Benchmark Reports Alone: A Cluster Randomized Trial in Intensive Care. Crit Care Med 2013;41(8):1893-904.

[26] de Vos ML, van der Veer SN, Graafmans WC, de Keizer NF, Jager KJ, Westert GP, et al. Process evaluation of a tailored multifaceted feedback program to improve the quality of intensive care by using quality indicators. BMJ Qual Saf 2013;22(3):233-41.

[27] Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care 2009;47(3):356-63.

General discussion | 171

9

[28] Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med 2010;70(10):1618-25.

[29] Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res 2005;5:50.

[30] Gude WT, Roos-Blom MJ, van der Veer SN, Dongelmans DA, de Jonge E, Francis JJ, et al. Health professionals’ perceptions about their clinical performance and the influence of audit and feedback on their intentions to improve practice: a theory-based study in Dutch intensive care units. Implement Sci 2018;13(1):33.

[31] Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet 2017;390(10092):415-23.

[32] Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust 2004;180(6 Suppl):S57-60.

[33] Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care 2001;13(6):469-74.

[34] Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293(10):1223-38.

[35] Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013;13:111.

[36] Downing NL, Rolnick J, Poole SF, Hall E, Wessels AJ, Heidenreich P, et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ Qual Saf 2019.

[37] van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-47.

[38] Arts DL, Voncken AG, Medlock S, Abu-Hanna A, van Weert HC. Reasons for intentional guideline non-adherence: A systematic review. Int J Med Inform 2016;89:55-62.

[39] Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018;25(5):476-81.

[40] Dowding D, Randell R, Gardner P, Fitzpatrick G, Dykes P, Favela J, et al. Dashboards for improving patient care: review of the literature. Int J Med Inform 2015;84(2):87-100.

172 | Chapter 9

[41] Bravo CA, Llovet D, Witteman HO, Desveaux L, Presseau J, Saragosa M, et al. Designing Emails Aimed at Increasing Family Physicians’ Use of a Web-Based Audit and Feedback Tool to Improve Cancer Screening Rates: Cocreation Process. JMIR Hum Factors 2018;5(3):e25.

[42] Vaisson G, Witteman HO, Bouck Z, Bravo CA, Desveaux L, Llovet D, et al. Testing Behavior Change Techniques to Encourage Primary Care Physicians to Access Cancer Screening Audit and Feedback Reports: Protocol for a Factorial Randomized Experiment of Email Content. JMIR Res Protoc 2018;7(2):e11.