Tackling alert fatigue with a semi-automated clinical decision support system: quantitative evaluation and end-user survey

DOI: https://doi.org/https://doi.org/10.57187/smw.2023.40082

Introduction

A medication error is any preventable event that can cause or lead to either inappropriate medication use or harm to the patient during the process of medication [1]. Studies suggest that approximately 5% of all medication involves a medication error; the estimate varies depending on the definition of medication error and the collective of patients observed in the study [2–4]. Medication errors are not only a risk to patient safety, but they also contribute to health costs [5–7].

Clinical decision support systems (CDSSs) embedded in hospital electronic health records (EHR) can result in significant reduction of medication errors [8, 9]. However, the most frequently mentioned problem of a CDSS is the phenomenon of alert fatigue, which describes the ignoring or overriding of alerts [10]. Common reasons for alert fatigue are the reception of high numbers of irrelevant alerts and desensitisation from repeated exposure to the same alert, particularly in a demanding and complex working environment [11]. Especially interruptive pop-up notifications seem to trigger alert fatigue, with override rates between 50% and >90% [12, 13]. There is a trade-off between interruption of workflow and visibility of the alerts for interruptive and non-interruptive CDSSs [14].

A CDSS can only achieve its potential if its target users actually use it [15]. Therefore, the focus has changed from sensitive alerts towards the increase of CDSS alert efficiency and usability [16, 17]. While the number of CDSSs is increasing, information concerning physicians’ perception of such CDSSs is still scarce. Recent surveys reveal that some physicians seem to be completely dissatisfied, complaining of high numbers of irrelevant alerts [18], while others appear to appreciate the added value despite the system's shortcomings [19].

In our hospital, we designed and introduced a semi-automated CDSS (KPharm) into the hospital's EHR. The semi-automated approach involves role tailoring, meaning that some alerts are automatically send to the prescribing physicians, while others are first assessed by a clinical pharmacist. With 20 different context-based algorithms currently running, we evaluated the CDSS’s performance in terms of alert burden and acceptance rate. Furthermore, we captured the physicians’ satisfaction with our CDSS with an end-user survey.

Methods

Setting

KPharm is a clinical decision support system that was developed and implemented in a tertiary care hospital with 669 beds in the north-western part of Switzerland. It was initiated as a collaboration between the department of internal medicine and the clinical pharmacy with the aim of increasing drug safety. The overall goal of KPharm is the early detection of medication errors or situations with an increased risk of an adverse drug event (ADE). A team of clinical pharmacists and physicians developed drug safety algorithms, each covering a variety of potential high-risk situations. The algorithms had been in use in an external system since 2018. Starting from 2020, they were continuously implemented in the hospital’s EHR (KISIM™ by CISTEC).

The algorithms were designed taking the following principles into consideration:

• High specificity: To fight alert fatigue, we aim to achieve a high specificity while maintaining a tolerable sensitivity.
• Timing: Considering the daily workflow of ward rounds, we introduced a time lag of one hour. This allows physicians to complete the task of prescribing and to work autonomously from the CDSS. The CDSS acts as a safety net. Many alerts have shown to be self-limiting in the first few hours.
• Non-interruptive notifications: Each interruption in the workflow increases the probability of medication errors [20]. Unlike common CDSSs, we use non-interruptive notifications that are displayed in the patient’s EHR. Here, each health care professional working with the patient can see them, which decreases the risk that they are being missed.
• Parametric: The sensitivity of the alerts can be controlled with parametric thresholds. Further, the CDSS can be easily accustomed to new drugs of hospital specific guidelines.
• Semi-automation: For each individual alert, we can define if it automatically produces a notification in the patient's EHR or if it is first evaluated by a clinical pharmacist.

The drug safety algorithms continuously check the medication of inpatients while taking into account various aspects, such as prescribed dose, already administered doses, patient characteristics, and laboratory values. Since diagnoses are not stored in a coded manner, the algorithms cannot consider them. When a potential drug-related problem is detected, an alert is triggered. The CDSS is semi-automated: alerts addressing critical errors with high specificity (e.g., duplication of anticoagulants or digoxin overdoses) are automatically displayed in the patient's EHR. For some alerts, the automatisation is only temporarily (on weekends, holidays). The remaining alerts are evaluated by the clinical pharmacist on day duty. In our hospital, clinical pharmacists’ day duty mainly consists of responding to the telephone and assessing the alerts, which permits the timely evaluation of these during office hours. To assure consistency, the alerts are processed according to an internal guidebook. The guidebook takes further aspects into account that cannot be used by the algorithm, e.g., because the information is not available in a structured form. If the alert is considered relevant, a notification is generated which is displayed in the “messages” line of the EHR. In critical cases, the prescribing physician is contacted directly by telephone. The algorithms will cancel the alerts and notifications autonomously as soon as the conditions that triggered them no longer apply. The workflow is illustrated in figure 1.

Figure 1Workflow of the KPharm clinical decision support system: After the physician prescribes a medication, the context-based KPharm algorithms access the information in the electronic health records, as well as the chronology of events, to trigger individualised alerts. Alerts are forwarded either directly to the physician or first to a clinical pharmacist for review. An automatic constant re-evaluation assures that only relevant alerts are displayed. KPharm: semi-automated clinical decision support system.

The text of the notification itself contains, in addition to the triggering factors, a brief explanation and a specific recommendation for action (figure 2). If desired, the physician can respond to the notification for clarification, cancel it, or leave it in the EHR as a reminder.

Figure 2Processing of all alerts in 2021: The left column shows the alerts that were automatically sent to physicians. The right column shows the alerts that were first sent to clinical pharmacists. All alerts that reached physicians through both ways are summarised in the box at the bottom.

By the end of 2021, we had integrated 19 algorithms that can trigger 194 different alerts, including error messages. The decision to automate individual alerts was taken as a group (CZ, RF, HD). We considered the frequency of alerts and how often they were correct and accepted by the physician, as well as the potential patient harm if they were not processed immediately. A list of the alerts and their automation status can be found in the appendix 1.

Data

The clinical decision support system is integrated into the hospitals’ EHR. The clinical pharmacists process the alerts in an interface for checking, where the status of each alert is documented. Alerts that are technically wrong (false positive) were identified during this step. Alerts that were no longer valid before being reviewed by a clinical pharmacist were automatically flagged as “self-limiting”. Alerts that were technically correct but not relevant for the individual patient were either paused or marked as “not relevant” and were not forwarded to the physician. The CDSS does not automatically capture whether the notifications are read and accepted by physicians. Thus, when the alert resulted in a notification to the physician, the clinical pharmacists performed a retrospective follow-up to assess whether the notification had been accepted or dismissed. We considered an intervention as accepted if it led to an adjustment in patient management within a reasonable period. The period considered reasonable depended on the recommendation of action in the notification and the time of day in which the notification was send. If discontinuation or dose adjustment of a drug was suggested, we expected a change in medication on the same day, or if the notification was sent late in the afternoon, the following morning. In case of uncertainty, e.g., if the notification was sent shortly before the patient was discharged from the hospital or if the medication was discontinued for a reason not related to the notification, the follow-up was deemed not assessable.

Web-based survey

The questionnaire was designed with key stakeholders of the development team in German language (appendix 2). The questionnaire is composed of three categories: demographic information, opinions specifically concerning KPharm, and opinions concerning CDSSs in general. The a priori outcome measures were overall satisfaction and the helpfulness of the individual algorithms. We collected the following demographic information: gender, age, professional title, clinic, percentage of employment, and time at workplace. Concerning KPharm, we asked about overall satisfaction, perceived frequency of alerts, and a rating of the individual algorithms. We proposed several statements covering relevant aspects of CDSSs to capture physicians’ attitudes towards KPharm. Subsequently, physicians rated how important those aspects are for CDSS in general. Additional questions targeted the preferred mode of receiving notifications, further drug related problems to be solved in the future, and experience with other CDSSs. We used an even Likert scale for items that required rating. At the time of the survey, 16 algorithms were fully operating, allowing for 193 individual alerts. For the survey, the algorithms for direct oral anticoagulants (DOACs) were treated as one.

The survey was distributed to physicians working on wards in our hospital via email using electronic software (SurveyMonkey™, Momentive Inc., San Mateo, California, USA). Physicians from wards that do not work with the EHR were not contacted. The survey period was 16 days, with two reminders being sent on days 7 and 14. We promoted the survey in the hospitals intranet and raffled three vouchers for a lunch at the hospital’s canteen amongst the participants as an incentive. By participating, the physicians agreed to their anonymised data being published.

Ethics approval

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the north-western and central Switzerland ethics committee (Project-ID: 2021-01379)

The online survey did not fall within the scope of the Swiss Human Research Act. Therefore, authorisation from the ethics committee was not required. All participants of the online survey agreed to their answers being published in an anonymised form.

Statistical analysis

Information on all alerts in 2021 was imported from the EHR. Quantitative data were summarised using counts and proportions. We calculated the acceptance rates as the fraction of accepted notifications out of all notifications for which the outcome was known. Alert burden was defined as the mean number of alerts pharmacists and physicians must process in a day.

The results of the online survey were received via export from the SurveyMonkey™ homepage. Demographic characteristics were aggregated to conduct descriptive analysis using counts and proportions and median and range, where appropriate. Responses to the question concerning overall satisfaction and alert burden were stratified by profession. Due to the small number of participants, attending physicians and chiefs of service were analysed together. We analysed Likert scales using counts and proportions and omitted non-assessable responses. For the statements covering relevant aspects of CDSSs, Cronbach’s alpha was calculated. With a value of 0.85, it showed good validity.

All analyses were carried out using Jupyter® Notebook (version 6.1.5) with Python™ (version 3.9.2) and the additional packages numpy (version 1.23.5), pandas (version 1.5.2), and matplotlib (version 3.6.3) [21–23]. The code is available from github upon request.

Results

Quantitative evaluation

In the year 2021, the CDSS analysed 22,195 patient charts and generated 10,556 alerts for 5,204 individual patients. The highest number of alerts received for an individual patient was 27. Only a small fraction (5.9%, n = 619) of the alerts was automated and directly generated a notification to the prescribing physician. The other alerts were directed to a clinical pharmacist for assessment. About a third of the alerts (32.2%, n = 3,394) were self-limiting, meaning that the alert had already ended itself before assessment. The remaining 6,543 alerts were assessed by the respective clinical pharmacist on duty, which corresponds to a workload of 17.2 alerts per day. In the course of 2021, clinical pharmacists performed 2,231 (34.1% of all assessed alerts) interventions by either phone call or notification. Other alerts were considered correct but not relevant for the specific clinical situation and were either ended (n = 1,948, 29.8%) or paused (n = 2,227, 34.0%). A small number of alerts (n = 135, 2.1%) were false positive. The processing of the alerts is displayed in figure 2.

Automated and non-automated alerts combined, 27.0% (n = 2,850) of all alerts in 2021 resulted in a notification to the physician. This corresponds to 7.8 alerts per day for the whole hospital. The team of clinical pharmacists performed a retrospective follow-up and was able to determine the outcome for 76.2% (n = 2,094) of the notifications. The acceptance rate was higher for the automated alerts (89.8%) than for pharmacist-reviewed alerts (68.4%). Overall, the physicians accepted 1,608 notifications, which translates to an acceptance rate of 72.4%. Alerts concerning the duplication of anticoagulants had the highest acceptance rate (93.3%, total number of alerts = 276), whereas alerts concerning the dosing of dabigatran had the lowest acceptance rate (33.3%, total number of alerts = 4).

Survey

The survey was sent to 568 physicians, of which 152 participated, representing a response rate of 26.8%. Of the participants, 113 completed the full survey, whereas 39 (24.2%) submitted only partially answered questions. The median age was 35 years (range: 25–60 years) and slightly more females participated (57.2%). Most participants were physicians at entry level, were employed full time in the internal medicine department, and had worked in our hospital for a medium of 3 years (range: 0–22 years). The characteristics of the participants are shown in table 1.

Overall satisfaction with the CDSS was high, with 66.1% being “satisfied”, 28.4% being “very satisfied”, and 5.5% being 'less satisfied'. Forty-three (28.3%) participants did not answer. Most participants (n = 75, 49.3%) indicated that they had received at least one alert in the previous year, whereas 48 (31.6%) had never received an alert and a further 29 did not respond. Five of the six doctors who were less satisfied with the system said that they had not received any alerts in the past year. Resident physicians received more notifications than attending physicians and chief of service/head of departments.

All algorithms were considered helpful, with the algorithm addressing the duplication of anticoagulants receiving the highest results and the algorithm for medication errors of proton pump inhibitors receiving the lowest rating; however, many physicians indicated not being familiar with specific algorithms (mean: 61.4% for all algorithms). The least known algorithm was “xanthine oxidase inhibitor” which was unknown to 86.6% of responding physicians (n = 119). In the course of 2021, this specific algorithm had only fired 27 alerts. An overview of all ratings is depicted in figure 3.

Figure 3Helpfulness of individual alerts. Distribution of answers on the perceived helpfulness of the individual algorithms. Helpfulness was rated using a Likert scale of 4 + 1 for each algorithm; the answer “I do not know the algorithm” was excluded in this figure. DOAC: direct oral anticoagulants (apixaban, edoxaban, rivaroxaban, and dabigatran were treated as one algorithm); NSAID: non-steroidal anti-inflammatory drugs; RAS: drugs inhibiting the renin–angiotensin system.

Physicians' attitudes towards relevant aspects of CDSSs were captured by proposing several statements and asking the physicians to rate the extent to which the statements were true in respect to our CDSS. In a second question, the physicians rated how important this statement is for CDSSs in general. All proposed statements were rated as rather important in a CDSS. For an ideal CDSS, convenient timing, relevance of medication errors, and addressing the correct health care professional received the most very important votes. KPharm received strong confirmation that the alerts were written in an understandable manner. Eight (10.1%, total respondents = 79) of the physicians did not agree to the statement that the timing of the alerts was convenient. This was the statement with the highest rate of disagreement. Overall, the agreement rates were high (figure 4).

Figure 4Agreement to satisfaction parameters for clinical decision support systems in general and for KPharm. Distribution of answers regarding satisfaction parameters. Physicians were asked how important each parameter is for a clinical decision support system in general (left) and if those statements were true in regards to KPharm (right). KPharm: semi-automated clinical decision support system.

Most physicians indicated that they preferred “pop-ups” (38.0%) as a mode of receiving notifications, followed by notifications in the EHR, emails, and phone calls (28.3%, 21.2%, and 9.2%, respectively). Physicians who responded with 'other' (3.3%) indicated that they would like the mode of notification to be adapted to the urgency of the content. With only 27 physicians making suggestions for future algorithms, we could not establish a clear hierarchy of topics. The most common answer (n = 6) was the need for an allergy alert. Out of the 114 respondents, only seven (6.1%) indicated that they already had experience with other CDSSs, obviating further questions aimed at a comparison of perceptions of different CDSSs.

Discussion

We evaluated the performance of our clinical decision support system KPharm in terms of acceptance rate and alert burden. The quantitative analysis revealed an overall acceptance rate of 72.4%, which is higher than many commercially available CDSSs [19, 24–26]. Since different methods for measuring acceptance are in use, the direct comparison of acceptance rates is difficult [27]. Recent reviews summarised the overwrite rates of CDSSs with drug-drug interaction alerts, which can be interpreted as the reciprocal of the acceptance rate, and found overwrite rates between 46.2% and 98% [12, 28]. However, those reviews included CDSSs since 2000, and advanced CDSSs with conceptualised alerts have been found to have improved performance parameters.

A promising approach to improving performance parameters is role tailoring. In a review by Hussain et al. comparing 39 CDSSs with different designs, only the four designs that involved role tailoring appeared to increase prescriber acceptance [27]. Muylle et al. implemented clinical pharmacist interventions to improve their CDSS [6]. Out of a total of 2,630 alerts within 8 months, 61 (2.3%) led to an intervention through clinical pharmacy. These interventions were accepted 53 times (86.9%). Likewise, Skalafouris et al. presented a CDSS that involves clinical pharmacists [29]. Of the 447 alerts within 132 days, 20.1% were forwarded to the physician by the clinical pharmacist. The physicians accepted 71.0% of the suggested interventions. The acceptance of CDSS alerts preselected by clinical pharmacists may be compared with the acceptance of pharmacist interventions without a CDSS, which have been studied more thoroughly [30–32]. According to the setting and medication errors addressed, the acceptance rates vary greatly and can range between 41% and 95% [30, 33]. While the acceptance rates of CDSSs with clinical pharmacists are not lower than those of pharmacists’ interventions, they are more time efficient [29].

With KPharm, we combine two strategies by having high priority and specificity alerts that go directly to the physician and semi-automated alerts that are first evaluated by a clinical pharmacist. In our analysis, the acceptance rate was higher for direct alerts than for pharmacist interventions. This may be due to an underlying selection bias, since we only automated alerts that have shown a high specificity and acceptance rate in the past.

A major factor contributing to alert fatigue is alert burden. The more alerts a physician receives, the less likely he will accept the intervention due to desensitisation [11]. The comparison of alert burden between different CDSSs is often not meaningful, since it depends largely on the number of medication errors covered by the algorithms. Nonetheless, in a cross-sectional survey with more than 1000 participants from 8 countries, around half of the respondents regarded possible alert overload as a major problem [34]. We found the alert burden of KPharm to be acceptable for both clinical pharmacists and physicians. In our experience, up to 20 alerts a day are very well manageable for one pharmacist, especially since the text for the notifications is already pre-written and rarely needs adjustment. Positive feedback from pharmacists indicates little alert fatigue on their side. Also, the physicians’ alert burden was perceived as low, with only 4.6% of physicians receiving several alerts per week. Moreover, 93.9% of physicians participating in the survey agreed with the statement that the number of alerts was appropriate.

Another factor that largely contributes to the low alert burden is the timing of the alerts. Taking clinical reality in hospitals into account, we introduced the time lag of one hour, in which the physicians had the possibility to detect and correct potential medication errors by themselves. Further, the implementation of the CDSS in the EHR allows for an hourly re-evaluation of the alerts, and the alert is cancelled automatically when the conditions that trigger it no longer apply. Interestingly, even with the time lag of an hour, a large proportion of alerts were self-limiting within the first four hours. Overall, the involvement of pharmacists and the hourly re-evaluation of alerts effectively reduced the alert burden of physicians by 73.4% from 10,705 alerts to 2850 alerts in 2021, eliminating one major risk factor for alert fatigue: irrelevant alerts.

When designing alerts with high specificity, we encountered several difficulties. In our EHR, diagnoses are not coded until after the patient leaves the hospital. Thus, they could not be integrated in the algorithms but would have been tremendously helpful in some situations. For example, the algorithm detecting unnecessary proton pump inhibitors would be better if patients with gastrointestinal bleeding were excluded automatically. In some cases, we found workarounds: the prescription of Entresto® (sacubitril/valsartan) or Verquvo® (vericiguat) can be used as a proxy for the diagnosis of heart insufficiency since this is the only indication for those drugs. Alerts that consider the chronology of events are dependent on a timely documentation.

We measured the physicians' satisfaction with our CDSS with an end-user survey. The results of the online survey showed a vast support amongst physicians for our CDSS. This is also supported by the response rate, which is higher than in similar surveys [18, 19, 35]. Some participants indicated that they were not receiving alerts in their daily work but participated in the survey because they welcomed the project and considered it relevant for drug safety.

While all algorithms were generally well accepted, those addressing haematological medication errors received the highest support. We assume that physicians’ perceptions of usefulness are determined by the gravity of the medication error: a duplicated anticoagulant is worse than receiving a proton pump inhibitor without indication. Interestingly, many physicians indicated not being familiar with certain algorithms. For this, we provide the following hypotheses. Firstly, high specificity reduces the number of alerts, and the quantity of alerts produced by the individual algorithms differs greatly. Therefore, some physicians may never have encountered such an alert. Secondly, not every algorithm is equally relevant to certain clinics. For example, a physician from the clinic of ophthalmology may rarely encounter an alert for methotrexate.

Limitations

The quantitative evaluation comes with several limitations. The CDSS does not automatically register if a notification is read by the physician. Therefore, it was necessary to perform a follow-up by revisiting the patient's chart. The follow-up was performed retrospectively by a team of clinical pharmacists. Due to the large number of different recommendations, we had no universal standard as to what “accepted” means. When in doubt, the alert was discussed with a colleague. However, in cases where such consent was not sought, personal interpretations of the clinical situation may differ. There is a possibility that measures such as dose adjustments were done independently from the notification. We tried to account for this by paying close attention to the chronology of events after the notification was posted in the patient’s chart. When in doubt, the alert was deemed ‘not assessable’ and excluded from the calculation of the acceptance rate. Since 9.6% (275 of 2850) of notifications were lost to follow-up and could not be evaluated, the data may be skewed. Moreover, if a patient received several relevant alerts on the same day, they were combined in one notification in order to reduce the “message burden”, while the others were paused. This analysis does not account for this, so the number of interventions is underestimated.

A limitation to the online survey may be an underlying selection bias. Since participation was voluntary, physicians with a positive attitude towards our CDSS may have been more inclined to participate. However, the response rate was high in comparison to other surveys [18, 19]. The raffling of a free lunch voucher, valued at approx. 15 CHF, may have been a strong incentive. Although we showed a picture of an alert and asked if physicians had already seen such an alert, other physicians may have been motivated to participate. The high number of physicians who claim to never have received an alert may influence the generality of the responses. Those physicians might be either senior physicians and/or physicians working in a ward predominantly using another EHR. Due to the raffle incentive, the email addresses of the participants were known to the authors, and fear of being identified as a poor prescriber might have influenced the responses, especially regarding the number of received alerts. Somewhat surprisingly, most of the physicians who indicated to be less satisfied with the system also stated that they had never received any alerts. Since we did not include a free text option to this question, the reasoning behind the dissatisfaction remains speculative.

For the evaluation of clinical decision support, PPV (positive predictive value) and NPV (negative predictive value) are relevant parameters. Previous studies have shown that the PPV of CDSSs is commonly low [36]. CDSSs that include a greater number of patients’ individual parameters tend to have a higher PPV than automated CDSSs. We are not able to provide those metrics at this point but are currently performing such analysis for individual alerts and intend to publish the results in the future.

Outlook

As of May 2022, we have 20 context-based algorithms embedded in the EHR, which allow us to find 202 individual potential drug-related problems. We plan to develop further algorithms in the future in close collaboration with our hospitals’ physicians. We also seek cooperation with other developers of CDSSs, as the improvement and spreading of effective and specific contextualised algorithms will ultimately benefit all patients. The KPharm algorithms and their implementation into our EHR (KISIM™) will be commercially available in the future.

We carefully considered the feedback from the survey to find ways to further tailor KPharm according to physicians’ needs. When comparing the physicians’ agreement in relation to the statements regarding important aspects of a CDSS, we noticed that while timing is regarded as very important in a CDSS in general, KPharm's rating in relation to this was comparably low. Moreover, physicians indicated that their preferred way of receiving notifications was a pop-up alert. Both aspects, timing and mode of notification, were discussed within the development team. While we will maintain our paradigm of non-interruptive alerts, we plan to automate further alerts with high specificity and acceptance rates. This will reduce the alert delay resulting from pharmacists assessing alerts.

Conclusion

The development of context-based algorithms with specific algorithms resulted in alerts with a comparably high acceptance rate. The involvement of clinical pharmacists in a semi-automated CDSS is a promising approach to limit the alert burden of physicians and tackle alert fatigue. The alert burden for the physicians was low, and a vast majority indicated that the number of alerts was appropriate. The CDSS is well accepted amongst our physicians.

Data sharing

The datasets are available from the authors upon reasonable request.

Acknowledgments

The authors would like to thank all participants of the online survey. Special thanks go to all the clinical pharmacists at the hospital who performed follow-ups on all alerts and Alison C Zaugg for language editing the manuscript.

Authors contributions: Conceptualisation: HD, RF, PS, CZ; Programming and implementation: RDI, HD, CZ; Funding acquisition: RF, PS, CZ; Data curation: HD, CZ; Formal analysis: HD; Writing – original draft: HD; Supervision, writing – review and editing: CZ, RF, RDI, PS. All authors have read and agreed to the final version of the manuscript.

References

1. Aronson JK. Medication errors: definitions and classification. Br J Clin Pharmacol. 2009 Jun;67(6):599–604. 10.1111/j.1365-2125.2009.03415.x 

2. Krähenbühl-Melcher A, Schlienger R, Lampert M, Haschke M, Drewe J, Krähenbühl S. Drug-related problems in hospitals: a review of the recent literature. Drug Saf. 2007;30(5):379–407. 10.2165/00002018-200730050-00003 

3. Nanji KC, Patel A, Shaikh S, Seger DL, Bates DW. Evaluation of Perioperative Medication Errors and Adverse Drug Events. Anesthesiology. 2016 Jan;124(1):25–34. 10.1097/ALN.0000000000000904 

4. Westbrook JI, Sunderland NS, Woods A, Raban MZ, Gates P, Li L. Changes in medication administration error rates associated with the introduction of electronic medication systems in hospitals: a multisite controlled before and after study. BMJ Health Care Inform. 2020 Aug;27(3):e100170. 10.1136/bmjhci-2020-100170 

5.          Convertino I, Salvadori S, Pecori A, Galiulo MT, Ferraro S, Parrilli M, et al. Potential Direct Costs of Adverse Drug Events and Possible Cost Savings Achievable by their Prevention in Tuscany, Italy: A Model-Based Analysis. Drug Saf. 2019 03;42(3):427-44. 

6. Muylle KM, Gentens K, Dupont AG, Cornu P. Evaluation of an optimized context-aware clinical decision support system for drug-drug interaction screening. Int J Med Inform. 2021 Apr;148:104393. 10.1016/j.ijmedinf.2021.104393 

7. Hardmeier B, Braunschweig S, Cavallaro M, Roos M, Pauli-Magnus C, Giger M, et al. Adverse drug events caused by medication errors in medical inpatients. Swiss Med Wkly. 2004 Nov;134(45-46):664–70. 10.4414/smw.2004.10801 

8. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003 Jun;163(12):1409–16. 10.1001/archinte.163.12.1409 

9. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020 Feb;3(1):17. 10.1038/s41746-020-0221-y 

10. Cash JJ. Alert fatigue. Am J Health Syst Pharm. 2009 Dec;66(23):2098–101. 10.2146/ajhp090181 

11.        Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R, et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017 04;17(1):36. 10.1186/s12911-017-0430-8 

12. Poly TN, Islam MM, Yang HC, Li YJ. Appropriateness of Overridden Alerts in Computerized Physician Order Entry: systematic Review. JMIR Med Inform. 2020 Jul;8(7):e15653. 10.2196/15653 

13.        Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc. 2018 05 01;25(5):476-81. 

14. Blecker S, Pandya R, Stork S, Mann D, Kuperman G, Shelley D, et al. Interruptive Versus Noninterruptive Clinical Decision Support: usability Study. JMIR Human Factors. 2019 Apr;6(2):e12469. 10.2196/12469 

15. Carroll AE. Averting Alert Fatigue to Prevent Adverse Drug Reactions. JAMA. 2019 Aug;322(7):601. 10.1001/jama.2019.11710 

16. Chou E, Boyce RD, Balkan B, Subbian V, Romero A, Hansten PD, et al. Designing and evaluating contextualized drug-drug interaction algorithms. JAMIA Open. 2021 Mar;4(1):ooab023. 10.1093/jamiaopen/ooab023 

17. Chien SC, Chen YL, Chien CH, Chin YP, Yoon CH, Chen CY, et al. Alerts in Clinical Decision Support Systems (CDSS): A Bibliometric Review and Content Analysis. Healthcare (Basel). 2022 Mar;10(4):601. 10.3390/healthcare10040601 

18.        Baysari MT, Zheng WY, Van Dort B, Reid-Anderson H, Gronski M, Kenny E. A Late Attempt to Involve End Users in the Design of Medication-Related Alerts: Survey Study. J Med Internet Res. 2020 03;22(3):e14855. 

19.        Van De Sijpe G, Quintens C, Walgraeve K, Van Laer E, Penny J, De Vlieger G, et al. Overall performance of a drug-drug interaction clinical decision support system: quantitative evaluation and end-user survey. BMC Med Inform Decis Mak. 2022 02 22;22(1):48. 10.1186/s12911-022-01783-z 

20. Johnson M, Sanchez P, Langdon R, Manias E, Levett-Jones T, Weidemann G, et al. The impact of interruptions on medication errors in hospitals: an observational study of nurses. J Nurs Manag. 2017 Oct;25(7):498–507. 10.1111/jonm.12486 

21. Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, et al. Array programming with NumPy. Nature. 2020 Sep;585(7825):357–62. 10.1038/s41586-020-2649-2 

22. McKinney W. Data structures for statistical computing in python. Proceedings of the 9th Python in Science Conference; 2010: Austin, TX; 2010. p. 51-6. 10.25080/Majora-92bf1922-00a 

23. Hunter JD. Matplotlib: A 2D Graphics Environment. Comput Sci Eng. 2007;9(3):90–5. 10.1109/MCSE.2007.55 

24.        Moja L, Polo Friz H, Capobussi M, Kwag K, Banzi R, Ruggiero F, et al. Effectiveness of a Hospital-Based Computerized Decision Support System on Clinician Recommendations and Patient Outcomes: A Randomized Clinical Trial. JAMA Netw Open. 2019 12;2(12):e1917094. 10.1001/jamanetworkopen.2019.17094 

25.        Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced Effectiveness of Interruptive Drug-Drug Interaction Alerts after Conversion to a Commercial Electronic Health Record. J Gen Intern Med. 2018 11;33(11):1868-76. 

26. Yoo J, Lee J, Rhee PL, Chang DK, Kang M, Choi JS, et al. Alert Override Patterns With a Medication Clinical Decision Support System in an Academic Emergency Department: Retrospective Descriptive Study. JMIR Med Inform. 2020 Nov;8(11):e23351. 10.2196/23351 

27.        Hussain MI, Reynolds TL, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc. 2019 10;26(10):1141-9. 

28. Villa Zapata L, Subbian V, Boyce RD, Hansten PD, Horn JR, Gephart SM, et al. Overriding Drug-Drug Interaction Alerts in Clinical Decision Support Systems: A Scoping Review. Stud Health Technol Inform. 2022 Jun;290:380–4. 10.3233/SHTI220101 

29. Skalafouris C, Reny JL, Stirnemann J, Grosgurin O, Eggimann F, Grauser D, et al. Development and assessment of PharmaCheck: an electronic screening tool for the prevention of twenty major adverse drug events. BMC Med Inform Decis Mak. 2022 May;22(1):146. 10.1186/s12911-022-01885-8 

30. Gaskin J, Conyard E. Clinical pharmacist interventions in the emergency department and their impact on preventable adverse drug events and associated cost avoidance. Eur J Hosp Pharm Sci Pract. 2017;24:A81. 

31. Cohen V, Jellinek SP, Hatch A, Motov S. Effect of clinical pharmacists on care in the emergency department: a systematic review. Am J Health Syst Pharm. 2009 Aug;66(15):1353–61. 10.2146/ajhp080304 

32. Drovandi A, Robertson K, Tucker M, Robinson N, Perks S, Kairuz T. A systematic review of clinical pharmacist interventions in paediatric hospital patients. Eur J Pediatr. 2018 Aug;177(8):1139–48. 10.1007/s00431-018-3187-x 

33. Richardson M, Dwyer CO, Gaskin J, Conyard E, Murphy K. The potential contribution of medicines to falls in older persons and the acceptance of pharmacist intervention. Int J Pharm Pract. 2020;28:57–8. 

34.        Gallo T, Curry SC, Padilla-Jones A, Heise CW, Ramos KS, Woosley RL, et al. A computerized scoring system to improve assessment of heparin-induced thrombocytopenia risk. J Thromb Haemost. 2019 02;17(2):383-8. 

35. Gallo T, Heise CW, Woosley RL, Tisdale JE, Antonescu CC, Gephart SM, et al. Clinician Satisfaction With Advanced Clinical Decision Support to Reduce the Risk of Torsades de Pointes. J Patient Saf. 2022 Sep;18(6):e1010–3. 10.1097/PTS.0000000000000996 

36. Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review. JMIR Med Inform. 2018 Jan;6(1):e3. 10.2196/medinform.7170 

Appendix 1

Appendix 2

The appendix 2 is available for download as a separate file at https://doi.org/10.57187/smw.2023.40082.