Tackling alert fatigue with a semi-automated clinical decision support system: quantitative evaluation and end-user survey
STUDY AIMS: Clinical decision support systems (CDSS) embedded in hospital electronic health records efficiently reduce medication errors, but there is a risk of low physician adherence due to alert fatigue. At the Cantonal Hospital Aarau, a CDSS is being developed that allows the highly accurate detection and correction of medication errors. The semi-automated CDSS sends its alerts either directly to the physician or to a clinical pharmacist for review first. Our aim was to evaluate the performance of the recently implemented CDSS in terms of acceptance rate and alert burden, as well as physicians’ satisfaction with the CDSS.
METHODS: All alerts generated by the clinical decision support systems between January and December 2021 were included in a retrospective quantitative evaluation. A team of clinical pharmacists performed a follow-up to determine whether the recommendation made by the CDSS was implemented by the physician. The acceptance rate was calculated including all alerts for which it was possible to determine an outcome. A web-based survey was conducted amongst physicians to assess their attitude towards the CDSS. The survey questions included overall satisfaction, helpfulness of individual algorithms, and perceived alert burden.
RESULTS: In 2021, a total of 10,556 alerts were generated, of which 619 triggered a direct notification to the physician and 2,231 notifications were send to the physician after evaluation by a clinical pharmacist. The acceptance rates were 89.8% and 68.4%, respectively, which translates as an overall acceptance rate of 72.4%. On average, clinical pharmacists received 17.2 alerts per day, while all of the hospital physicians together received 7.8 notifications per day. In the survey, 94.5% of physicians reported being satisfied or very satisfied with the CDSS. Algorithms addressing potential medication errors concerning anticoagulants received the highest usefulness ratings.
CONCLUSION: The development of this semi-automated clinical decision support system with context-based algorithms resulted in alerts with a high acceptance rate. Involving clinical pharmacists proved a promising approach to limit the alert burden of physicians and thus tackle alert fatigue. The CDSS is well accepted by our physicians.
- Aronson JK. Medication errors: definitions and classification. Br J Clin Pharmacol. 2009 Jun;67(6):599–604. 10.1111/j.1365-2125.2009.03415.x
- Krähenbühl-Melcher A, Schlienger R, Lampert M, Haschke M, Drewe J, Krähenbühl S. Drug-related problems in hospitals: a review of the recent literature. Drug Saf. 2007;30(5):379–407. 10.2165/00002018-200730050-00003
- Nanji KC, Patel A, Shaikh S, Seger DL, Bates DW. Evaluation of Perioperative Medication Errors and Adverse Drug Events. Anesthesiology. 2016 Jan;124(1):25–34. 10.1097/ALN.0000000000000904
- Westbrook JI, Sunderland NS, Woods A, Raban MZ, Gates P, Li L. Changes in medication administration error rates associated with the introduction of electronic medication systems in hospitals: a multisite controlled before and after study. BMJ Health Care Inform. 2020 Aug;27(3):e100170. 10.1136/bmjhci-2020-100170
- Convertino I, Salvadori S, Pecori A, Galiulo MT, Ferraro S, Parrilli M, et al. Potential Direct Costs of Adverse Drug Events and Possible Cost Savings Achievable by their Prevention in Tuscany, Italy: A Model-Based Analysis. Drug Saf. 2019 03;42(3):427-44.
- Muylle KM, Gentens K, Dupont AG, Cornu P. Evaluation of an optimized context-aware clinical decision support system for drug-drug interaction screening. Int J Med Inform. 2021 Apr;148:104393. 10.1016/j.ijmedinf.2021.104393
- Hardmeier B, Braunschweig S, Cavallaro M, Roos M, Pauli-Magnus C, Giger M, et al. Adverse drug events caused by medication errors in medical inpatients. Swiss Med Wkly. 2004 Nov;134(45-46):664–70. 10.4414/smw.2004.10801
- Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003 Jun;163(12):1409–16. 10.1001/archinte.163.12.1409
- Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020 Feb;3(1):17. 10.1038/s41746-020-0221-y
- Cash JJ. Alert fatigue. Am J Health Syst Pharm. 2009 Dec;66(23):2098–101. 10.2146/ajhp090181
- Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R, et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017 04;17(1):36. 10.1186/s12911-017-0430-8
- Poly TN, Islam MM, Yang HC, Li YJ. Appropriateness of Overridden Alerts in Computerized Physician Order Entry: systematic Review. JMIR Med Inform. 2020 Jul;8(7):e15653. 10.2196/15653
- Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc. 2018 05 01;25(5):476-81.
- Blecker S, Pandya R, Stork S, Mann D, Kuperman G, Shelley D, et al. Interruptive Versus Noninterruptive Clinical Decision Support: usability Study. JMIR Human Factors. 2019 Apr;6(2):e12469. 10.2196/12469
- Carroll AE. Averting Alert Fatigue to Prevent Adverse Drug Reactions. JAMA. 2019 Aug;322(7):601. 10.1001/jama.2019.11710
- Chou E, Boyce RD, Balkan B, Subbian V, Romero A, Hansten PD, et al. Designing and evaluating contextualized drug-drug interaction algorithms. JAMIA Open. 2021 Mar;4(1):ooab023. 10.1093/jamiaopen/ooab023
- Chien SC, Chen YL, Chien CH, Chin YP, Yoon CH, Chen CY, et al. Alerts in Clinical Decision Support Systems (CDSS): A Bibliometric Review and Content Analysis. Healthcare (Basel). 2022 Mar;10(4):601. 10.3390/healthcare10040601
- Baysari MT, Zheng WY, Van Dort B, Reid-Anderson H, Gronski M, Kenny E. A Late Attempt to Involve End Users in the Design of Medication-Related Alerts: Survey Study. J Med Internet Res. 2020 03;22(3):e14855.
- Van De Sijpe G, Quintens C, Walgraeve K, Van Laer E, Penny J, De Vlieger G, et al. Overall performance of a drug-drug interaction clinical decision support system: quantitative evaluation and end-user survey. BMC Med Inform Decis Mak. 2022 02 22;22(1):48. 10.1186/s12911-022-01783-z
- Johnson M, Sanchez P, Langdon R, Manias E, Levett-Jones T, Weidemann G, et al. The impact of interruptions on medication errors in hospitals: an observational study of nurses. J Nurs Manag. 2017 Oct;25(7):498–507. 10.1111/jonm.12486
- Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, et al. Array programming with NumPy. Nature. 2020 Sep;585(7825):357–62. 10.1038/s41586-020-2649-2
- McKinney W. Data structures for statistical computing in python. Proceedings of the 9th Python in Science Conference; 2010: Austin, TX; 2010. p. 51-6. 10.25080/Majora-92bf1922-00a
- Hunter JD. Matplotlib: A 2D Graphics Environment. Comput Sci Eng. 2007;9(3):90–5. 10.1109/MCSE.2007.55
- Moja L, Polo Friz H, Capobussi M, Kwag K, Banzi R, Ruggiero F, et al. Effectiveness of a Hospital-Based Computerized Decision Support System on Clinician Recommendations and Patient Outcomes: A Randomized Clinical Trial. JAMA Netw Open. 2019 12;2(12):e1917094. 10.1001/jamanetworkopen.2019.17094
- Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced Effectiveness of Interruptive Drug-Drug Interaction Alerts after Conversion to a Commercial Electronic Health Record. J Gen Intern Med. 2018 11;33(11):1868-76.
- Yoo J, Lee J, Rhee PL, Chang DK, Kang M, Choi JS, et al. Alert Override Patterns With a Medication Clinical Decision Support System in an Academic Emergency Department: Retrospective Descriptive Study. JMIR Med Inform. 2020 Nov;8(11):e23351. 10.2196/23351
- Hussain MI, Reynolds TL, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc. 2019 10;26(10):1141-9.
- Villa Zapata L, Subbian V, Boyce RD, Hansten PD, Horn JR, Gephart SM, et al. Overriding Drug-Drug Interaction Alerts in Clinical Decision Support Systems: A Scoping Review. Stud Health Technol Inform. 2022 Jun;290:380–4. 10.3233/SHTI220101
- Skalafouris C, Reny JL, Stirnemann J, Grosgurin O, Eggimann F, Grauser D, et al. Development and assessment of PharmaCheck: an electronic screening tool for the prevention of twenty major adverse drug events. BMC Med Inform Decis Mak. 2022 May;22(1):146. 10.1186/s12911-022-01885-8
- Gaskin J, Conyard E. Clinical pharmacist interventions in the emergency department and their impact on preventable adverse drug events and associated cost avoidance. Eur J Hosp Pharm Sci Pract. 2017;24:A81.
- Cohen V, Jellinek SP, Hatch A, Motov S. Effect of clinical pharmacists on care in the emergency department: a systematic review. Am J Health Syst Pharm. 2009 Aug;66(15):1353–61. 10.2146/ajhp080304
- Drovandi A, Robertson K, Tucker M, Robinson N, Perks S, Kairuz T. A systematic review of clinical pharmacist interventions in paediatric hospital patients. Eur J Pediatr. 2018 Aug;177(8):1139–48. 10.1007/s00431-018-3187-x
- Richardson M, Dwyer CO, Gaskin J, Conyard E, Murphy K. The potential contribution of medicines to falls in older persons and the acceptance of pharmacist intervention. Int J Pharm Pract. 2020;28:57–8.
- Gallo T, Curry SC, Padilla-Jones A, Heise CW, Ramos KS, Woosley RL, et al. A computerized scoring system to improve assessment of heparin-induced thrombocytopenia risk. J Thromb Haemost. 2019 02;17(2):383-8.
- Gallo T, Heise CW, Woosley RL, Tisdale JE, Antonescu CC, Gephart SM, et al. Clinician Satisfaction With Advanced Clinical Decision Support to Reduce the Risk of Torsades de Pointes. J Patient Saf. 2022 Sep;18(6):e1010–3. 10.1097/PTS.0000000000000996
- Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review. JMIR Med Inform. 2018 Jan;6(1):e3. 10.2196/medinform.7170