Clinical decision support systems

DOI: https://doi.org/10.4414/smw.2014.14073

Introduction

The volume, complexity and dynamics of clinical information are a challenge for physicians and other health professionals. Clinical decision support (CDS) systems help to tackle this challenge. A CDS system links patient data with a knowledge base to generate information and suggestions that help providers improve the health care they deliver [1–3]. A knowledge base is a database that stores knowledge in a suitable form depending on its use [4].

There are several definitions of CDS in the literature showing the wide range of functionality included in these systems. Shortliffe defined CDS in the following words: “A medical decision-support system is any computer programme designed to help health professionals make clinical decisions” [5]. According to Sim et al. CDS is defined as “software that is designed to be a direct aid to clinical decision-making, in which the characteristics of an individual patient are matched to a computerised clinical knowledge base and patient-specific assessments or recommendations are then presented to the clinician or the patient for a decision” [2].

Computerised physician order entry (CPOE) means that physicians order online – and it is linked with CDS. The combination of CPOE and CDS has proven to be very effective for reducing the frequency of potential adverse drug events [6].

Technically, CDS software incorporates the generic steps input, processing, and output: (i.) The patient-specific data are entered by health professionals involved in the care, (ii.) processed and linked to knowledge stored in a data base, and (iii.) notifications are communicated back to the clinicians [7].

CDS approaches pursuing various objectives have been developed and evaluated [8]. For instance, CDS successfully fostered the prevention of deep-vein thrombosis [9], improved the adherence to glucose regulation guidelines for critically ill patients [10], increased the use of vaccinations and other preventive measures [11], significantly decreased the rate of serious medication errors [6], and one system identified drug-induced thrombocytopenia in an intensive care unit [12].

CDS types and functions

Basic actions of CDS systems include alerting, reminding, critiquing (rejecting orders), interpreting, predicting, diagnosing, assisting and suggesting [13, 14].

Shortliffe distinguished three basic CDS types: (i.) CDS for information management to provide information on patients or medical knowledge, (ii.) focusing attention such as alerts, and (iii.) patient-specific consultation in order to receive recommendations and customised information on a single patient [5]. More than twenty years later Wright et al. [15] studied and summarised CDS taxonomies and categorised them into the following six types: (i.) “medication dosing support”, (ii.) “order facilitators”, (iii.) “point-of-care alerts/reminders”, (iv.) “relevant information display”, (v.) “expert systems”, and (vi.) “workflow support” (see table 1). Furthermore, these authors discriminated front-end and back-end capabilities in CDS. Front-end capabilities in CDS are described as “intervention types available to end-users created using specific clinical knowledge bases and application logic” such as drug-drug interaction checks, whereas back-end capabilities are summarised as “discrete system capabilities such as alert triggers, available data input elements, and end-user notification methods” [15].

Wright et al. [15] described “medication dosing support” as tools that support drug order decisions such as choosing a dose while adjusting it to the patient’s renal and hepatic functions. “Order facilitators” support the ordering physician in offering templates with order sets, such as for the diagnosis and treatment of myocardial infarctions or common daily drug orders. “Point-of-care alerts/reminders” may show a drug-drug interaction, remind the ordering physician of a due HbA_1c check in a diabetic patient or alert health staff of critical laboratory values. “Relevant information display” may show patient specific information such as showing potassium levels when ordering digoxin, displaying relevant information such as renal and hepatic function as well as body surface area when ordering chemotherapy and sporting cost of drugs and laboratory tests. “Expert systems” support physicians by offering complex decision support combining patient characteristics with other electronically available data. Examples are differential diagnostic suggestions, antibiotic therapy suggestions and prognostic models. “Workflow support” encompasses tools such as process templates such as for patients being transferred to the intensive care unit or medication reconciliation functions [15].

Impact of CDS

Excellent decisions due to CDS are expected to lead to higher patient safety with better treatment quality, less adverse events and reduced costs.

There are many different applications that support clinical physician decisions and a few have been found to improve outcomes. For instance, a recent study by Bourgeois et al. presented a CDS intervention for acute respiratory illnesses in children which reduced antibiotic use, although the net impact was modest [16]. Furthermore, CDS may improve patient safety and especially reduce serious medication errors [17]. This is of particular importance, since three out of four adverse drug events (ADEs) have been shown to be preventable, and ADEs are common in tertiary referral hospitals as well as in community hospitals [18]. A review by Wolfstadt et al. mentions that 50% of the identified studies showed that CPOE combined with CDS reduces the incidence of ADEs significantly [19]. Similar results were published by Kaushal et al. showing that CDS can reduce medication error rates substantially [20]. In a survey of six community hospitals in the greater Boston area, 9 out of 10 adverse drug events were preventable and all of these could potentially have been prevented with a CDS system including renal dose checking [21]. Furthermore, recommended therapies are being followed, if the information is delivered at the point of care and no physician response is needed [8].

There is good evidence that CDS improves performance around prevention [8]. As an example, CDS systems that alert physicians regarding the need for venous thromboembolism prophylaxis have proven to be very successful. The study by Kucher at al. showed a significant reduction of symptomatic and asymptomatic deep-vein thrombosis due to CDS alerts [9]. Others found that (i.) electronic thromboprophylaxis reminders have a sustained effect [22, 23], (ii.) reminders after both admissions and transfers have a significant impact on the awareness of thromboembolism prevention [24], and (iii.) the timing of CDS needs to be addressed in order to thoroughly evaluate such interventions [22, 24].

Alerts should provide knowledge gain without too many workflow interruptions. Otherwise they may cause “alert fatigue” (cf. next chapter),regarding drug-drug interactions for example [25], without reaching any patient safety goals [26]. Wipfli and Lovis discussed points to consider designing and installing alerts [27].

Some aspects of CDS impact need additional assessment. There is still insufficient data on the influence of CDS on management of patients with multiple chronic diseases, clinician workload, length of stay of hospitalised patients, mortality and economic impact, warranting further studies in these and other areas [8, 28].

Potential harms

The Institute of Medicine’s book “To Err is Human” [29] advocated the use of CPOE in combination with CDS to increase patient safety. However, it is clear that health information technology can also create new safety issues and some have expressed doubts about whether CDS improves patient outcomes [30–32] – we believe that it does, but its impact varies substantially by content domain, and it does not necessarily produce improvements in any specific area.

Koppel et al. [33] identified and observed 22 situations where CPOE facilitated medication errors, such as misleading display of dose ranges. As a consequence the authors suggested to (i.) emphasise workflow more than CPOE in reducing medication risks, (ii.) evaluate the implemented CPOE system, (iii.) fix CPOE problems as soon as possible, (iv.) consider and investigate various causes of medication errors, and (v.) progressively improve quality while “recognising that all changes generate new error risks” [33].

An article published by Han et al. [34] reported a significant increase in the mortality of children after a commercial CPOE system had been implemented (adjusted odds ratio 3.28). In essence, the authors explained the increased mortality by delayed administration of critical medications due to mal-performance of the CPOE system – which apparently required an additional physician solely entering orders, led to complicated communication between health professionals, and decreased the time of physicians and nurses at the bedside [34].

That publication raised a number of comments: Some readers acknowledged the importance of this contribution to the literature in the field, but also, they recommended to carefully plan CPOE and CDS implementations as well as thoroughly examine systems and functions only in pilot wards, before stepwise introducing them [31, 35, 36]. One group highlighted the reported policy changes (removal of crucial drugs from location of care, interdiction of prescribing before patient’s arrival, pharmacy received orders not before approval by a nurse) contributing to critical administration delays, which were independent of the technology [37]. Changing the policies during or shortly before a CPOE implementation process should be avoided [36]. However, the reportedly inadequate performance of the network and CPOE system, and the missed preparation of electronic order sets have been criticised [35, 37]. The latter was one of the most serious omissions in the view of Jacobs et al. (38), and this group also emphasised that – as long as the pharmacist accessed the electronic patient chart – other health professionals were “locked out,” [34] in that they were unable to enter additional orders.

Furthermore, some commentators highlighted that the implementation of computerisation in the clinical setting is complex and inevitably changes the workflows [35, 36]. However, it is crucial that health professionals are always able to care for the patient, including the possibility of using paper orders [35, 36]. It has been stated that it is unreasonable to implement CPOE in only six days [31, 35] as described [34]. In contrast, such implementations have been estimated to require from 1 to 3 years [36, 39]. Finally, Han et al. [40], among many others including the Institute of Medicine, suggested establishing an independent board that ensures the safety of health information technology (HIT), similar to safety arrangements in the aviation industry [36], and in fact, such a group is now being set up in the U.S.

Another group [41] observed and interviewed health professionals of five U.S. hospitals and identified a large number of issues due to CPOE and particularly CDS, including additional workload and too many false positive alerts. As a consequence, Weiner et al. [42] introduced the term “e-iatrogenesis”.

False positive alerts increase the risk of alert fatigue [25]. This psychological phenomenon results from high numbers of clinically insignificant alerts that consume time and mental energy, potentially causing overrides of highly important notifications [43]. However, by increasing the specificity of electronic notifications, the risk of alert fatigue will be minimised. For instance, Hulgan et al. [44] presented a sophisticated algorithm that detected drip-fed patients – before electronically suggesting the oral route for intravenous quinolones.

Strom et al. [45] evaluated interruptive alerts in a randomised controlled trial: The aim of the CDS intervention was to avoid a potentially harmful drug-drug interaction between warfarin and trimethoprim-sulfamethoxazole. It was possible to override these hard-stop alerts (i) by entering the indication would be Pneumocystis carinii pneumonia or (ii) by calling the pharmacy. On the one hand, the intervention had a significant impact on prescription behaviour, but on the other hand, the institutional review board terminated the study early because of inacceptable treatment delays in four patients due to the interruptive alerts. A high proportion of all known drug interactions are of low clinical significance [43]. Not so the interaction between warfarin and trimethoprim-sulfamethoxazole, which increases the bleeding risk substantially [46]. Strom et al. [45] implemented a strong CDS intervention, though, regardless of whether the patients were already stabilised on warfarin (adding trimethoprim-sulfamethoxazole increases bleeding risk), or whether they already received trimethoprim-sulfamethoxazole and warfarin administration was initiated (little risk for bleeding) [47]. Horn concluded that “a reminder to appropriately monitor the patient’s INR would have mitigated the risk of the interaction while avoiding the unintended consequences” [47]. However, a hard-stop alert may not be the best option to avoid the outlined drug-drug interaction, because there are patients who urgently need both drugs – whereas absolute contraindications (e.g. isotretinoin during pregnancy) warrant strong interventions [48].

Implementation of CDS: points to consider

Computerised CDS has been investigated for several decades [49, 50]. The trend away from the “Greek oracle” approach of diagnostic decision support systems to a more supporting kind of software was accompanied by improving hardware and philosophical changes regarding the human-computer interaction (the physician outgrew the role of a passive observer) [50]. More recently, it has been realised that the way CDS is implemented and presented is critical for a system to succeed (see table 2), and that time consumption is a major factor determining failure [51].

Kawamoto et al. [52] analysed publications on electronic and non-electronic decision support systems and presented four CDS features that contributed to improved clinical practice: (i) “provide decision support automatically as part of clinician workflow”, (ii) “deliver decision support at the time and location of decision making”, (iii) “provide actionable recommendations”, and (iv) “use a computer to generate the decision support”.

Notifications should be specific and fit the context when displayed, in order to minimise overriding [25]. However, besides patient-specificity, physician-related factors should also be considered while designing interventions [53], and the understanding of human factors have been recognised to be important to the acceptance of CDS [54]. Software needs to be tailored to suit the users of different target groups among health professionals [55].

There is a need for governance when implementing and employing CDS systems. Wright et al. [56] offer a comprehensive summary with six recommendations outlined in table 3. It is important to realise the impact CDS may have on clinical practice as well as surrounding clinical expert systems. Accordingly, a governance concept followed up on by a group of specialists in clinical as well as technical aspects is warranted. Therefore, Eppenga et al. suggest evaluating CDS systems repeatedly, in order to review, adjust and improve algorithms [57].

Outlook and future research

O’Connor et al. [58] published an overview and outlook of CDS in diabetes care, and some points discussed in this article are generalisable to other future CDS developments. For instance, CDS should suggest therapeutic options whenever it warns against problems, individual customisation of CDS will play an increasing role, CDS should be designed to save time, and CDS is moving toward patient-centred decision support [58]. In fact, CDS directed not only to health professionals but also to the patient seems to be a powerful approach as concluded in two recent meta-analyses [59, 60]. This is an important result that needs to be considered while developing novel CDS interventions.

Furthermore, O’Connor et al. hope that – within the next 10 years – the compatibility issues across different electronic health records will be solved, CDS implementations will provide “validated transparent clinical algorithms”, CDS will be able to prioritise recommendations, and genetic markers will be part of the input data considered by the systems [58].

Features identified to be crucial in successful CDS interventions are “provide decision support automatically as part of clinician workflow”, “deliver decision support at the time and location of decision making”, “provide actionable recommendations”, “use a computer to generate the decision support” [52], and “integration with charting or order entry system, promotion of action rather than inaction, no need for additional clinician data entry, justification of decision support via research evidence, local user involvement, and provision of decision support results to patients as well as providers” [59]. Studies are needed to prospectively investigate the individual role and importance of these features one by one [59].

It has repeatedly been stated [30–32, 58, 59, 61] that study results, regarding whether CDS actually improves clinical outcomes, are still scarce. The limited power of available studies is an important reason for this knowledge gap [24, 48]. Murphy [61] proposed cohort studies as a means to investigate the long-term outcomes of CDS on morbidity and mortality. Quality measures and financial incentives might additionally boost the meaningful implementation and effectiveness of CDS [61].

It is likely that knowledge implemented in CDS tools will be shared more widely in the future, by uploading knowledge to repositories, by disclosing the specifications of algorithms, and by sharing executable modules (cf. www.opencds.org) [48, 62]. The knowledge management of CDS is a novel research field with great promise [63]. It includes the use of specialised management software, considers online tools and online collaborations, the dissemination of CDS knowledge, and knowledge generation by data mining.

Hongsermeier et al. [64] mention the need for protection of intellectual property invested in CDS in order to foster sharing of CDS knowledge. However, it is difficult to patent intellectual property of the knowledge integrated in CDS. Additionally another aspect is that – while health care providers are considered to be the “learned intermediaries” deciding to follow CDS recommendations or not – the liability principles in terms of CDS and its developers are still not well established [65]. Finally, the policies on premarket review and regulations on CDS are – if not obscure – in progress [66]. A comparison of the CDS regulations in the U.S. and the E.U. has been published by Andersson [67].

Implementation of customisation regarding alerts for providers and regarding information for patients is complex [68], however, the user profiles of the target groups should be extended, for example personal options such as ‘‘remind me in one week’’ and ‘‘don’t show this message again’’ could play a role [55]. Researchers should always consider the impact of both the back-end (algorithms and knowledge base) and front-end (user interface) of CDS interventions [51]. Sophisticated algorithms allow for increasing the specificity of notifications [44, 69, 70], and in turn, decrease the risk for alert fatigue [25]. In addition, the design of the human-computer interface should consider human factors principles, and the extent to which it does so will affect performance [54, 62, 71].

CDS systems – whether commercial or home-grown – need to be carefully addressed in an on-going way. There is no doubt that CDS specialists will increasingly be a wanted human resource [60]. Furthermore, the more patient-specific and clinically significant the content of CDS notifications is, the more users will accept and value computerised recommendations. CDS works best when providers are led to make it easy to do the right thing, rather than through approaches in which – after a lot of ordering work is done – critiques are provided at the end. Overall, much of the benefit which is realised from electronic health records comes from the embedded CDS, and it is essential to do a good job with it.

References

  1 Wyatt J, Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. Proceedings / the Annual Symposium on Computer Application [sic] in Medical Care Symposium on Computer Applications in Medical Care. 1991:3–7.

  2 Sim I, Gorman P, Greenes RA, et al. Clinical decision support systems for the practice of evidence-based medicine. Journal of the American Medical Informatics Association: JAMIA. 2001;8(6):527–34.

  3 Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. Journal of the American Medical Informatics Association: JAMIA. 2007;14(2):141–5.

  4 Mylopoulos J, Chaudhri V, Plexousakis D, Shrufi A, Topologlou T. Building knowledge base management systems. The VLDB Journal. 1996;5:238–63.

  5 Shortliffe EH. Computer programs to support clinical decision making. JAMA: the journal of the American Medical Association. 1987;258(1):61–6.

  6 Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA: the journal of the American Medical Association. 1998;280(15):1311–6.

  7 Berner ES. Clinical Decision Support Systems: State of the Art. http://healthit.ahrq.gov/sites/default/files/docs/page/09–0069–EF_1.pdf, accessed 4 May 2014.

  8 Bright TJ, Wong A, Dhurjati R, et al. Effect of Clinical Decision-Support SystemsA Systematic Review. Ann Intern Med. 2012;157(1):29–43.

  9 Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352(10):969–77.

10 Rood E, Bosman RJ, van der Spoel JI, Taylor P, Zandstra DF. Use of a computerized guideline for glucose regulation in the intensive care unit improved both guideline adherence and glucose regulation. Journal of the American Medical Informatics Association: JAMIA. 2005;12(2):172–80.

11 Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001;345(13):965–70.

12 Harinstein LM, Kane-Gill SL, Smithburger PL, Culley CM, Reddy VK, Seybert AL. Use of an abnormal laboratory value-drug combination alert to detect drug-induced thrombocytopenia in critically Ill patients. J Crit Care. 2012;27(3):242–9.

13 Randolph AG, Haynes RB, Wyatt JC, Cook DJ, Guyatt GH. Users’ Guides to the Medical Literature: XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. JAMA: the journal of the American Medical Association. 1999;282(1):67–74.

14 Pryor TA. Development of decision support systems. Int J Clin Monit Comput. 1990;7(3):137–46.

15 Wright A, Sittig DF, Ash JS, et al. Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. Journal of the American Medical Informatics Association: JAMIA. 2011;18(3):232–42.

16 Bourgeois FC, Linder J, Johnson SA, Co JP, Fiskio J, Ferris TG. Impact of a computerized template on antibiotic prescribing for acute respiratory infections in children and adolescents. Clinical pediatrics. 2010;49(10):976–83.

17 Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med. 2003;348(25):2526–34.

18 Hug BL, Witkowski DJ, Sox CM, et al. Adverse Drug Event Rates in Six Community Hospitals and the Potential Impact of Computerized Physician Order Entry for Prevention. J Gen Intern Med. 2009 Nov 6.

19 Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23(4):451–8.

20 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163(12):1409–16.

21 Hug BL, Witkowski DJ, Sox CM, et al. Occurrence of adverse, often preventable, events in community hospitals involving nephrotoxic drugs or those excreted by the kidney. Kidney Int. 2009;76(11):1192–8.

22 Beeler PE, Kucher N, Blaser J. Sustained impact of electronic alerts on rate of prophylaxis against venous thromboembolism. Thromb Haemost. 2011;106(4):734–8.

23 Lecumberri R, Marques M, Diaz-Navarlaz MT, et al. Maintained effectiveness of an electronic alert system to prevent venous thromboembolism among hospitalized patients. Thromb Haemost. 2008;100(4):699–704.

24 Beeler PE, Eschmann E, Schumacher A, Studt JD, Amann-Vesti B, Blaser J. Impact of electronic reminders on venous thromboprophylaxis after admissions and transfers. J Am Med Inform Assoc. 2014 Mar 26.

25 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. Journal of the American Medical Informatics Association: JAMIA. 2006;13(2):138–47.

26 Phansalkar S, van der Sijs H, Tucker AD, et al. Drug–drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2012 September 25, 2012.

27 Wipfli R, Lovis C. Alerts in clinical information systems: building frameworks and prototypes. Stud Health Technol Inform. 2010;155:163–9.

28 Payne TH, Bates DW, Berner ES, et al. Healthcare information technology and economics. J Am Med Inform Assoc. 2012 July 10, 2012.

29 Kohn L. To err is human: an interview with the Institute of Medicine’s Linda Kohn. Jt Comm J Qual Improv. 2000;26(4):227–34.

30 Berger RG, Kichak JP. Computerized physician order entry: helpful or harmful? Journal of the American Medical Informatics Association: JAMIA. 2004;11(2):100–3.

31 Gesteland PH, Nebeker JR, Gardner RM. These are the technologies that try men's souls: common-sense health information technology. Pediatrics. 2006;117(1):216–7.

32 Hemens BJ, Holbrook A, Tonkin M, et al. Computerized clinical decision support systems for drug prescribing and management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011;6:89.

33 Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005;293(10):1197–203.

34 Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics. 2005;116(6):1506–12.

35 Rosenbloom ST, Harrell FE, Jr., Lehmann CU, Schneider JH, Spooner SA, Johnson KB. Perceived increase in mortality after process and policy changes implemented with computerized physician order entry. Pediatrics. 2006;117(4):1452–5; author reply 5–6.

36 Sittig DF, Ash JS, Zhang J, Osheroff JA, Shabot MM. Lessons from “Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system”. Pediatrics. 2006;118(2):797–801.

37 Longhurst C, Sharek P, Hahn J, Sullivan J, Classen D. Perceived increase in mortality after process and policy changes implemented with computerized physician order entry. Pediatrics. 2006;117(4):1450–1; author reply 5–6.

38 Jacobs BR, Brilli RJ, Hart KW. Perceived increase in mortality after process and policy changes implemented with computerized physician order entry. Pediatrics. 2006;117(4):1451–2; author reply 5–6.

39 Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. Journal of the American Medical Informatics Association: JAMIA. 2003;10(4):322–9.

40 Han YY, Carcillo JA, Venkataraman ST, et al. In Reply. Pediatrics. 2006;117(4):1455–6.

41 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. Journal of the American Medical Informatics Association: JAMIA. 2006;13(5):547–56.

42 Weiner JP, Kfuri T, Chan K, Fowles JB. “e-Iatrogenesis”: the most critical unintended consequence of CPOE and other HIT. Journal of the American Medical Informatics Association: JAMIA. 2007;14(3):387–8; discussion 9.

43 Peterson JF, Bates DW. Preventable medication errors: identifying and eliminating serious drug interactions. J Am Pharm Assoc (Wash). 2001;41(2):159–60.

44 Hulgan T, Rosenbloom ST, Hargrove F, et al. Oral quinolones in hospitalized patients: an evaluation of a computerized decision support intervention. J Intern Med. 2004;256(4):349–57.

45 Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170(17):1578–83.

46 Glasheen JJ, Fugit RV, Prochazka AV. The risk of overanticoagulation with antibiotic use in outpatients on stable warfarin regimens. J Gen Intern Med. 2005;20(7):653–6.

47 Horn JR. Hard-stops for drug interactions. Arch Intern Med. 2011;171(7):706; author reply -7.

48 Bates DW. CPOE and clinical decision support in hospitals: getting the benefits: comment on “Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction”. Arch Intern Med. 2010;170(17):1583–4.

49 Morrell J, Podlone M, Cohen SN. Receptivity of physicians in a teaching hospital to computerized drug interaction monitoring and reporting system. Med Care. 1977;15(1):68–78.

50 Miller RA. Medical diagnostic decision support systems – past, present, and future: a threaded bibliography and brief commentary. Journal of the American Medical Informatics Association: JAMIA. 1994;1(1):8–27.

51 Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. Journal of the American Medical Informatics Association: JAMIA. 2003;10(6):523–30.

52 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765.

53 Eschmann E, Beeler PE, Kaplan V, Schneemann M, Zund G, Blaser J. Patient- and physician-related risk factors for hyperkalaemia in potassium-increasing drug-drug interactions. Eur J Clin Pharmacol. 2014;70(2):215–23.

54 Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. Journal of the American Medical Informatics Association: JAMIA. 2014 Apr 29.

55 Beeler PE, Eschmann E, Rosen C, Blaser J. Use of an on-demand drug-drug interaction checker by prescribers and consultants: a retrospective analysis in a Swiss teaching hospital. Drug safety: an international journal of medical toxicology and drug experience. 2013;36(6):427–34.

56 Wright A, Sittig DF, Ash JS, et al. Governance for clinical decision support: case studies and recommended practices from leading institutions. J Am Med Inform Assoc. 2011;18(2):187–94.

57 Eppenga WL, Derijks HJ, Conemans JMH, Hermens WAJJ, Wensing M, De Smet PAGM. Comparison of a basic and an advanced pharmacotherapy-related clinical decision support system in a hospital care setting in the Netherlands. J Am Med Inform Assoc. 2012;19(1):66–71.

58 O’Connor PJ, Desai JR, Butler JC, Kharbanda EO, Sperl-Hillen JM. Current status and future prospects for electronic point-of-care clinical decision support in diabetes care. Curr Diab Rep. 2013;13(2):172–6.

59 Lobach D, Sanders GD, Bright TJ, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evidence report/technology assessment. 2012(203):1–784.

60 Roshanov PS, Fernandes N, Wilczynski JM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657.

61 Murphy EV. Clinical Decision Support: Effectiveness in Improving Quality Processes and Clinical Outcomes and Factors That May Influence Success. Yale J Biol Med. 2014;87(2):187–97.

62 Sittig DF, Wright A, Osheroff JA, et al. Grand challenges in clinical decision support. J Biomed Inform. 2008;41(2):387–92.

63 Sittig DF, Wright A, Simonaitis L, et al. The state of the art in clinical knowledge management: an inventory of tools and techniques. Int J Med Inform. 2010;79(1):44–57.

64 Hongsermeier T, Maviglia S, Tsurikova L, et al. A legal framework to enable sharing of Clinical Decision Support knowledge and services across institutional boundaries. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2011;2011:925–33.

65 Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce “alert fatigue” while still minimizing the risk of litigation. Health affairs. 2011;30(12):2310–7.

66 Thompson BM, Brooke MJ. FDA's approach to clinical decision support software: A brief summary. http://www.fiercehealthit.com/special-reports/fdas-approach-clinical-decision-support-software-brief-summary, accessed 28 August 2014.

67 Andersson C. Comparing EU and US Approaches to Regulating Clinical Decision Support Software. http://www.emdt.co.uk/article/comparing-eu-and-us-approaches-regulating-clinical-decision-support-software-brief-summary-e, accessed 28 August 2014.

68 Bottiger Y, Laine K, Andersson ML, et al. SFINX-a drug-drug interaction database designed for clinical decision support systems. Eur J Clin Pharmacol. 2009;65(6):627–33.

69 Seidling HM, Klein U, Schaier M, et al. What, if all alerts were specific – estimating the potential impact on drug interaction alert burden. Int J Med Inform. 2014;83(4):285–91.

70 Seidling HM, Schmitt SP, Bruckner T, et al. Patient-specific electronic decision support reduces prescription of excessive doses. Quality & safety in health care. 2010;19(5):e15.

71 Coleman JJ, van der Sijs H, Haefeli WE, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC medical informatics and decision making. 2013;13:111.