Screening for delirium with the Intensive Care Delirium Screening Checklist (ICDSC): a re-evaluation of the threshold for delirium

DOI: https://doi.org/10.4414/smw.2018.14597

Introduction

Delirium is characterised as a neuropsychiatric syndrome with an abrupt onset and fluctuating disturbances in consciousness and cognition, as well as a range of noncognitive domains including disturbances in motor behaviour, emotionality and sleep-wake cycle, caused by an underlying aetiology [1, 2].

Among psychiatric syndromes, delirium is the most common across various healthcare settings [3, 4]. Up to 70% of cardiosurgical patients develop this syndrome [5, 6], and in mechanically ventilated patients, the rates of delirium reach 80% [7]. Delirium has been recognised to have adverse short-term [8, 9] and long-term consequences for patients and the healthcare system [10]. These include a prolonged stay on the intensive care unit (ICU) [11, 12], higher rates and prolongation of mechanical ventilation [13], increased morbidities and mortality [13, 14] and, as a long term-consequence, functional impairment and cognitive disabilities [15] requiring institutionalisation [12].

Several instruments have been developed to improve the screening for and detection of delirium. In the intensive care setting, one of the most commonly used screening instruments is the Intensive Care Delirium Screening Checklist (ICDSC) [16].

From a review [17], the ICDCS was evaluated in four studies including 59 to 126 patients. Delirium rates ranged from 16 to 39%, and the sensitivity and specificity ranged from 43 to 96% and 73 to 95%, respectively. A meta-analysis indicated sensitivity and specificity of 74 and 81.9%, respectively, and, overall, the accuracy was considered good [17]. Conversely, other studies indicated lower sensitivities (43 to 47%) with high specificity (>94%) [18, 19].

Among the subtypes of delirium, lower rates were documented in the correct detection of the hypoactive subtype (42.9 vs 32%) [18]. Only one study evaluated the symptom profile of delirium with the ICDSC, and the most frequent symptoms were inattention, disorientation and psychomotor agitation [20].

Similarly, discrepancies in the concurrent validity between the ICDSC and Confusion Assessment Method for the ICU (CAM-ICU) have been reported, with Cohen’s κ ranging from 0.59 to 0.80 [21, 22]. However, the ICDSC was considered inferior to the CAM-ICU with respect to sensitivity and specificity [17]. It has been debated whether changing the cut-off score for the ICDSC could improve its accuracy: namely, decreasing this score from ≥4 to ≥3 [23] or, conversely, increasing this score to ≥5 and defining subsyndromal delirium at ≥3 [21]. With a threshold of ≥3, the sensitivity increased from 75 to 90%; however, the specificity also decreased from 74.3 to 61.5%. In an attempt to exclude subsyndromal delirium – defined as an ISCDSC score ≥3 – by increasing the threshold to ≥5, the specificity was increased from 72.4 to 86.5% [23].

Thus, although the ICDSC has been commonly accepted as an appropriate screening tool for delirium in the intensive care setting, inconsistencies, in particular with respect to its optimal cut-off score [18, 19, 24], remain. From previous results, the CAM-ICU detected only half the cases of delirium, whereas the ICDSC was able to detect two out of three patients with delirium [25]. Thus for a screening instrument, the ICDSC underperformed. Therefore, the aim of this study was to evaluate the concurrent validity, sensitivity and specificity, as well the positive and negative prediction of the ICDSC threshold set at a lower score of ≥3, the current standard ≥4, and an increased score of ≥5 versus a diagnosis of delirium determined with the Diagnostic and Statistical Manual, 4th edition, text revision (DSM-IV-TR), which is considered the gold standard for diagnosing delirium. Further, we evaluated whether a modified ICDSC cut-off score increased its agreement with the CAM-ICU.

Methods

Patients

All patients in this prospective, descriptive cohort and diagnostic accuracy study were recruited on a cardiovascular-surgical 12-bed intensive care unit at the University Hospital Zurich between May 2013 and April 2015. Inclusion criteria were being adult, ability to consent and intensive care management for more than 18 hours. Exclusion criteria were the inability to consent, or past or present substance use disorder.

Procedures

All patients in this study were informed of the procedures and written informed consent was obtained. For those patients unable to provide consent at the initial attempt, due either to more severe delirium, their medical condition and sedation, or frailty, proxy assent from the next of kin or a responsible caregiver was obtained instead. After medical stabilisation, consent was obtained, or when participation and consent were refused, the patient was excluded.

Four raters trained in the application of the DSM-IV-TR criteria assessed delirium. In the training process, cases were discussed in order to achieve agreement. Then, for the purpose of the study, the majority of patients was assessed by one of the raters individually; to assess inter-rater reliability, 28 patients were assessed by all psychiatrists, who were unaware of each other’s assessment. In this instance, the assessments included in the study derived from the primary, testing psychiatrist; the assessments from the secondary psychiatrists were omitted from the dataset for the study.

The baseline assessment included an interview with the patient, determination of the presence of delirium according to the DSM-IV-TR criteria [1], and then administration of the ICDSC [16] and CAM-ICU [26] by nurses and doctors specifically trained in their use. Both nurses and patients were blinded to the psychiatric assessment, and records were kept separately. The ICDSC was administered regularly every shift, i.e., every eight hours, whereas the CAM-ICU was performed either when delirium was diagnosed on the basis of an ICDSC score of ≥4, or on notification of inclusion in the study. In addition, for the evaluation of the motor subtype of delirium, the respective motor items from the Delirium Rating Scale-Revised-1998 (DRS-R-98; items 7 and 8) were recorded. The assessment was completed by retrieving collateral information from nursing and medical-surgical staff, the electronic medical record system (Klinikinformationssystem, KISIM, CisTec AG, Zurich), and family or caregivers.

Measurements

Diagnostic and Statistical Manual-IV-TR

The diagnosis of delirium was determined by use of the DSM-IV-TR [1], including four criteria: (1) disturbance of consciousness (i.e., reduced clarity of awareness of the environment) with reduced ability to focus, sustain, or shift attention; (2) a change in cognition (such as memory deficit, disorientation, language disturbance) or the development of a perceptual disturbance that is not better accounted for by a pre-existing, established, or evolving dementia; (3) the disturbance developing over a short period of time (usually hours to days) and tending to fluctuate during the course of the day; and (4) evidence from the history, physical examination, and laboratory findings that (i) the disturbance was caused by the direct physiological consequences of a general medical condition, (ii) the symptoms in criterion (i) developed during substance intoxication, or during or shortly after a withdrawal syndrome, or (iii) the delirium has more than one aetiology.

The Intensive Care Delirium Screening Checklist

The ICDSC [16] is a screening instrument including eight items based on the DSM-IV-TR criteria specifically designed for the intensive care setting with two points: absent or present. This scale was designed for patients with limited communication abilities such as intubated patients. The items include the assessment of: (1) consciousness (comatose, soporose, awake, or hypervigilant); (2) orientation; (3) hallucinations or delusions; (4) psychomotor activity; (5) inappropriate speech or mood; (6) attentiveness; (7) sleep-wake cycle disturbances; and (8) fluctuation of symptomatology. The maximum score is eight; scores of ≥4 indicate the presence of delirium and scores of >4 are not indicative of the severity of delirium [16]. Each item is rated on the patient’s behaviour at the time of assessment and the inter-rater reliability between intensive care staff was considered adequate [27].

The Confusion Assessment Method for the Intensive Care Unit

The CAM-ICU [26] reflects the DSM-III-R criteria [28] and is designed specifically for the ICU. This scale rates: (1) acute onset and fluctuating course; (2) inattention; (3) altered level of consciousness; and (4) disorganised thinking. Feature 1 is recorded as absent or present, feature 2, including recognising letters scores with number of errors of more than two, as present, feature 3 scores the Richmond Agitation Assessment Scale (RASS) other than alert and calm (RASS – 0) as present, and feature 4, including simple questions and instructions, with a combined number of errors of more than one as present. Feature 1 plus 2 and either 3 or 4 present indicate the presence of delirium. The CAM-ICU reflects only the presence or absence of delirium and does not provide a measure of severity.

Delirium Rating Scale-Revised-98

The DRS-R-98 [29] is a 16-item scale with 13 items describing severity, in addition to three diagnostic items, with four points – absent (0), mild (1), moderate (2) or severe impairment (3). The items required for this study tested for psychomotor activity: item 7 – increased and item 8 – decreased psychomotor behaviour and were classified as their respective subtypes: hyperactive, hypoactive, mixed and no motor subtype. The hyperactive subtype requires a score of 1 and more on increased motor behaviour, in the absence of hypoactivity, the hypoactive subtype a score of 1 and more on decreased motor behaviour, in the absence of hyperactivity, the mixed subtype both hypo- and hyperactivity, and last, the no-motor-subtype the absence of hyper- or hypoactivity as evidenced by the corresponding items. The rating describes the preceding 24 hours of psychomotor behaviour.

Statistical methods

All statistical procedures were conducted with the Statistical Package for Social Sciences (SPSS) version 22. Descriptive statistics were used for the characterisation of patients with delirium versus those without delirium. For the determination of differences between those with and without delirium, Student’s t-test was used for variables on a continuous scale such as the age of the patients. For items on categorical scales, such as gender distribution, diagnosis of delirium or the presence of items on the ICDSC, Pearson’s χ² was computed.

The inter-rater reliability with respect to the DSM-IV-TR diagnosis was determined by Intraclass correlation (ICC) with agreement of >0.80 defined as perfect [30].

The concurrent validity of ICDSC was calculated versus the DSM-IV-TR diagnosis of delirium setting the threshold at ≥3, ≥4, and ≥5.

All scales represented two levels indicating the absence or presence of delirium. Cohen’s κ was determined as measure of concordance with 0.41–0.60 defined as moderate, 0.61–0.80 as substantial and >0.80 as perfect [30]. The respective sensitivities and specificities, as well as corresponding positive and negative predictive values (PPVs and NPVs) were calculated and their confidence intervals (CIs) determined as exact Clopper-Pearson confidence intervals.

For all implemented tests, the significance level alpha was set at p <0.05.

Results

Characteristics of patients with delirium versus those without

The patients with delirium were older (69 vs 62 years), included more men (two thirds of patients) and were assessed at a later time in their hospitalisation (seventh vs fourth day) (table 1). In those with delirium as diagnosed by DSM-IV-TR, the ICDSC detected delirium in close to 80% when the cut-off was set at ≥3, in nearly two third of patients when it was set at the current cut-off of ≥4, and in less than 50% when set at ≥5 . Comparable to the ICDSC threshold for delirium of ≥5, the CAM-ICU detected delirium in only every other patient (46.7%).

Within the individual ICDSC items, with the exception of sleep-wake cycle disturbances (item 7), all items were more frequently scored in the delirious patients. The total ICDSC scores were higher in the presence of delirium.

The levels of sedation as measured with the Richmond Agitation and Sedation Scale (RASS) varied more in the patients with delirium. Although more than 80% ranged from drowsiness (RASS −1) to restlessness (RASS +1), almost 20% had levels of sedation of more than 1 or less than −1. Among those without delirium, RASS levels ranged only from −1 to 1.

The motor subtype, as determined from the motor items of the DRS-R-98, indicated that in those with delirium, the hypoactive and mixed subtype were more prevalent, whereas in those without delirium, no motor alterations prevailed.

Inter-rater reliability with respect to DSM-IV-TR diagnosis

With respect to the DSM-IV-TR diagnosis of delirium, the overall rating agreement between the psychiatrists’ assessment was perfect (ICC 0.96, CI 0.93–0.98, p<0.001).

ICDSC ≥3, ≥4 and ≥5 versus DSM-IV-TR and CAM-ICU versus ICDSC

The allocation of the presence and absence of delirium is presented in table 2. The following evaluations of concurrent validity are listed in tables 3 and 4 .

Reduced threshold for delirium set at ICDSC ≥3 versus DSM-IV-TR

When the cut-off score was decreased to ≥3, only 14.5% of patients with delirium as determined by means of the DSM-IV-TR were not recognised; however, more than 80% were correctly identified (82.9%) (table 1). Decreasing the threshold for delirium increased the rate of false positives to 11.9%. All ICDSC items and the total score were higher in those with delirium. The concurrent validity, defined as the corresponding κ, was 0.68, indicating substantial agreement between this threshold and the DSM-IV-TR-defined diagnosis of delirium. The sensitivity and specificity reached 83 and 85%, respectively, and both the positive and negative prediction exceeded 79%.

Currently accepted threshold for delirium set at ICDSC ≥4 versus DSM-IV-TR

The currently implemented cut-off score of ≥4 correctly identified 90% of those with delirium, but missed the diagnosis in more than every fifth patient (table 1). The rate of falsely identified delirium was 5%.

All ICDSC items were recorded at higher rates in those with delirium, which was reflected in a higher total score. The corresponding κ was 0.59, indicating a moderate agreement, almost reaching substantial agreement. The sensitivity was 62%, both specificity and positive prediction reached 90% and above, whereas the negative prediction reached 77%.

Increased threshold for delirium set at ICDSC ≥5 versus DSM-IV-TR

When the threshold was increased to ≥5, the rate of incorrectly identified delirium increased to almost one third, whereas 90% of those with delirium were correctly recognised and the rate for false positives was 4% (table 1).

Once again, both individual ICDSC items and total score were more prevalent and higher in those with delirium. The agreement with the DSM-IV-TR-determined diagnosis of delirium was only moderate (κ 0.44), the corresponding sensitivity reached only 44%, both specificity and positive prediction reached and exceeded 90% and the negative prediction was 70%.

ICDSC ≥3, ≥4 or ≥5 versus the CAM-ICU

Varying the threshold for delirium on the ICDSC to ≥3 or to ≥5 did not change its concurrent moderate agreement with the CAM-ICU-defined diagnosis of delirium. When the threshold was decreased to ≥3, the respective κ was 0.45 (CI 0.33–0.57, p<0.001), when increased to ≥5, the κ was 0.52 (CI 0.38–0.66, p<0.001), versus the current threshold of ≥4 with a κ of 0.56 (CI 0.43–0.69, p<0.001).

Discussion

Summary of main findings

In clinical routine, the ICDSC is a useful and very specific instrument for the detection of delirium in the intensive care setting. However, as previously suggested, the threshold for delirium, set as an ICDSC total score of ≥4, had shortcomings in terms of accuracy in detecting this syndrome when compared with a threshold of ≥3, in contrast to increasing the threshold to ≥5.

Although decreasing the threshold to ≥3 increased the rate of false positives, the rate of false negatives was lower than with thresholds of ≥4 or ≥5. In addition, this cut-off score identified 83% of delirious patients correctly, and reached substantial agreement with the DSM-IV-TR-defined diagnosis of delirium, whereas the other thresholds had only moderate agreement. This lower threshold was the most sensitive, was specific and yielded high rates of positive and negative prediction. Increasing the threshold for delirium to ≥5 produced few false negatives and was very specific, but proved the weakest in sensitivity to identify delirium, which is most important for a screening instrument. The agreement with the DSM-IV-TR diagnosis was the lowest.

Based on these results, the most useful approach was decreasing the threshold for delirium to ≥3, irrespective of subsyndromal delirium.

Comparison with the existing literature

Use of an ICDSC cut-off score of ≥4 yields conflicting results. A number of studies indicated that the ICDSC is a very sensitive and specific instrument, generally, with specificity reaching 95% [21, 31], but other studies indicated low sensitivities (43 to 47%) while maintaining high specificities (>94%) [18, 19]. As previously shown, the cut-off score of ≥4 failed to detect one third of cases with delirium [25]; this was the reason for this evaluation study of alternative cut-offs.

As previously suggested, varying the threshold for delirium by decreasing ths score from ≥4 to ≥3 [23], or increasing it to ≥5 and setting subsyndromal delirium at ≥5 [21] could improve its accuracy. Previous studies suggested that decreasing the threshold to ≥3 increased the sensitivity from 75 to 90%, but the specificity decreased from 74.3 to 61.5% [23]. By increasing the threshold to ≥5, the specificity was increased from 72.4 to 86.5% [21]. In the end, both approaches may be similar, since increasing the threshold for delirium to ≥5 includes subsyndromal delirium at ≥3, with the pitfall that the ICDSC has not been validated for subsyndromal delirium. In this study, decreasing the ICDSC threshold for delirium to ≥3 achieved substantial agreement with the DSM-IV-TR-determined diagnosis of delirium, increased the sensitivity from 61 to almost 83%, while remaining very specific (85.5%), with substantial positive and negative prediction, and a decreased rate of false negatives. This approach was very useful in reducing the under-detection of delirium and subsequently reducing risks for adverse outcomes. Whether this approach included subsyndromal forms of delirium could not be determined since the range for subsyndromal delirium for the ICDSC remains unclear.

Conversely, increasing the threshold for delirium to ≥5 did not prove to be a beneficial approach. The concurrent validity with DSM-IV-TR was the lowest, the sensitivity was decreased from 62 to 44%, while reaching substantial specificity and positive prediction. However, with this approach, the rate of false negatives (undetected delirium) was the highest.

Reports of the concurrent validity and agreement between the ICDSC and CAM-ICU also vary. One study comparing the ICDSC and CAM-ICU without using the DSM criteria for the diagnosis for delirium found substantial agreement between both scales (κ 0.80) [22], whereas another study including the DSM-IV-TR diagnosis found only moderate agreement (κ 0.59) [21] and raised the question as to whether varying the ICDSC threshold for delirium could strengthen the agreement. From these findings, the agreement between both scales was only moderate (κ 0.56) and was not increased by setting the threshold at a lower or higher score (≥3, κ 0.45; ≥5, κ 0.52).

Discrepancies between the ICDSC versus DSM-IV-TR

Potential explanations for the discrepancies between the ICDSC- and DSM-IV-TR-determined diagnoses of delirium include human error and lack of training. Nurses were instructed in the proper administration of the ICDSC and cases were discussed, but no formal inter-rater reliability was achieved and potential bias in scoring was possible. Because of the design of this study, this potential confounder could not be excluded. In contrast, the overdiagnosis of delirium by the expert raters was excluded by the perfect agreement between their assessments. Another potential confounder was the level of sedation. Within those with delirium, the RASS level ranged from −3 to 2, whereas in those not delirious, the RASS levels ranged only from −1 to 1. Although more than 90% of delirious patients had RASS levels of −1 to 1, the RASS level could have influenced the prevalence rate for delirium, since the level of sedation increases delirium rates [32].

Clinical relevance

The ICDSC is a brief, observational screening instrument with only eight items commonly used in the intensive care setting. Lowering the threshold for delirium to ≥3 enhanced the ability of this instrument to accurately detect delirium and allowed confirmation of delirium with the CAM-ICU in a setting known for its high prevalence rates of delirium. Although lowering the threshold for delirium to ≥3 produced more false positives, this is irrelevant for a screening instrument with a focus on sensitivity rather than specificity.

Strengths and limitations

This study has several strengths. Almost 300 patients were prospectively screened and rated for delirium using the ICDSC versus the DSM-IV-TR criteria, and 289 patients were included. With respect to diagnosis of delirium with the DSM-IV-TR criteria, the inter-rater agreement was perfect. However, a number of limitations have to be noted, including the lack of formal inter-rater reliability in the performance of the ICDSC, not allowing to exclude its proper administration, the high prevalence of hypoactive delirium, which was indebted to the critical care population studied, the absence of baseline cognitive recording owed to the prospective setting of the study. Further, this study was cross-sectional and longitudinal studies of the concurrent validity of the ICDSC, in particular with the optimal threshold for delirium, are required.

Conclusion

In summary, varying the ICDSC threshold for delirium proved to be a beneficial approach in improving the accuracy of this scale. In particular, lowering this score to ≥3 increased the sensitivity while maintaining the specificity, and positive and negative prediction, as well as reducing the rate of under-detection at the cost of over-identification. In contrast, increasing the threshold for delirium to ≥5 decreased the sensitivity while maintaining the specificity, positive and negative prediction, but the under-detection of delirium increased and was substantial. Altogether, these findings favoured decreasing the ICDSC threshold for delirium score to ≥3.

References

1American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, Text Revision. Washington, DC: American Psychiatric Association; 2000. pp 124–7. 

2 Trzepacz PT , Breitbart W , Franklin J , Levenson J , Martini R , Wang P ; American Psychiatric Association. Practice guideline for the treatment of patients with delirium. Am J Psychiatry. 1999;156(5, Suppl):1–20.  

3 Inouye SK , Westendorp RG , Saczynski JS . Delirium in elderly people. Lancet. 2014;383(9920):911–22. doi:.https://doi.org/10.1016/S0140-6736(13)60688-1  

4 Bucht G , Gustafson Y , Sandberg O . Epidemiology of delirium. Dement Geriatr Cogn Disord. 1999;10(5):315–8. doi:.https://doi.org/10.1159/000017161  

5 Norkiene I , Ringaitiene D , Misiuriene I , Samalavicius R , Bubulis R , Baublys A , et al. Incidence and precipitating factors of delirium after coronary artery bypass grafting. Scand Cardiovasc J. 2007;41(3):180–5. doi:.https://doi.org/10.1080/14017430701302490  

6 Gottesman RF , Grega MA , Bailey MM , Pham LD , Zeger SL , Baumgartner WA , et al. Delirium after coronary artery bypass graft surgery and late mortality. Ann Neurol. 2010;67(3):338–44.  

7 Pun BT , Ely EW . The importance of diagnosing and managing ICU delirium. Chest. 2007;132(2):624–36. doi:.https://doi.org/10.1378/chest.06-1795  

8 Santos FS , Velasco IT , Fráguas R, Jr . Risk factors for delirium in the elderly after coronary artery bypass graft surgery. Int Psychogeriatr. 2004;16(2):175–93. doi:.https://doi.org/10.1017/S1041610204000365  

9 Rosen SF , Clagett GP , Valentine RJ , Jackson MR , Modrall JG , McIntyre KE . Transient advanced mental impairment: an underappreciated morbidity after aortic surgery. J Vasc Surg. 2002;35(2):376–81. doi:.https://doi.org/10.1067/mva.2002.119233  

10 Koster S , Hensens AG , van der Palen J . The long-term cognitive and functional outcomes of postoperative delirium after cardiac surgery. Ann Thorac Surg. 2009;87(5):1469–74. doi:.https://doi.org/10.1016/j.athoracsur.2009.02.080  

11 Ely EW , Shintani A , Truman B , Speroff T , Gordon SM , Harrell FE, Jr , et al. Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit. JAMA. 2004;291(14):1753–62. doi:.https://doi.org/10.1001/jama.291.14.1753  

12 Ouimet S , Kavanagh BP , Gottfried SB , Skrobik Y . Incidence, risk factors and consequences of ICU delirium. Intensive Care Med. 2007;33(1):66–73. doi:.https://doi.org/10.1007/s00134-006-0399-8  

13 Heymann A , Radtke F , Schiemann A , Lütz A , MacGuill M , Wernecke KD , et al. Delayed treatment of delirium increases mortality rate in intensive care unit patients. J Int Med Res. 2010;38(5):1584–95. doi:.https://doi.org/10.1177/147323001003800503  

14 Balas MC , Happ MB , Yang W , Chelluri L , Richmond T . Outcomes Associated With Delirium in Older Patients in Surgical ICUs. Chest. 2009;135(1):18–25. doi:.https://doi.org/10.1378/chest.08-1456  

15 Bickel H , Gradinger R , Kochs E , Förstl H . High risk of cognitive and functional decline after postoperative delirium. A three-year prospective study. Dement Geriatr Cogn Disord. 2008;26(1):26–31. doi:.https://doi.org/10.1159/000140804  

16 Devlin JW , Fong JJ , Schumaker G , O’Connor H , Ruthazer R , Garpestad E . Use of a validated delirium assessment tool improves the ability of physicians to identify delirium in medical intensive care unit patients. Crit Care Med. 2007;35(12):2721–4, quiz 2725. doi:.https://doi.org/10.1097/00003246-200712000-00007  

17 Gusmao-Flores D , Salluh JI , Chalhub RA , Quarantini LC . The confusion assessment method for the intensive care unit (CAM-ICU) and intensive care delirium screening checklist (ICDSC) for the diagnosis of delirium: a systematic review and meta-analysis of clinical studies. Crit Care. 2012;16(4):R115. doi:.https://doi.org/10.1186/cc11407  

18 van Eijk MM , van Marum RJ , Klijn IA , de Wit N , Kesecioglu J , Slooter AJ . Comparison of delirium assessment tools in a mixed intensive care unit. Crit Care Med. 2009;37(6):1881–5. doi:.https://doi.org/10.1097/CCM.0b013e3181a00118  

19 Neufeld KJ , Leoutsakos JS , Sieber FE , Joshi D , Wanamaker BL , Rios-Robles J , et al. Evaluation of two delirium screening tools for detecting post-operative delirium in the elderly. Br J Anaesth. 2013;111(4):612–8. doi:.https://doi.org/10.1093/bja/aet167  

20 Marquis F , Ouimet S , Riker R , Cossette M , Skrobik Y . Individual delirium symptoms: do they matter? Crit Care Med. 2007;35(11):2533–7. doi:.https://doi.org/10.1097/01.CCM.0000284506.43390.F3  

21 Gusmao-Flores D , Salluh JI , Dal-Pizzol F , Ritter C , Tomasi CD , Lima MA , et al. The validity and reliability of the Portuguese versions of three tools used to diagnose delirium in critically ill patients. Clinics (Sao Paulo). 2011;66(11):1917–22.  

22 Plaschke K , von Haken R , Scholz M , Engelhardt R , Brobeil A , Martin E , et al. Comparison of the confusion assessment method for the intensive care unit (CAM-ICU) with the Intensive Care Delirium Screening Checklist (ICDSC) for delirium in critical care patients gives high agreement rate(s). Intensive Care Med. 2008;34(3):431–6. doi:.https://doi.org/10.1007/s00134-007-0920-8  

23 George C , Nair JS , Ebenezer JA , Gangadharan A , Christudas A , Gnanaseelan LK , et al. Validation of the Intensive Care Delirium Screening Checklist in nonintubated intensive care unit patients in a resource-poor medical intensive care setting in South India. J Crit Care. 2011;26(2):138–43. doi:.https://doi.org/10.1016/j.jcrc.2010.11.002  

24 van Eijk MM , van den Boogaard M , van Marum RJ , Benner P , Eikelenboom P , Honing ML , et al. Routine use of the confusion assessment method for the intensive care unit: a multicenter study. Am J Respir Crit Care Med. 2011;184(3):340–4. doi:.https://doi.org/10.1164/rccm.201101-0065OC  

25 Boettger S , Nuñez DG , Meyer R , Richter A , Fernandez SF , Rudiger A , et al. Delirium in the intensive care setting: A reevaluation of the validity of the CAM-ICU and ICDSC versus the DSM-IV-TR in determining a diagnosis of delirium as part of the daily clinical routine. Palliat Support Care. 2017;15(6):675–83. doi:.https://doi.org/10.1017/S1478951516001176  

26 Ely EW , Inouye SK , Bernard GR , Gordon S , Francis J , May L , et al. Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA. 2001;286(21):2703–10. doi:.https://doi.org/10.1001/jama.286.21.2703  

27 Bergeron N , Dubois MJ , Dumont M , Dial S , Skrobik Y . Intensive Care Delirium Screening Checklist: evaluation of a new screening tool. Intensive Care Med. 2001;27(5):859–64. doi:.https://doi.org/10.1007/s001340100909  

28American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 3rd Edition, revised. Washington, DC: American Psychiatric Association; 1987. 

29 Trzepacz PT , Mittal D , Torres R , Kanary K , Norton J , Jimerson N . Validation of the Delirium Rating Scale-revised-98: comparison with the delirium rating scale and the cognitive test for delirium. J Neuropsychiatry Clin Neurosci. 2001;13(2):229–42. doi:.https://doi.org/10.1176/jnp.13.2.229  

30DeVellis RF. Scale development: Theory and applications. Los Angeles, CA: Sage publications; 2012. pp 109–10.  

31 Bergeron N , Dubois MJ , Dumont M , Dial S , Skrobik Y . Intensive Care Delirium Screening Checklist: evaluation of a new screening tool. Intensive Care Med. 2001;27(5):859–64. doi:.https://doi.org/10.1007/s001340100909  

32 Haenggi M , Blum S , Brechbuehl R , Brunello A , Jakob SM , Takala J . Effect of sedation level on the prevalence of delirium when assessed with CAM-ICU and ICDSC. Intensive Care Med. 2013;39(12):2171–9. doi:.https://doi.org/10.1007/s00134-013-3034-5