Development and validation of a multivariable risk score for prolonged length of stay in the surgical intensive care unit

DOI: https://doi.org/10.4414/smw.2019.20122

Introduction

Improvements in the medical and nursing fields have led to an increasing number of intensive care patients who survive the acute phase of critical illness [1]. After a stay in the intensive care unit (ICU), the majority of patients recover. However, there is also a growing number of so-called chronically critically ill patients [2].

The definitions of chronically critical illness (CCI) in the literature vary greatly regarding characteristic symptoms and criteria [3, 4]. In summary, CCI is a syndrome with significant metabolic, neuroendocrine, neuropsychiatric and immunological dysfunctions [5]. In this illness trajectory, physiological responses during the acute phase of critical illness fail to balance out the induced stress. Physical disorders and mandatory intensive care interventions lead to dysregulation, a loss of resources and consequently to a prolonged stay in the ICU as a main criterion for CCI [6–8].

Of mechanically ventilated patients, 5 to 10% develop CCI, consuming 20 to 40% of the time resources of the ICU [6, 9]. Half of all CCI patients require long-term care combined with a great risk of nursing home admission. In addition, 40% of CCI patients are readmitted to an acute care hospital within the first 6 months following discharge [5].

The in-hospital mortality rate among CCI patients is 20 to 40% [6]; the 1-year mortality rate is 30 to 72% [5]; 12% of them can live independently after 1 year [6].

CCI represents a huge challenge for the affected patients, their families and the inter-professional team at the ICU. Healthcare systems need to pay special attention to CCI patients [8]. This could be strengthened by the systematic identification of patients with a high risk of CCI in the earliest possible stage of its development [10]. Because of differences in the definitions and the limited applicability of the existing predictive models, the timely detection of CCI is challenging. Systematic and timely identification of patients with a prolonged ICU length of stay (PICULOS) at a meaningful point in time could be a potential advantage and a first step toward optimal, evidence-based treatment and care planning for CCI patients [5, 6].

The definition of PICULOS as a length of stay (LOS) of more than 20 days is commonly employed in studies to predict a high risk of CCI [11–13].

Several authors have described the predictive factors associated with PICULOS, which can be categorised as occurring before admission (e.g., comorbidities [5, 14]), on the day of admission (e.g., illness severity [15]), and during the acute phase of a critical illness (e.g., presence of a severe infection [3, 4, 6, 16]), hypoalbuminaemia [13, 17], respiratory insufficiency [2, 18]), and after the first week in ICU (e.g., nutrition problems [11, 18]), consciousness disturbances [19]).

According to Bellar et al. [8], the chronic phase of a critical illness begins on the seventh to tenth day of an ICU stay. This could be a meaningful point in time to predict PICULOS of more than 20 days.

This study addressed the following question: how do a set of seven pre-selected factors perform (sensitivity, specificity, predictive values) in predicting PICULOS (>20 d.) in adult surgical ICU patients on the seventh day of their stay (day 7).

Based on a preliminary project, this study aimed to develop and validate a risk score to predict PICULOS in patients of one surgical ICU in order to contribute to the timely identification of patients who are at a high risk of CCI.

Materials and methods

Preliminary project

Prior to this study, an inter-professional team of the surgical ICU at University Hospital, Basel, associated knowledge about CCI with potentially predictive factors, based on a systematic literature review, personal experience and a self-developed conceptual model of allostatic load in CCI patients [20]. In this nurse-led quality development project, we hypothesised that the systematic identification of PICULOS could be a first step toward optimal care planning for CCI patients in this local context.

Through decision making by consensus and after checking the local data, the project team selected seven predictive factors for assessing the risk of PICULOS (>20 d.) on the seventh day of stay (day 7). Table 1 describes the identified predictors and their operationalisation, measurement and statement in terms of literature and context. These factors form the basis for the present study.

Design

This single centre cohort study was based on the recommendations of prognosis research [25, 26], which suggest the exploration of different possible predictive factors in combination with elementary clinical information. We used the prognosis research strategy (PROGRESS) framework [27] to achieve the highest possible standard of study quality, design and analysis.

Setting

We performed the study as part of the nurse-led quality development project at the surgical ICU at University Hospital, Basel, Switzerland. Annually, the surgical ICU team (22 beds) cares for approximately 2600 adult patients and covers all surgical and medical disciplines, with a median age of 66 years (interquartile range [IQR] 53–76) and a median LOS of 0.96 days (IQR 0.77–1.87) (Data from the minimal dataset (2016) of the Swiss Society of Intensive Care Medicine [SGI]).

Data Collection

Two consecutive datasets were collected: one for the development (n = 304) and the other for the validation (n = 101) of the predictive model (fig. 1). We included all adult patients (≥18 years) with LOS of seven or more consecutive days (≥7d.), between 1 January 2014 and 31 March 2016 for the development dataset (retrospective) and between 1 April 2016 and 31 December 2016 for the validation dataset. The data collection was ongoing in everyday practice, after implementation of the risk score in our practice. Patients who were discharged to another ICU or who died between day 7 and day 20 were excluded.

Figure 1 Populations, exclusion, study samples and groups. DD = development dataset; ICU = intensive care unit; LOS = length of stay; VD = validation dataset

All data could be collected completely and were entered in an encoded IBM SPSS^© Version 22 database. All variables were collected at the highest possible level of measurement (e.g., ratio scale). The file was created by manually obtaining the data from two different, routinely used electronic medical record systems (MetaVision^©, ISMed^©).

Variables and measurements

Candidate predictive factors of PICULOS were identified in the preliminary project (table 1).

Based on scientific literature [8, 11–13] and personal experience, the outcome variable PICULOS was previously dichotomised as “negative” if the LOS was 7 to 20 days (group 1), and “positive” if LOS was >20 days (group 2). Day 1 was the day of admission, regardless of the time of day. Each day of stay counted as a whole day, regardless of the amount of time spent in the unit on the admission and discharge days.

Additionally, patient characteristics (sex, age on admission day, medical discipline: heart, thoracic, traumatology/orthopaedic, visceral, neurosurgery, other) were collected from the medical files.

Ethics approval and consent to participate

The process of collection, storage and processing of data was approved by the corresponding ethics committee (Ethikkommission Nordwest- und Zentralschweiz, EKNZ 2016-00948).

Our study involved pre-existing data only (“further use research”). We did not obtain consent to use all of the data based on different reasons, e.g. high morbidity and mortality rates. Our results may help future CCI patients to recover faster.

Data analysis

We analysed all of the variables in the development dataset descriptively, summarising them as measures of central tendency (mean, median) and dispersion (standard deviation [SD], IQR, range).

Predictors were entered into a multiple logistic regression analysis in order to model the probability of the patients staying longer than 20 days, and we retained significant factors only by manual backward deletion, while monitoring the estimates, confidence intervals and inferences of the remaining variables in the model. We also checked for nonlinear relationships using spline functions. In the case of nonlinearity, the threshold values were defined, which were also validated by the literature and practical experience. On the basis of the resulting regression model, a risk score for each patient was calculated by summing the first two digits of the obtained odds ratios (ORs) (multiplied by 10 then rounded up). The diagnostic characteristics of this risk score were explored using a receiver-operating characteristic (ROC), which provided us with the optimal cut-off for deciding whether or not a patient was at risk of an excessively long stay, using the most distant point from the curve to the diagonal.

The developed algorithm was validated using the validation dataset. We additionally checked exploratively whether the previously detected nonlinearities were located at the same values as found in the development set. IBM SPSS^© 22 and SAS 9.4 were used for the data analysis.

Results

The demographic and clinical characteristics, as well as candidate predictors, of the development sample (n = 304) are presented in table 2.

Regression analysis

The results of the regression analyses in table 3 show: (A) an initial model; (B) a model with retained significant variables; and (C) a model with the need for parenteral nutrition on day 7 (yes/no) and dichotomised variables split along the discontinuous relationships found between PICULOS and the variables Charlson Comorbidity Index (>2 points), mechanical ventilation (>14 hr on day 7), and minimum albumin (<20 g/l from days 1–7). The final models revealed mechanical ventilation >14 hr (OR 9.79; 95% CI 4.73–20.27) to be the strongest predictor. The determination coefficient of the final simplified model (R² = 0.36) indicted that it explained as much of the variability as the initial model (R² = 0.37)

The risk scores were derived from the odds ratios and calculated as follows: mechanical ventilation >14 hr on day 7 scored 98 points, the need for parenteral nutrition on day 7 scored 36 points, the lowest albumin concentration <20 g/l between day 1 and day 7 scored 28 points, and a Charlson Comorbidity Index >2 on day 1 scored 23 points. Appendix 1 includes a tool that can be used to enter individual data to calculate the risk of a long ICU stay.

The risk score had an area under the ROC curve of 0.82 (95% CI 0.77–0.87) with regard to the prediction of PICULOS (fig. 2). The point of maximum discriminatory power derived from the ROC was at 100 points, where the sensitivity was 82% (indicating the proportion of true positives among all of those who stayed for longer than 20 days) and the specificity was 73% (indicating the proportion of true short-stayers among all those staying for fewer than 21 days). The positive predictive value of 48% indicated that, where the risk score >100 points, the chance that the patient actually stayed for longer than 20 days was slightly less than half. The negative predictive value of 93% reflects the chance that those with scores <100 points would indeed stay for 20 days or fewer (table 4). Interestingly, a risk score of 100 points coincided with the start of an over-proportional increase in the probability of a stay exceeding 20 days beyond that score (fig. 3). This warranted dichotomising the risk score using 100 points as a cut-off in order to determine those patients with positive scores. If entered into a logistic regression analysis, this binary variable predicted PICULOS still at an acceptable R² of 0.30.

Figure 2 Receiver operating characteristic of the risk score in development sample (n = 304)

Figure 3 Nonlinear relationship between the risk score and probability to stay >20 days in development sample (n = 304)

Validation

The characteristics of the validation dataset (n = 101) are presented in table 5. Application of the risk score to the prediction of PICULOS resulted in an area under the ROC curve of 0.89 (95% CI 0.83–0.96). For confirmatory purposes, we explored whether a nonlinear trend was again present in the risk score, and found that the optimal cut-off at 100 points was identical to that for the development set. The diagnostic parameters using this cut-off were slightly higher compared to the developmental data (sensitivity 88%, specificity 75%, positive predictive value 53%, negative predictive value 95%). If entered into a logistic regression analysis, this binary variable predicted PICULOS at an R² of 0.40.

Discussion

Based on a regression model, we developed a dichotomised risk score for predicting on day 7 a PICULOS of ˃20 days (fig. 4). The model included the factors mechanical ventilation for >14 hr and the need for parenteral nutrition on day 7, lowest albumin <20 g/l in the first 7 days and a Charlson Comorbidity Index >2. This was in line with the results of other studies, reporting specific pre-existing diseases [12], hypoalbuminaemia [13], a dependence on MV [11], and parenteral nutrition [12] as predictive factors. The score with a sufficient discriminatory ability facilitated the timely identification of patients with PICULOS on day 7. Almost all of the patients with a negative test result were discharged between days 7 and 20.

Figure 4 Multivariable model to predict a prolonged length of stay in the surgical ICU in Basel. C-Index = Charlson Comorbidity Index; MV = mechanical ventilation, PICULOS = prolonged intensive care unit length of stay

In our study, mechanical ventilation for more than 14 hr on day 7 was the most influential factor in predicting PICULOS. A positive test result (>100 points) was impossible without mechanical ventilation >14 hr on day 7. This result suggests that even patients who are ventilated for less than 24 hr on day 7 may be at a high risk of PICULOS. The considerable contribution of mechanical ventilation is also confirmed by other studies [11, 13, 15, 18, 28, 29], although our study showed that a combination of several predictive factors has a higher predictive value than the factor mechanical ventilation >14 hr alone.

Significantly more patients with LOS >20 days needed parenteral nutrition. The need for parenteral nutrition was also a predictive factor in the model of Chen et al. [12]. The high demand for enteral nutrition was striking in all patients with an ICU stay >7 days. Parenteral nutrition seems to be a sign of intestinal absorption failure, prohibiting protein anabolism.

A minimum albumin value <20 g/l between days 1 and 7 was found to be a significant predictive factor. The chosen operationalisation proved to be highly suitable. The mere presence of hypoalbuminaemia (albumin value <34 g/l [30]) as a predictive factor would have been insufficient, since all patients with LOS ≥7 days in our study had an albumin value below this defined threshold. Lee et al. [31] also showed that, amongst other things, plasma albumin predicted ICU LOS in general surgery, but it must be kept in mind that albumin levels may change as a result of the infusion of albumin. This lies beyond the scope of our present study.

Our results confirmed the importance of specific comorbidities in determining outcomes following critical illness [32, 33], and the association between chronic comorbidities and PICULOS [3, 5]. The updated version of the Charlson Comorbidity Index [14], originally prepared to predict mortality within 1 year after hospital discharge, is also an appropriate risk factor to identify PICULOS.

The factors SAPS II, therapeutic need for anti-infective drugs, and SAS did not contribute significantly to the prediction of PICULOS. In the case of SAPS II, this may indicate that a very long ICU stay does not depend significantly on the specific acute critical illness. The therapeutic need for anti-infective drugs between days 1 and 7 was also not affirmed as an independent predictor. This could be because a large proportion of the sampled patients (74.3%) needed anti-infective drugs. In the literature, infections are often used as a variable for predicting LOS in ICU [12, 15], notwithstanding the fact that it is unclear whether it is a cause [2, 16], a developmental factor [15], or a consequence [6, 8] of CCI. Likewise, despite finding low SAS values (1–2) for patients with PICULOS >20 days, no independent relationship could be found in an analysis controlled for the variable time of MV, the reason for which is unclear. Brain dysfunction, as well as cognitive symptoms including delirium and memory gaps, are described as typical criteria for CCI [5, 6, 10, 34]. However, the exact cause-and-effect relationship between disturbances in consciousness and the development of CCI remains to be investigated.

In this study, we used PICULOS as an operationalisation of CCI. This outcome variable enabled us to include patients with different diagnoses and treatments. In addition, we were able to form study groups for the group comparisons (see fig. 1). Our open research method emphasises literature recommendations, because PICULOS includes various criteria of CCI, such prolonged mechanical ventilation (PMV) [2].

The inclusion of patients with LOS ≥7 days and the assessment on day 7 seem to be advantageous. This group differed significantly from many factors in the entire ICU population (see table 2). Widyastuti et al. [35] were not able to predict a long ICU stay in individual patients on the basis of assessment on day 1, because most patients had short ICU stays (75th percentile: 1 day). This result appears to be relevant in our unit, where patients also have a low average LOS. The assessment on day 7 allows the inclusion of factors that reflect the acute phase of illness, such as hypoalbuminaemia. Thus, it is possible to recognise systematically patients at risk, at the beginning of the chronic phase.

Limitations

PICULOS and the corresponding predictors depend on local processes and treatment strategies; for instance, ICUs that use more albumin infusions to increase the serum levels of albumin near 30 g/l will be unable to use our dichotomised score [36]. Experts from our ICU selected the predictive factors through a consensus process, focussing on data available from patient records. This contained a certain degree of subjectivity and might limit the generalisability of our results. Factors that could not simply be derived from the existing documentation or factors that are very difficult to operationalise may have been falsely excluded: for example, sociodemographic factors, such as resilience or family support. We operationalised our outcome variable PICULOS as a dichotomous variable of LOS of more than 20 days. This artificially determined threshold could disadvantage patients who fail fully to meet this predefined condition but who are nevertheless at a high risk of CCI (e.g., a patient discharged from the ICU to an intermediate care unit after 18 days). Riley et al. [25] recommend analysing continuous factors using their continuous scale. They also recommend a prospective rather than a retrospective design, as this produces clear inclusion criteria, more complete baseline and follow-up data, as well as a greater standardisation of the diagnostic and therapeutic procedures. However, there were no missing data in our study, we collected the data directly from the individual patients’ documentation, and we discussed special cases within the study team. Despite these limitations, the risk score works in our setting appropriately, and our approach to develop a local risk score can easily be adapted to other ICUs.

Conclusions for clinical practice

This study marked an important step toward equipping the involved health professionals with an extended understanding of PICULOS, the trajectory of CCI and prognosis research as intrinsic aspects of clinical care [27].

Our model is suitable for systematic application within our ICU. It is important that all responsible health professionals are informed of any positive scores (e.g., during inter-professional ward rounds). To improve the quality of clinically collected data, electronic documentation systems could assist the automatic calculation of the score. We have already successfully implemented this within our electronic documentation. The transferability of our results into comparable settings still has to be investigated.

More objective estimated probabilities can supplement the clinical reasoning and decision making of health professionals [26]. However, it must be kept in mind that the predictive models form only part of good qualitative reasoning. Patients who are at increased risk of CCI (and not only PICULOS) urgently require a comprehensive assessment and patient-centred treatment planning.

The study did not investigate whether the risk score alone affects the development or trajectory of CCI. However, based on the risk score, we intensified the care for patients at risk of PICULOS in our ICU while integrating all therapeutic professionals, tailored family information, systematic communication and coordination, and the development of an evidence-based assessment and treatment plan. From this, one can infer that the assessment based on our model could be the first step toward the optimal treatment and care of patients with a high risk of CCI.

Implications for further research

Updating and advancing a model by exploring additional prognostic factors is often desirable [37]. According to other studies, the presence of pressure sores [5, 11, 13] or the patients’ physical capacity before critical illness [38] could be suitable additional factors. Investigation of psychosocial factors (e.g., resilience, marital status, social support) might also prove rewarding.

Another implication for research is the examination of the clinical impact of the tool on decision-making and patient outcomes [39]. In a comparative study, one patient group with usual care should be compared with another group in which the model’s predictions are made available to health professionals to guide their treatment decisions [37].

Availability of data and materials

The dataset generated and analysed during the current study is not publicly available because the study was an internal quality development project. With the approval of the ethics committee (Ethikkommission Nordwest- und Zentralschweiz EKNZ), the patient data were used without the consent of the patients, as this was disproportional for this purpose. The dataset is available from the corresponding author on reasonable request.

Appendix 1 Tool to calculate the risk of a long ICU stay

The appendix is available as a separate file for downloading at https://smw.ch/en/article/doi/smw.2019.20122/

Acknowledgements

Allison Dwileski, scientific secretary in the Department of Anaesthesiology, University Hospital Basel: editing of the publication text in English. Helmut Wesch, English teacher: Editing of the publication text in English. Assessment of the text as a nonprofessional.

References

1 Rosseau S , Suttorp N . Der chronisch kritisch kranke Patient [The chronic critically ill patient]. Med Klin Intensivmed Notf Med. 2013;108(4):266. In German. doi:.https://doi.org/10.1007/s00063-012-0162-6  

2 Carson SS . Definitions and epidemiology of the chronically critically ill. Respir Care. 2012;57(6):848–56, discussion 856–8. doi:.https://doi.org/10.4187/respcare.01736  

3 Kahn JM , Le T , Angus DC , Cox CE , Hough CL , White DB , et al.; ProVent Study Group Investigators. The epidemiology of chronic critical illness in the United States. Crit Care Med. 2015;43(2):282–7. doi:.https://doi.org/10.1097/CCM.0000000000000710  

4 MacIntyre NR , Epstein SK , Carson S , Scheinhorn D , Christopher K , Muldoon S ; National Association for Medical Direction of Respiratory Care. Management of patients requiring prolonged mechanical ventilation: report of a NAMDRC consensus conference. Chest. 2005;128(6):3937–54. doi:.https://doi.org/10.1378/chest.128.6.3937  

5 Wiencek C , Winkelman C . Chronic critical illness: prevalence, profile, and pathophysiology. AACN Adv Crit Care. 2010;21(1):44–61, quiz 63. doi:.https://doi.org/10.1097/NCI.0b013e3181c6a162  

6 Nelson JE , Cox CE , Hope AA , Carson SS . Chronic critical illness. Am J Respir Crit Care Med. 2010;182(4):446–54. doi:.https://doi.org/10.1164/rccm.201002-0210CI  

7 Beckie TM . A systematic review of allostatic load, health, and health disparities. Biol Res Nurs. 2012;14(4):311–46. doi:.https://doi.org/10.1177/1099800412455688  

8 Bellar A , Kunkler K , Burkett M . Understanding, recognizing, and managing chronic critical illness syndrome. J Am Acad Nurse Pract. 2009;21(11):571–8. doi:.https://doi.org/10.1111/j.1745-7599.2009.00451.x  

9 Rodríguez Villar S , Barrientos Yuste RM . Long-term admission to the intensive care unit: a cost-benefit analysis. Rev Esp Anestesiol Reanim. 2014;61(9):489–96. doi:.https://doi.org/10.1016/j.redar.2014.02.008  

10 Jeitziner MM , Massarotto P , Barandun Schäfer U . Symptombelastung und entsprechende Interventionen. Intensiv. 2015;23(03):123–7. In German. doi:.https://doi.org/10.1055/s-0035-1550608  

11 Loss SH , Marchese CB , Boniatti MM , Wawrzeniak IC , Oliveira RP , Nunes LN , et al. Prediction of chronic critical illness in a general intensive care unit. Rev Assoc Med Bras (1992). 2013;59(3):241–7. doi:.https://doi.org/10.1016/j.ramb.2012.12.002  

12 Chen HY , Vanness DJ , Golestanian E . A simplified score for transfer of patients requiring mechanical ventilation to a long-term care hospital. Am J Crit Care. 2011;20(6):e122–30. doi:.https://doi.org/10.4037/ajcc2011775  

13 Szubski CR , Tellez A , Klika AK , Xu M , Kattan MW , Guzman JA , et al. Predicting discharge to a long-term acute care hospital after admission to an intensive care unit. Am J Crit Care. 2014;23(4):e46–53. doi:.https://doi.org/10.4037/ajcc2014985  

14 Quan H , Li B , Couris CM , Fushimi K , Graham P , Hider P , et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173(6):676–82. doi:.https://doi.org/10.1093/aje/kwq433  

15 Higgins TL , McGee WT , Steingrub JS , Rapoport J , Lemeshow S , Teres D . Early indicators of prolonged intensive care unit stay: impact of illness severity, physician staffing, and pre-intensive care unit length of stay. Crit Care Med. 2003;31(1):45–51. doi:.https://doi.org/10.1097/00003246-200301000-00007  

16 Boniatti MM , Friedman G , Castilho RK , Vieira SR , Fialkow L . Characteristics of chronically critically ill patients: comparing two definitions. Clinics (São Paulo). 2011;66(4):701–4. doi:.https://doi.org/10.1590/S1807-59322011000400027  

17 Lee JJ , Waak K , Grosse-Sundrup M , Xue F , Lee J , Chipman D , et al. Global muscle strength but not grip strength predicts mortality and length of stay in a general population in a surgical intensive care unit. Phys Ther. 2012;92(12):1546–55. doi:.https://doi.org/10.2522/ptj.20110403  

18 Estenssoro E , Reina R , Canales HS , Saenz MG , Gonzalez FE , Aprea MM , et al. The distinct clinical profile of chronically critically ill patients: a cohort study. Crit Care. 2006;10(3):R89. doi:.https://doi.org/10.1186/cc4941  

19 Marchioni A , Fantini R , Antenora F , Clini E , Fabbri L . Chronic critical illness: the price of survival. Eur J Clin Invest. 2015;45(12):1341–9. doi:.https://doi.org/10.1111/eci.12547  

20Wesch C. Master Thesis: Developing and validating a multivariable tool to predict a prolonged length of stay in the ICU: A retrospective exploratory cohort study, in Institute of Nursing Science. 2017, University of Basel. 

21 Higgins PA , Daly BJ , Lipson AR , Guo SE . Assessing nutritional status in chronically critically ill adult patients. Am J Crit Care. 2006;15(2):166–76, quiz 177.  

22 Le Gall JR , Lemeshow S , Saulnier F . A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA. 1993;270(24):2957–63. doi:.https://doi.org/10.1001/jama.1993.03510240069035  

23 Cox CE . Persistent systemic inflammation in chronic critical illness. Respir Care. 2012;57(6):859–64, discussion 864–6. doi:.https://doi.org/10.4187/respcare.01719  

24 Riker RR , Picard JT , Fraser GL . Prospective evaluation of the Sedation-Agitation Scale for adult critically ill patients. Crit Care Med. 1999;27(7):1325–9. doi:.https://doi.org/10.1097/00003246-199907000-00022  

25 Riley RD , Hayden JA , Steyerberg EW , Moons KG , Abrams K , Kyzas PA , et al.; PROGRESS Group. Prognosis Research Strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10(2):e1001380. doi:.https://doi.org/10.1371/journal.pmed.1001380  

26 Moons KGM , Kengne AP , Woodward M , Royston P , Vergouwe Y , Altman DG , et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683–90. doi:.https://doi.org/10.1136/heartjnl-2011-301246  

27 Hemingway H , Croft P , Perel P , Hayden JA , Abrams K , Timmis A , et al.; PROGRESS Group. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346(feb05 1):e5595. doi:.https://doi.org/10.1136/bmj.e5595  

28 Lei Q , Chen L , Jin M , Ji H , Yu Q , Cheng W , et al. Preoperative and intraoperative risk factors for prolonged intensive care unit stay after aortic arch surgery. J Cardiothorac Vasc Anesth. 2009;23(6):789–94. doi:.https://doi.org/10.1053/j.jvca.2009.05.028  

29 Hein OV , Birnbaum J , Wernecke K , England M , Konertz W , Spies C . Prolonged intensive care unit stay in cardiac surgery: risk factors and long-term-survival. Ann Thorac Surg. 2006;81(3):880–5. doi:.https://doi.org/10.1016/j.athoracsur.2005.09.077  

30 Vincent J-L , Dubois M-J , Navickis RJ , Wilkes MM . Hypoalbuminemia in acute illness: is there a rationale for intervention? A meta-analysis of cohort studies and controlled trials. Ann Surg. 2003;237(3):319–34. doi:.https://doi.org/10.1097/01.SLA.0000055547.93484.87  

31 Lee JH , Kim J , Kim K , Jo YH , Rhee J , Kim TY , et al. Albumin and C-reactive protein have prognostic significance in patients with community-acquired pneumonia. J Crit Care. 2011;26(3):287–94. doi:.https://doi.org/10.1016/j.jcrc.2010.10.007  

32 de Groot V , Beckerman H , Lankhorst GJ , Bouter LM . How to measure comorbidity. a critical review of available methods. J Clin Epidemiol. 2003;56(3):221–9. doi:.https://doi.org/10.1016/S0895-4356(02)00585-1  

33 Rochon PA , Katz JN , Morrow LA , McGlinchey-Berroth R , Ahlquist MM , Sarkarati M , et al. Comorbid illness is associated with survival and length of hospital stay in patients with chronic disability. A prospective comparison of three comorbidity indices. Med Care. 1996;34(11):1093–101. doi:.https://doi.org/10.1097/00005650-199611000-00004  

34 Puntillo KA , Morris AB , Thompson CL , Stanik-Hutt J , White CA , Wild LR . Pain behaviors observed during six common procedures: results from Thunder Project II. Crit Care Med. 2004;32(2):421–7. doi:.https://doi.org/10.1097/01.CCM.0000108875.35298.D2  

35 Widyastuti Y , Stenseth R , Wahba A , Pleym H , Videm V . Length of intensive care unit stay following cardiac surgery: is it impossible to find a universal prediction model? Interact Cardiovasc Thorac Surg. 2012;15(5):825–32. doi:.https://doi.org/10.1093/icvts/ivs302  

36 Caironi P , Tognoni G , Masson S , Fumagalli R , Pesenti A , Romero M , et al.; ALBIOS Study Investigators. Albumin replacement in patients with severe sepsis or septic shock. N Engl J Med. 2014;370(15):1412–21. doi:.https://doi.org/10.1056/NEJMoa1305727  

37 Steyerberg EW , Moons KG , van der Windt DA , Hayden JA , Perel P , Schroter S , et al.; PROGRESS Group. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381. doi:.https://doi.org/10.1371/journal.pmed.1001381  

38 Kasotakis G , Schmidt U , Perry D , Grosse-Sundrup M , Benjamin J , Ryan C , et al. The surgical intensive care unit optimal mobility score predicts mortality and length of stay. Crit Care Med. 2012;40(4):1122–8. doi:.https://doi.org/10.1097/CCM.0b013e3182376e6d  

39Reuschenbach B, Mahler C, Ahlsdorf E. Pflegebezogene Assessmentinstrumente. Internationales Handbuch für Pflegeforschung und-praxis. 2011. Bern, Hans Huber.