Skip to main navigation menu Skip to main content Skip to site footer

Original article

Vol. 152 No. 1112 (2022)

Derivation and validation of a prediction model to establish nursing-sensitive quality benchmarks in medical inpatients: a secondary data analysis of a prospective cohort study

  • Daniel Koch
  • Alexander Kutz
  • Thomas Volken
  • Claudia Gregoriano
  • Antoinette Conca
  • Michael Kleinknecht-Dolf
  • Philipp Schuetz
  • Beat Mueller
Cite this as:
Swiss Med Wkly. 2022;152:w30152


BACKGROUND: Hospitals are using nursing-sensitive outcomes (NSOs) based on administrative data to measure and benchmark quality of nursing care in acute care wards. In order to facilitate comparisons between different hospitals and wards with heterogeneous patient populations, proper adjustment procedures are required. In this article, we first identify predictors for common NSOs in acute medical care of adult patients based on administrative data. We then develop and cross-validate an NSO-oriented prediction model.

METHODS: We used administrative data from seven hospitals in Switzerland to derive prediction models for each of the following NSO: hospital-acquired pressure ulcer (≥ stage II), hospital-acquired urinary tract infection, non-ventilator hospital-acquired pneumonia and in-hospital mortality. We used a split dataset approach by performing a random 80:20 split of the data into a training set and a test set. We assessed discrimination of the models by area under the receiver operating characteristic curves. Finally, we used the validated models to establish a benchmark between the participating hospitals.

RESULTS: We considered 36,149 hospitalisations, of which 51.9% were male patients with a median age of 73 years (with an interquartile range of 59–82). Age and length of hospital stay were independently associated with all four NSOs. The derivation and validation models showed a good discrimination in the training (AUC range: 0.75–0.84) and in the test dataset (AUC range: 0.77–0.81), respectively. Variation among different hospitals was relevant considering the risk for hospital-acquired pressure ulcer (≥ stage II) (adjusted Odds ratio [aOR] range: 0.51 [95% CI: 0.38–0.69] – 1.65 [95% CI: 1.33–2.04]), the risk for hospital-acquired urinary tract infection (aOR range: 0.46 [95% CI: 0.36–0.58] – 1.45 [95% CI: 1.31–1.62]), the risk for non-ventilator hospital-acquired pneumonia (aOR range: 0.28 [95% CI: 0.09–0.89] – 2.87 [95% CI: 2.27–3.64]), and the risk for in-hospital mortality (aOR range: 0.45 [95% CI: 0.36–0.56] – 1.39 [95% CI: 1.23–1.60]).

CONCLUSION: The application of risk adjustment when comparing nursing care quality is crucial and enables a more objective assessment across hospitals or wards with heterogeneous patient populations. This approach has potential to establish a set of benchmarks that could allow comparison of outcomes and quality of nursing care between different hospitals and wards.


  1. Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K. Nurse-staffing levels and the quality of care in hospitals. N Engl J Med. 2002 May;346(22):1715–22.
  2. Kane RL, Shamliyan TA, Mueller C, Duval S, Wilt TJ. The association of registered nurse staffing levels and patient outcomes: systematic review and meta-analysis. Med Care. 2007 Dec;45(12):1195–204.
  3. Needleman J, Buerhaus P, Pankratz VS, Leibson CL, Stevens SR, Harris M. Nurse staffing and inpatient hospital mortality. N Engl J Med. 2011 Mar;364(11):1037–45.
  4. Aiken LH, Sermeus W, Van den Heede K, Sloane DM, Busse R, McKee M, et al. Patient safety, satisfaction, and quality of hospital care: cross sectional surveys of nurses and patients in 12 countries in Europe and the United States. BMJ. 2012 Mar;344 mar20 2:e1717.
  5. Aiken LH, Clarke SP, Sloane DM, Sochalski J, Silber JH. Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. JAMA. 2002 Oct;288(16):1987–93.
  6. Driscoll A, Grant MJ, Carroll D, Dalton S, Deaton C, Jones I, et al. The effect of nurse-to-patient ratios on nurse-sensitive patient outcomes in acute specialist units: a systematic review and meta-analysis. Eur J Cardiovasc Nurs. 2018 Jan;17(1):6–22.
  7. Oner B, Zengul FD, Oner N, Ivankova NV, Karadag A, Patrician PA. Nursing-sensitive indicators for nursing care: A systematic review (1997-2017). Nurs Open. 2021 May;8(3):1005–22.
  8. Doran D. Functional Status. Nursing-sensitive Outcomes: state of the science. Londom: Jones and Bartlett Publishers International. 2003.
  9. Doran DM, Harrison MB, Laschinger HS, Hirdes JP, Rukholm E, Sidani S, et al. Nursing-sensitive outcomes data collection in acute care and long-term-care settings. Nurs Res. 2006 Mar-Apr;55(2 Suppl):S75–81.
  10. Sim J, Crookes P, Walsh K, Halcomb E. Measuring the outcomes of nursing practice: A Delphi study. J Clin Nurs. 2018 Jan;27(1-2):e368–78.
  11. Sim J, Joyce-McCoach J, Gordon R, Kobel C. Development of a data registry to evaluate the quality and safety of nursing practice. J Adv Nurs. 2019 Sep;75(9):1877–88.
  12. Dubois CA, D’amour D, Brault I, Dallaire C, Déry J, Duhoux A, et al. Which priority indicators to use to evaluate nursing care performance? A discussion paper. J Adv Nurs. 2017 Dec;73(12):3154–67.
  13. Connolly D, Wright F. The nursing quality indicator framework tool. Int J Health Care Qual Assur. 2017 Aug;30(7):603–16.
  14. National Quality Forum. National Voluntary Consensus Standards for Nursing-Sensitive Care: An Initial Performance Measure Set2004.
  15. Hannah K, White PA, Kennedy MA, Hammell N, editors. C-HOBIC–standardized information to support clinical practice and quality patient care across Canada. NI 2012: 11th International Congress on Nursing Informatics, June 23-27, 2012, Montreal, Canada; 2012: American Medical Informatics Association.
  16. Dubois C-A, D’Amour D, Pomey M-P, Girard F, Brault I. Conceptualizing performance of nursing care as a prerequisite for better measurement: a systematic and interpretive review. 2013.
  17. Paulsen RA. Taking nurse staffing research to the unit level. Nurs Manage. 2018 Jul;49(7):42–8.
  18. Kutz A, Koch D, Conca A, Baechli C, Haubitz S, Regez K, et al. Integrative hospital treatment in older patients to benchmark and improve outcome and length of stay - the In-HospiTOOL study. BMC Health Serv Res. 2019 Apr;19(1):237.
  19. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.
  20. Twigg DE, Gelder L, Myers H. The impact of understaffed shifts on nurse-sensitive outcomes. J Adv Nurs. 2015 Jul;71(7):1564–72.
  21. Twigg DE, Geelhoed EA, Bremner AP, Duffield CM. The economic benefits of increased levels of nursing care in the hospital setting. J Adv Nurs. 2013 Oct;69(10):2253–61.
  22. Maas ML, Johnson M, Moorhead S. Classifying nursing-sensitive patient outcomes. Image J Nurs Sch. 1996;28(4):295–301.
  23. Griffiths P, Jones S, Maben J, Murrells T. State of the art metrics for nursing: a rapid appraisal. London: King’s College London; 2008.
  24. Recio-Saucedo A, Dall’Ora C, Maruotti A, Ball J, Briggs J, Meredith P, et al. What impact does nursing care left undone have on patient outcomes? Review of the literature. J Clin Nurs. 2018 Jun;27(11-12):2248–59.
  25. Schubert M, Glass TR, Clarke SP, Aiken LH, Schaffert-Witvliet B, Sloane DM, et al. Rationing of nursing care and its relationship to patient outcomes: the Swiss extension of the International Hospital Outcomes Study. Int J Qual Health Care. 2008 Aug;20(4):227–37.
  26. Schubert M, Clarke SP, Glass TR, Schaffert-Witvliet B, De Geest S. Identifying thresholds for relationships between impacts of rationing of nursing care and nurse- and patient-reported outcomes in Swiss hospitals: a correlational study. Int J Nurs Stud. 2009 Jul;46(7):884–93.
  27. Twigg D, Duffield C, Bremner A, Rapley P, Finn J. The impact of the nursing hours per patient day (NHPPD) staffing method on patient outcomes: a retrospective analysis of patient and staffing data. Int J Nurs Stud. 2011 May;48(5):540–8.
  28. Butler M, Collins R, Drennan J, Halligan P, O’Mathúna DP, Schultz TJ, et al. Hospital nurse staffing models and patient and staff-related outcomes. Cochrane Database Syst Rev. 2011 Jul;(7):CD007019.
  29. Soban LM, Hempel S, Munjas BA, Miles J, Rubenstein LV. Preventing pressure ulcers in hospitals: A systematic review of nurse-focused quality improvement interventions. Jt Comm J Qual Patient Saf. 2011 Jun;37(6):245–52.
  30. Lankshear AJ, Sheldon TA, Maynard A. Nurse staffing and healthcare outcomes: a systematic review of the international research evidence. ANS Adv Nurs Sci. 2005 Apr-Jun;28(2):163–74.
  31. Fuchs MA, Sexton DJ, Thornlow DK, Champagne MT. Evaluation of an evidence-based, nurse-driven checklist to prevent hospital-acquired catheter-associated urinary tract infections in intensive care units. J Nurs Care Qual. 2011 Apr-Jun;26(2):101–9.
  32. Bernard MS, Hunter KF, Moore KN. A review of strategies to decrease the duration of indwelling urethral catheters and potentially reduce the incidence of catheter-associated urinary tract infections. Urol Nurs. 2012 Jan-Feb;32(1):29–37.
  33. Ausserhofer D, Schubert M, Desmedt M, Blegen MA, De Geest S, Schwendimann R. The association of patient safety climate and nurse-related organizational factors with selected patient outcomes: a cross-sectional survey. Int J Nurs Stud. 2013 Feb;50(2):240–52.
  34. Kovner C, Jones C, Zhan C, Gergen PJ, Basu J. Nurse staffing and postsurgical adverse events: an analysis of administrative data from a sample of U.S. hospitals, 1990-1996. Health Serv Res. 2002 Jun;37(3):611–29.
  35. Cho SH, Ketefian S, Barkauskas VH, Smith DG. The effects of nurse staffing on adverse events, morbidity, mortality, and medical costs. Nurs Res. 2003 Mar-Apr;52(2):71–9.
  36. Carryer JB, Diers D, McCloskey B, Wilson D. Effects of health policy reforms on nursing resources and patient outcomes in New Zealand. Policy Polit Nurs Pract. 2010 Nov;11(4):275–85.
  37. Weissman JS, Rothschild JM, Bendavid E, Sprivulis P, Cook EF, Evans RS, et al. Hospital workload and adverse events. Med Care. 2007 May;45(5):448–55.
  38. Mattke S, Needleman J, Buerhaus P, Stewart M, Zelevinsky K. Evaluating the role of patient sample definitions for quality indicators sensitive to nurse staffing patterns. Med Care. 2004 Feb;42(2 Suppl):II21–33.
  39. Tourangeau AE, Giovannetti P, Tu JV, Wood M. Nursing-related determinants of 30-day mortality for hospitalized patients. Can J Nurs Res. 2002 Mar;33(4):71–88.
  40. Estabrooks CA, Midodzi WK, Cummings GG, Ricker KL, Giovannetti P. The impact of hospital nursing characteristics on 30-day mortality. Nurs Res. 2005 Mar-Apr;54(2):74–84.
  41. Tourangeau AE, Doran DM, McGillis Hall L, O’Brien Pallas L, Pringle D, Tu JV, et al. Impact of hospital nursing care on 30-day mortality for acute medical patients. J Adv Nurs. 2007 Jan;57(1):32–44.
  42. Kelly DM, Kutney-Lee A, McHugh MD, Sloane DM, Aiken LH. Impact of critical care nursing on 30-day mortality of mechanically ventilated older adults. Crit Care Med. 2014 May;42(5):1089–95.
  43. Federal Statistical Office (FSO). Variablen der Medizinischen Statistik, Spezifikationen gültig ab 1.1.2019. 2018.
  44. Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K. Nurse staffing and patient outcomes in hospitals. Final report. Rockville, MD: Health Resources and Services Administration, US Department of Health and Human Services. 2001.
  45. Stagg V. CHARLSON: Stata module to calculate Charlson index of comorbidity. 2017.
  46. Twigg DE, Pugh JD, Gelder L, Myers H. Foundations of a nursing-sensitive outcome indicator suite for monitoring public patient safety in Western Australia. Collegian. 2016;23(2):167–81.
  47. Smith IL, Nixon J, Brown S, Wilson L, Coleman S. Pressure ulcer and wounds reporting in NHS hospitals in England part 1: audit of monitoring systems. J Tissue Viability. 2016 Feb;25(1):3–15.
  48. Swiss National Association for Quality Development in Hospitals and Clinics. Sturz und Dekubitus Akutsomatik Erwachsene - Nationaler Vergleichsbericht2019.
  49. Ho C, Jiang J, Eastwood CA, Wong H, Weaver B, Quan H. Validation of two case definitions to identify pressure ulcers using hospital administrative data. BMJ Open. 2017 Aug;7(8):e016438.
  50. Backman C, Vanderloo SE, Miller TB, Freeman L, Forster AJ. Comparing physical assessment with administrative data for detecting pressure ulcers in a large Canadian academic health sciences centre. BMJ Open. 2016 Oct;6(10):e012490.
  51. Tubaishat A, Papanikolaou P, Anthony D, Habiballah L. Pressure Ulcers Prevalence in the Acute Care Setting: A Systematic Review, 2000-2015. Clin Nurs Res. 2018 Jul;27(6):643–59.
  52. Tomova-Simitchieva T, Akdeniz M, Blume-Peytavi U, Lahmann N, Kottner J. [The Epidemiology of Pressure Ulcer in Germany: systematic Review]. Gesundheitswesen. 2019 Jun;81(6):505–12.
  53. Öztürk R, Murt A. Epidemiology of urological infections: a global burden. World J Urol. 2020 Nov;38(11):2669–79.
  54. Kalánková D, Kirwan M, Bartoníčková D, Cubelo F, Žiaková K, Kurucová R. Missed, rationed or unfinished nursing care: A scoping review of patient outcomes. J Nurs Manag. 2020 Nov;28(8):1783–97.
  55. Jhung MA, Banerjee SN. Administrative coding data and health care-associated infections. Clin Infect Dis. 2009 Sep;49(6):949–55.
  56. Quan H, Eastwood C, Cunningham CT, Liu M, Flemons W, De Coster C, et al.; IMECCHI investigators. Validity of AHRQ patient safety indicators derived from ICD-10 hospital discharge abstract data (chart review study). BMJ Open. 2013 Oct;3(10):e003716.
  57. Padula WV, Makic MB, Mishra MK, Campbell JD, Nair KV, Wald HL, et al. Comparative effectiveness of quality improvement interventions for pressure ulcer prevention in academic medical centers in the United States. Jt Comm J Qual Patient Saf. 2015 Jun;41(6):246–56.
  58. Berlowitz DR, Brandeis GH, Morris JN, Ash AS, Anderson JJ, Kader B, et al. Deriving a risk-adjustment model for pressure ulcer development using the Minimum Data Set. J Am Geriatr Soc. 2001 Jul;49(7):866–71.
  59. Berlowitz DR, Ash AS, Brandeis GH, Brand HK, Halpern JL, Moskowitz MA. Rating long-term care facilities on pressure ulcer development: importance of case-mix adjustment. Ann Intern Med. 1996 Mar;124(6):557–63.
  60. Letica-Kriegel AS, Salmasian H, Vawdrey DK, Youngerman BE, Green RA, Furuya EY, et al. Identifying the risk factors for catheter-associated urinary tract infections: a large cross-sectional study of six hospitals. BMJ Open. 2019 Feb;9(2):e022137.
  61. Li F, Song M, Xu L, Deng B, Zhu S, Li X. Risk factors for catheter-associated urinary tract infection among hospitalized patients: A systematic review and meta-analysis of observational studies. J Adv Nurs. 2019 Mar;75(3):517–27.
  62. Di Pasquale M, Aliberti S, Mantero M, Bianchini S, Blasi F. Non-Intensive Care Unit Acquired Pneumonia: A New Clinical Entity? Int J Mol Sci. 2016 Feb;17(3):287.
  63. Sopena N, Heras E, Casas I, Bechini J, Guasch I, Pedro-Botet ML, et al. Risk factors for hospital-acquired pneumonia outside the intensive care unit: a case-control study. Am J Infect Control. 2014 Jan;42(1):38–42.
  64. Federal Statistical Office (FSO). Variablen der Medizinischen Statistik Spezifikationen gültig ab 1.1.2019 2019 14.01.2021. Available from:
  65. Sabrià M, Sopena N. Nosocomial Pneumonia Outside the Intensive Care Unit. Clin Pulm Med. 2011;18(5):215–21.

Most read articles by the same author(s)

1 2 3 > >>