Derivation and validation of a prediction model to establish nursing-sensitive quality benchmarks in medical inpatients: a secondary data analysis of a prospective cohort study

BACKGROUND
Hospitals are using nursing-sensitive outcomes (NSOs) based on administrative data to measure and benchmark quality of nursing care in acute care wards. In order to facilitate comparisons between different hospitals and wards with heterogeneous patient populations, proper adjustment procedures are required. In this article, we first identify predictors for common NSOs in acute medical care of adult patients based on administrative data. We then develop and cross-validate an NSO-oriented prediction model.


METHODS
We used administrative data from seven hospitals in Switzerland to derive prediction models for each of the following NSO: hospital-acquired pressure ulcer (≥ stage II), hospital-acquired urinary tract infection, non-ventilator hospital-acquired pneumonia and in-hospital mortality. We used a split dataset approach by performing a random 80:20 split of the data into a training set and a test set. We assessed discrimination of the models by area under the receiver operating characteristic curves. Finally, we used the validated models to establish a benchmark between the participating hospitals.


RESULTS
We considered 36,149 hospitalisations, of which 51.9% were male patients with a median age of 73 years (with an interquartile range of 59-82). Age and length of hospital stay were independently associated with all four NSOs. The derivation and validation models showed a good discrimination in the training (AUC range: 0.75-0.84) and in the test dataset (AUC range: 0.77-0.81), respectively. Variation among different hospitals was relevant considering the risk for hospital-acquired pressure ulcer (≥ stage II) (adjusted Odds ratio [aOR] range: 0.51 [95% CI: 0.38-0.69] - 1.65 [95% CI: 1.33-2.04]), the risk for hospital-acquired urinary tract infection (aOR range: 0.46 [95% CI: 0.36-0.58] - 1.45 [95% CI: 1.31-1.62]), the risk for non-ventilator hospital-acquired pneumonia (aOR range: 0.28 [95% CI: 0.09-0.89] - 2.87 [95% CI: 2.27-3.64]), and the risk for in-hospital mortality (aOR range: 0.45 [95% CI: 0.36-0.56] - 1.39 [95% CI: 1.23-1.60]).


CONCLUSION
The application of risk adjustment when comparing nursing care quality is crucial and enables a more objective assessment across hospitals or wards with heterogeneous patient populations. This approach has potential to establish a set of benchmarks that could allow comparison of outcomes and quality of nursing care between different hospitals and wards.

BACKG ROUND: Hospitals are using nursing-sensitive outcomes (NSOs) based on administrative data to measure and benchmark quality of nursing care in acute care wards. In order to facilitate comparisons between different hospitals and wards with heterogeneous patient populations, proper adjustment procedures are required. In this article, we first identify predictors for common NSOs in acute medical care of adult patients based on administrative data. We then develop and cross-validate an NSO-oriented prediction model. METHODS: We used administrative data from seven hospitals in Switzerland to derive prediction models for each of the following NSO: hospital-acquired pressure ulcer (≥ stage II), hospital-acquired urinary tract infection, non-ventilator hospital-acquired pneumonia and in-hospital mortality. We used a split dataset approach by performing a random 80:20 split of the data into a training set and a test set. We assessed discrimination of the models by area under the receiver operating characteristic curves. Finally, we used the validated models to establish a benchmark between the participating hospitals. RESULTS: We considered 36,149 hospitalisations, of which 51.9% were male patients with a median age of 73 years (with an interquartile range of 59-82). Age and length of hospital stay were independently associated with all four NSOs. The derivation and validation models showed a good discrimination in the training (AUC range: 0.75-0.84) and in the test dataset (AUC range: 0.77-0.81), respectively. Variation among different hospitals was relevant considering the risk for hospital-acquired pressure ulcer (≥ stage II) ( CONCLUSION: The application of risk adjustment when comparing nursing care quality is crucial and enables a more objective assessment across hospitals or wards with heterogeneous patient populations. This approach has potential to establish a set of benchmarks that could allow comparison of outcomes and quality of nursing care between different hospitals and wards.

Background
The relationship between higher levels of qualified nursing staff (registered nurses vs. non-registered nurses) and patient outcomes has been established.The relationship between higher levels of qualified nursing staff (registered nurses vs. non-registered nurses) and patient outcomes has been established [1][2][3][4][5]. Higher nurse staffing is, for example, associated with a lower incidence of hospital-acquired pressure ulcers, hospital-acquired pneumonia, and in-hospital mortality in medical inpatients [6,7]. These outcomes, which are influenced by nursing care, are generally understood as nursing-sensitive outcomes (NSO). NSOs have been defined as outcomes "that are relevant, based on nurses' scope and domain of practice, and for which there is empirical evidence linking nursing inputs and interventions to the outcomes'' [8,9].
The primary aim of this study was to identify predictors for four common NSOs of acute medical care of adult patients based on administrative data for the development and cross-validation of an NSO-related prediction model. The secondary aim of this study was to establish a set of benchmarksbetween seven Swiss hospitals using the four pre-specified NSOs. There is broad consensus that analyses and comparisons of NSO should occur at the ward level so that specific quality improvement actions can be taken [17]. Therefore, in addition to the overall hospital view presented here, which provides an impression of the overall comparison between medical departments, the proposed procedure will also be applicable at the ward level. This may allow comparisons within a department.

Design
This was a secondary data analysis of a prospective cohort study (In-HospiTOOL study) [18]. The "In-HospiTOOL" study was a quasi-experimental investigator-initiated, multicenter comparative effectiveness trial investigating the impact of an interprofessional discharge planning tool on length of hospital stay and other outcomes. The study established a representative benchmarking database to pro-mote the measurement of quality of care across different sized Swiss hospitals.

Study population and setting
We included all consecutively admitted adult (≥18 years) emergency patients from July 2017 to January 2019. Patients had to be hospitalized on a medical ward in one of the following seven secondary and tertiary care hospitals: Cantonal Hospital Aarau, Cantonal Hospital Baden, Cantonal Hospital Muensterlingen, Hospital Muri, Hospital Zofingen, Hospital Interlaken, and University Hospital Basel. We excluded patients with a length of hospital stay shorter than 24 hours or longer than 90 days and patients who have been treated in the intensive care unit (ICU) as part of their hospitalisation because the NSOs under study have not been developed for use in these patient populations.

Covariates of interest
We conducted a literature review to identify covariates that may affect the occurrence of an NSO. Based on previous studies, we used a basic set of adjustment variables for all models: age, gender, Charlson Comorbidity Index [19], length of hospital stay, Major Diagnostic Category (MDC) according to Diagnosis Related Groups (DRG), and type of hospital [20,21]. Further covariates related to individual NSOs are described in table 1. Data availability was a limiting factor in the selection of covariates.

Data collection
We used administrative data provided by the coding department as well as data from the electronic patient record of each of the participating hospitals between July 1, 2017, and January 31, 2019, as part of the In-HospiTOOL study. The datasets were linked at the case number level and data consistency between the different datasets was checked to ensure data quality. The administrative data comprises a set of uniform, clearly defined variables created by the Swiss Federal Statistical Office, that are therefore comparable among hospitals [43]. The diagnosis coding takes place after the hospitalisation has been completed, based on the discharge reports and the electronic patient record. To enable the coding of an outcome, the prerequisite had to be fulfilled that the outcome had been correctly recorded by a nurse or physician in the electronic patient chart or diagnosis list as part of the routine processes during hospitalisation. A single patient may have more than one hospital admission within the study period. Information on status of readmission to the same hospital according to the definition of Swiss-DRG, i.e. within 18 days after hospital discharge, was available for each hospitalisation. Length of hospital stay was calculated based on Swiss-DRG definition by day of admission and each subsequent day without the day of discharge.

Statistical analysis
We stratified sociodemographic characteristics and covariates by the four NSOs. Discrete variables were expressed as frequency (percentage) and continuous variables as medians and interquartile ranges (IQR). We used International Classification of Diseases (ICD) 10 codes to create variables to indicate whether patients experienced a NSO during their hospital stay using algorithms previously developed by Needleman, Buerhaus [44] and used in similar research projects [20,27]. For example, a hospital-acquired pressure ulcer (≥ stage II) was identified for any patient who had a secondary diagnosis code of L89 (inclusion criteria -see table 1) unless they had a length of stay <4 days, a major diagnostic category of 9 or a diagnosis code between G80-G83. These exclusion criteria are described in figure 1 (second level). The Charlson Comorbidity Index was calculated using the Stata command "charlson" [45].
To assess associations between predictors, covariates and NSOs, we performed logistic regression models. The area under the receiver operating characteristic curve (AUC) was used as a measure of discrimination. To ensure higher generalisability of the results and to avoid overfitting, we used a split dataset approach by performing a random 80:20 split of the data into a training set and a test set, respectively, while maintaining the proportion of outcomes within each of the two samples. The model fit by decile (estimated and observed probabilities) was plotted for each model (see figures A-1-A-4 in the appendix). We used likelihood-ratio tests (LR) to compare models with all predefined covariates with restricted models. For the bench-mark comparison, data from a single hospital were compared with those of the remaining six hospitals. For this purpose, we used logistic regression models and reported the crude and adjusted odds ratios as measures of association. We considered a two-tailed P-value at a 5% alpha level for statistical significance. Statistical analyses were performed using Stata 15.1 (StataCorp, College Station, TX, USA). All results are presented in an anonymous form to avoid identification of an individual hospital.

Ethics approval and consent to participate
All patients were informed by a flyer about their study participation after admission. As a quality improvement and control study, the institutional review board (IRB) of Northwestern Switzerland approved the study and waived the need for individual informed consent by formulating a declaration of no objection (AG/SO 2009/074 and EKNZ BASEC PB_2017-00449).

Study population
Of 45,146 hospitalisations, we excluded 8,997 with a length of hospital stay <24 h or >90 days, age <18 years, ICU stay or due to missing ICD-10 diagnosis resulting in 36,149 hospitalisations for the final analysis (see figure 1). There were no other missing data besides the ones mentioned. The median age of the overall population was 73 years (IQR 59-82); 51.9% were male and 80.3% were Swiss residents. The most common reasons for hospital admission regarding MDC were diseases of the circulatory system (n = 8873, 24.5%) and diseases of the respiratory system (n = 5008, 13.9%). Within this sample, 436 patients experienced a hospital-acquired pressure ulcer (≥ stage 2), 2,412 experienced a hospital-acquired urinary tract infection, 339 had a non-ventilator hospital-acquired pneumonia, and 1,525 died in the hospital. Further baseline characteristics are shown in table 2.

Prediction model derivation and validation
Most predefined covariates showed a significant association with the corresponding NSO (

Clinical outcomes
Length of hospital stay, median (IQR  All final models showed a good AUC in both the training set (AUC range: 0.75-0.84) and the test set (AUC range: 0.77-0.81), respectively. We did not observe a decrease in discrimination between the training and test sets (p >0.05) (table 3).

Benchmarking between different hospitals
While the prevalence of hospital-acquired pressure ulcer (≥ stage II) was lower in hospital A and higher in hospital F, there were no differences among the remaining hospitals. Hospitals C and D showed a higher prevalence of hospitalacquired urinary tract infection, while hospitals B, F, and G showed a lower prevalence and two hospitals (A and E) showed no difference compared to the other hospitals included. When comparing with the six remaining hospitals,   in-hospital mortality was lower in hospitals B, E, F and G and higher in hospitals A and C, respectively. Prevalence of hospital-acquired pneumonia was lower in hospitals A, E, F and G, but higher in hospital C ( figure 2).

Discussion
The key findings of this study are two-fold: first, the derived and validated prediction models showed a good discrimination ability for four well-studied NSOs (hospitalacquired pressure ulcer (≥ stage II), hospital-acquired urinary tract infection, non-ventilator hospital-acquired pneumonia, and in-hospital mortality). Second, we found relevant variation in risk of achieving an adverse NSO, suggesting that the outcome-related quality of nursing care differs among the investigated hospitals.
The indicators presented in this study may help to compare quality of care between NSOs of different hospitals using uniformly defined administrative data. This approach allows a timely evaluation of the results without additional effort for data generation. Internationally, mainly administrative data is used in the development of NSO-sets. A recent example is the nursing-sensitive outcome indicator suite for monitoring public patient safety in Western Australia, which was shown to be methodologically robust [46]. These results are largely consistent with the C-statistics in our study, confirming the external validity of our results.
Regarding hospital-acquired pressure ulcer (≥ stage II), we found little variance between the hospitals. On the one hand, the reason may be missing data due to underreporting [47]. Although, cross-sectional surveys by the ANQ showed similar but even lower prevalence rates (1.8%) compared to our study data (1.92%). It has been reported that administrative data do not provide valid data for this NSO due to several reasons [48][49][50]. This fact seems to be confirmed in our data, as we would expect prevalence rates of five percent or higher [51,52]. Therefore, before using administrative data to calculate the frequency of this NSO, we recommend reviewing the guideline-compliant documentation of the NSO in the electronic documentations as well as the coding procedure. On the other hand, it is important that the selected outcomes also have a clinically relevant frequency and variability. Otherwise, they offer no benefit in terms of quality development. Regarding hospital-acquired urinary tract infection the measured prevalence in our cohort (6.78%) is consistent with the expected prevalence (between 5.1 and 9.4% [53]), so we do not expect relevant under-or over-recording in the administrative data used.
With regards to the application of the developed method as a benchmark comparison by means of real-world data, two hospitals (B and E) showed a tendency towards lower risks in all NSOs, with some being statistically significant. Other hospitals, however, had higher risks of all (C) or almost all (D) NSOs, again with some reaching statistical significance. While our results should not be used for judging on a hospital's quality per se, it may provide an overview nevertheless which is notably more comprehensive than reporting individual outcomes based on cross-sectional surveys. For future studies, evaluations at the ward level are needed so that it can be investigated whether the prevalence and variation of the outcomes show relevant differences between the wards. Results on this level may support decision makers to reevaluate their pathways in providing care and thus to improve quality of nursing care. Nursingrelated reasons for the differing frequency of negative outcomes could be, for example, an inadequate skill and grade mix, staffing ratios, or insufficiently planned or standardisednursing processes [54].
The strength of this study is the large sample size. While this study covers hospitals from different regions in Switzerland, results might be generalisedat a larger level. However, the results of this study must be interpreted in the context of the study design. First, the use of administrative data is prone to information bias as hospitalisations will be selected according to the ICD-10 codes with the risk of misclassification and underreporting of diagnoses. Thus, frequency of NSO is usually underestimated, especially due to its low financial relevance [55,56]. Second, we did not have severity estimates of the hospitalised cases. Third, the non-experimental, observational design of our study limits the ability to draw a firm causal link. Fourth, since we do not have information about clinical parameters, we will be unable to account for unmeasured residual confounding and we were limited in selecting all appropriate covariates for the models. Fifth, we did not have data on nurse staffing in the study period. Sixth, external validation of our models is needed in follow-up studies.

Conclusion
The application of risk adjustment when comparing quality of nursing care enables a more objective assessment across hospitals or wards with heterogeneous patient populations. This approach has potential to establish a set of benchmarks allowing comparison of quality of nursing care between different hospitals or wards with manageable effort.

Data sharing statement
The data that support the findings of this study are available from the corresponding author, [DK], upon reasonable request. Figure S1: Model fit by decile (hospital-acquired pressure ulcer (≥ stage II).    Integer between 0 and 29: The CCI is a method of categorising comorbidities of patients based on the International Classification of Diseases (ICD) diagnosis codes. Each comorbidity category has an associated weight (from 1 to 6), based on the adjusted risk of mortality or resource use, and the sum of all the weights results in a single comorbidity score for a patient. A score of zero indicates that no comorbidities were found. The higher the score, the more likely the predicted outcome will result in mortality or higher resource use (19).