External validation of the 80 + score and comparison with three clinical scores identifying patients at least 75 years old at risk of unplanned readmission within 30 days after discharge

Schwab Camilleabc, Le Moigne Alexisa, Fernandez Christineacd, Durieux Pierreef, Sabatier Brigittebe, Korb-Savoldelli Virginieab a Université Paris-Sud, Faculté de Pharmacie, Châtenay-Malabry, France b Service de Pharmacie, Hôpital Européen Georges Pompidou, Assistance Publique – Hôpitaux de Paris, France c Sorbonne Universités, UPMC Université Paris 06, UMR-S 1136, Institut Pierre Louis d’Epidémiologie et de Santé Publique, Paris, France d Service de Pharmacie, Hôpital Saint-Antoine, Assistance Publique – Hôpitaux de Paris, France e INSERM UMR 1138, Equipe 22, Centre de Recherche des Cordeliers, Universités Paris 5 et 6, France f Département de Santé Publique et Informatique Médicale, Hôpital Européen Georges Pompidou, Assistance Publique – Hôpitaux de Paris, France


Introduction
The elderly (people over 75 years of age or those over 65 with comorbidities) are at high risk of potentially avoidable hospital readmission.A potentially avoidable readmission is an unplanned readmission caused by known morbidities and occurring within 30 days of discharge [1].About 20% of elderly inpatients are subject to potentially avoidable readmission [2].As there is no consensus on a classification of avoidable readmissions [3,4], all unplanned readmissions are considered.These readmissions are deleterious for the elderly: a readmission increases the risk of dependence and functional or psychosocial decline [5].Moreover, they increase the risk of decompensation of other comorbid conditions, thus increasing the frailty of elderly patients [6].In the United States, costs associated with these readmissions amount to 17.5 billion dollars [7]; it is therefore, a public health priority to reduce readmission rates of the elderly.According to the French Ministry of Health, it is neither "necessary nor efficient to intervene for every patient" aged 75 and over to reduce the readmission rate [8].Therefore, it is crucial to develop and apply tools to identify patients at risk of readmission and then to set up interventions.Clinical scores are tools that make it possible to identify elderly inpatients at risk of readmission.In the literature, there are many scores that identify elderly inpatients at risk of readmission [9].Among these, the LACE index, created in 2010 [10], the HOSPITAL score, 2010 [11] or the Triage Risk Screening Tool (TRST), 2003 [12] can easily be calculated from data in medical records and are widely used.The purpose of this study was to implement the chosen score in the hospital's medical soft-ware, in order to generate an automatic alert on high-risk patients.In 2015, Alassaad et al. constructed a clinical score including iatrogenic factors (prescription of drugs for peptic ulcer, opioid drugs or non-tricyclic antidepressants) [13].This focus on iatrogenic factors is interesting, as 10 to 30% of readmissions are drug-related [14], and iatrogenic factors can be eliminated by adjusting the prescription.This "80+ score", which focuses on iatrogenic factors, must be compared with already existing scores focusing on demographic and sociological factors or comorbidities: the LACE index [10], the HOSPITAL score [11] and the TRST [12].Clinical scores require internal and external validation.Internal validation means testing the score in the population used to create it.It describes the performance of the score (calibration and discrimination).Then, prior to general application, the score has to be externally validated -tested in a new population that ensures the reproducibility of the score.According to the TRIPOD statement [15], external validation may use types of participants different from the development population.Thus, we choose to externally validate the LACE and HOSPITAL scores, which were developed in general adult population, on an elderly population.Unlike the LACE index [10], the HOSPITAL score [11] and the TRST [12], the 80+ score has not been externally validated to date.The main objective of this study was to carry out an external validation of the 80+ score and to compare it with the three other scores.

Study design
We used the patient data from a monocentric, retrospective, matched, case-control study using the clinical data warehouse [16] of a French university hospital, the Hôpital européen Georges Pompidou.This is a teaching hospital of 795 beds with 24 clinical departments (15 medical wards and 9 surgical units).The clinical information system is coupled with a clinical data warehouse, which makes reuse of healthcare data and clinical research possible.The main objective of the first study was to research iatrogenic risk factors for unplanned readmissions.

Participants
Elderly patients are people over 75 years of age or those over 65 with comorbidities; in order to avoid the inclusion of patients between 65 and 75 years old without comorbidities, we included only patients over 75 years old.Patients were admitted to hospital, in a medical or surgical ward, between 1 October 2014 and 30 September 2015.Cases were elderly patients who experienced an unscheduled readmission within 30 days after the index discharge.Readmissions had to be via the accident and emergency department (AED) to be considered unscheduled.The patients were either discharged after the AED visit or hospitalised.The control group consisted of elderly patients who had not had any unscheduled readmissions during the 30 days after the index discharge.All controls who died during the index hospitalisation were excluded.Cases and controls were randomly matched for sex and age.We excluded all patients lost to follow-up who were potentially dead after the index admission.

Study size
The sample size of our study was calculating according to Collins et al. [17].They suggested that, to externally validate prognostic models, a minimum of 100 events and ideally 200 (or more) events are required.

Presentation of the four scores (table 1)
A clinical score is a tool that can be used by hospital practitioners to guide medical interventions.They are built with a series of variables corresponding to medical background, and sociological, biological or clinical observations.For each variable, the patient is attributed a number of points.Points are added up, giving the final value of the score.The scoring system ranges from a minimum to a maximum, where the likelihood of readmission increases with the calculated value of the score.The 80+ score [13] is a point score system used for risk estimation that was developed at the Uppsala University Hospital in 2015.It includes seven variables and ranges from −2 to 10.A patient with a score greater than 3 has an 80% risk of readmission.The 80+ score was internally validated in a prospective randomised control trial [13] and showed good performances, with a c-statistic of 0.72 (95% confidence interval [CI] 0.66-0.77)and a good calibration.However, to our knowledge, this score has never been externally validated, unlike the other three scores.The LACE index [10] is a point score initially conceived in Ontario, Canada in 2010.It ranges from 0 to 44.A patient with a score greater than 10 is considered at high risk for unplanned hospital readmission.This score was derived and validated in a prospective cohort study.This internal  validation showed good discrimination, with a c-statistic of 0.684 [10].
The HOSPITAL score [18], developed in Massachusetts, USA in 2013, is a score built with seven variables forming the acronym HOSPITAL.The score ranges from 0 to 13.A patient is considered at high risk of readmission when he or she has a score equal or greater than 7.The HOSPI-TAL score was internally validated in a retrospective cohort study [18] and showed good performances with a cstatistic of 0.71.
The TRST [12] is a five-item clinical prediction, a rule designed for rapid administration by the triage nurse in an AED after minimal training; it was developed in Cleveland, USA in 2003.The score ranges from 0 to 5. A patient is at high risk of readmission if the score is greater than 2.
The TRST was internally validated in a prospective cohort study [12] and showed a good performances with an area under the receiver operating characteristic curve (AUC of the ROC curve) of 0.72 for readmissions at 30 days.The table 2 shows the differences between the development data and our validation data in setting, eligibility criteria and outcome.

Data collection
All patient data and medical history records were retrieved using the clinical data warehouse.An ethics committee approved the study protocol (CERHUPO, CDW_2015_0023).For biological data (estimated glomerular filtration rate [eGFR], haemoglobin and sodium levels), the last available analyses before discharge were chosen.For medication data (drug for peptic ulcer, opioid, non-tricyclic antidepressant and polypharmacy), we used the prescription at discharge.For other data, all available medical records were analysed to find the patient's medical background (pulmonary and malignant disease, Charlson index and history of hospital admission), mental and physical state of the patient (cognition, falls, staff recommendation) and procedures performed during the hospitalisation.Medical records are completed by healthcare professionals.It is recognised that in medical records, doctors enumerate comorbidities and do not always record specific data that are unknown.We therefore considered that the absence of a specific piece of information (such as asthma, cancer history) in the medical report meant the absence of the condition.

Statistical analysis
For data management, the outliers were corrected after a second check of the medical record: when an outlier was detected, the medical record was read to confirm the variable.The missing data of qualitative variables were imputed according to the context.For example, elderly patients usually live at home, and not in a nursing home.Thus, if it was not mentioned that the patient lived in a nursing home, he or she was assumed to be living at home.The data were then corrected.The quantitative variables were imputed by the median (less than 20% of missing data) or by multiple imputation (more than 20% of missing data).All scores were calculated by adding the points of each given variable.Cases and controls were analysed for each variable of each score.The qualitative variables were compared using Pearson's χ 2 test.A univariate analysis was performed with a t-test to compare scores between the two groups.Discrimination is the ability of the score to separate cases from control groups.To determine the discriminatory ability of the scores, the c-statistic was calculated.The c-statistic gives the probability that a randomly selected patient has a higher score than a patient not -readmitted.A value below 0.5 indicates that the score has poor discriminatory ability.A score has a good discrimination when the c-statistic ranges from 0.7 to 1.It has fair discrimination when ranging from 0.5 to 0.7.The graphic representation of the c-statistic is the receiving operating characteristic (ROC) curve.The ROC curves of the four scores were drawn, and the areas under the curve (AUCs) were calculated.Calibration shows the ability of the model to generate probabilities that match the observed rates.We used the Hosmer-Lemeshow χ 2 test to assess the calibration of the selected scores.If the p-value is greater than 0.05, the score has a good calibration.We then estimated the sensitivity, the specificity, the positive predictive value, the negative predictive value, the positive likelihood ratio and the negative likelihood ratio.Analyses were performed with the R software (R version 3.3.2),with a significance level of α = 0.05.

Results
From 1 September 2014 to 31 October 2015, 6574 elderly patients meeting our criteria were admitted to the hospital.Within 30 days after discharge of the index admission, 456 patients (6.4%) were readmitted, and 6118 were not.These 456 patients were randomly matched with 456 controls among the 6118 patients, creating 456 case-control pairs.Of those 456 pairs, 237 controls were lost during follow- up, resulting in a final pool of 219 pairs.The flow of participants is described in figure 1.

Patient characteristics
A total of 438 patients were included, whom 52% were women.The mean age was 84.4 years (standard deviation [SD] 5.90).On admission, 46% of the patients were in a re- lationship, 51% were single (3% missing values).The patients' baseline characteristics are listed in table 3.

Score results
Using a t-test, the means of the scores were compared between cases and controls (table 4).The 80+ score, the LACE index and the HOSPITAL score had p-values of 0.87, 0.24 and 0.60, respectively, meaning that there was no significant difference between cases and controls.In contrast, for the TRST, the mean score of the cases was significantly different from the mean score of the controls (p <0.001).

Diagnostic values of the scores
The discrimination was calculated with the AUC of the ROC curve (table 5).The 80+ score had the lowest AUC (0.506) and TRST had the highest (0.589; fig.2).
The calibration was estimated with the Hosmer-Lemeshow χ 2 test (table 6).

Main results
In our study, the new 80+ score showed an AUC of the ROC curve of 0.506.Thus it has not demonstrated better discrimination than the three other scores.
The TRST performed best in identifying elderly patients at risk for readmission within 30 days of discharge.Even though they showed fair calibration, the 80+ score, the HOSPITAL score and the LACE index had not strong enough discriminatory power.The four scores had all previously been internally validated.In our study we observed that the calibration and validation of these four scores were not as good as had been reported in the internal validation studies.One of the possible reasons for this difference was the selection of our patients, who were all alive 30 days after discharge, whereas most of these other studies also included deaths, meaning that patients in our study were in a better health.Indeed, in a validation study of the LACE index, performed at the Department of Medicine for the Elderly Cambridge University Hospital [19], the LACE index also showed poor performances in predicting readmission, but better performance in predicting death.This supports the hypothesis that these scores perform better at identifying both patients at risk of readmission and at risk of dying.The HOSPITAL score, an easy-to-use score, was externally validated in a large international (USA, Canada, Switzerland, Israel) retrospective cohort study [11] of 121,136 patients, and showed very good calibration and discrimination.Our study included only 219 control-case pairs, and therefore probably did not have the same statistical power.The HOSPITAL score was also validated in a Swiss population of patients aged 50 years old and older [20].In this second validation study, with a population more similar to ours, showed very good calibration (p = 0.89) and discrimination (c-statistic = 0.72).It was a prospective cohort study and the first outcome was either death or unplanned readmission within 30 days, whereas our primary outcome was only unplanned readmission within 30 days after discharge.As we had an older population (inpatients 75 years and older vs 50 years and older for Aubert et al. [20]), the death rate might have been higher.Moreover, Cooksley et al. [21] have shown that the discriminatory power of HOSPITAL and LACE scores decreased with increasing age.The 80+ score is a recent clinical score identifying elderly patients at risk of readmission.Its strength is that it takes into account medication.Thus, if the prescription is adjusted during the hospitalisation, the risk of readmission is decreased.The same applies to the TRST, which takes into account the number of drugs, so that the prescription can also be adjusted according to the benefit-risk balance of the treatment.Conversely, the LACE index and the HOS-PITAL score do not take into account prescriptions.They focus on admission characteristics (index admission type, length of stay and history of hospitalisation).In a recent systematic review, Pedersen et al. identified length of stay and comorbid conditions as risk factors for readmission [22].Thus, some drugs have an impact on the benefit-risk ratio, in favour of risk, when administered to elderly patients.There are lists of potentially inappropriate medica- Original article tions (PIMs) [23][24][25], although the results of studies on PIMs and readmissions are conflicting.A study showed that the prescription of PIMs had no effect on readmission [26], but another study showed that the prescription of benzodiazepines, some of which (long-acting benzodiazepines) are listed as PIMs, was associated with a higher readmission risk (odds ratio 1.23) [27].This might explain the lower performance of the 80+ score.When the 80+ score was internally validated, it showed better discriminatory ability than in our study.This difference may be due to the difference of population in the two studies.Alassaad et al. [13] included patients aged 80 years and older and acutely admitted to an internal medicine ward at a university hospital, whereas we included patients aged 75 years and older admitted (acutely or not) to medical and surgical wards.

Limitations
We acknowledge that our study has limitations.It was a single-centre retrospective case-control study.Sex and age were matched in order to select the controls and obtain a relatively homogeneous population.Matching on length of stay or comorbidities, which are items of the LACE and HOSPITAL scores, would not have improved the power of the study.All the data were retrieved from an electronic database and we made the assumption that if a specific condition was not in the medical record, it was absent.Therefore, it is less powerful than a prospective study with a questionnaire where each variable of the score would be filled in with either a negative or positive answer.In the same way, we excluded elderly under 75 years old, i.e., we ignored patients aged 65 to 75 years with comorbidities.Deaths within 30 days and patients lost to follow-up were not included.Another limitation due to the use of an electronic database is that we did not consider readmissions that did not originate from an AED visit or that occurred in another hospital.It could explain why the readmission rate was relatively low (6.4%).

Recommendations for further research
In the cohort of this study, the 80+ score showed poorer discrimination than the three other scores.All the four scores analysed showed less ability to identify potentially avoidable readmission of elderly patients than in other studies found in the literature.One of the perspectives of this study would be to identify patients at risk of readmission using the TRST and then set a medication reconciliation protocol with case and control groups and finally assess the efficiency of such interventions in a population at risk.

Disclosure statement
No financial support and no other potential conflict of interest relevant to this article was reported.
rate (eGFR) Lengths of hospitalisation Haemoglobin level at discharge History of cognitive impairment Level of social support Discharge from an Oncology unit Pulmonary disease (asthma or chronic obstruction pulmonary disease) Acuteness of the admission Sodium level at discharge Difficulty walking/transferring or recent falls Malignant disease Procedures during hospital stay Prescription of a drug for peptic ulcer or gastro-oesophageal reflux disease Comorbidities of patients Index admission type Taking five or more medications Prescription of an opioid drug Number of hospital admissions during the previous year Prescription of an antidepressant drug (except tricyclic antidepressant) AED admissions Length of hospitalisation AED use in previous 30 days or hospitalisation in previous 90 days AED = accident and emergency department; TRST = Triage Risk Screening Tool Original article Swiss Med Wkly.2018;148:w14624 Swiss Medical Weekly • PDF of the online version • www.smw.chPublished under the copyright license "Attribution -Non-Commercial -No Derivatives 4.0".No commercial reuse without permission.See http://emh.ch/en/services/permissions.html.

Figure 1 :
Figure 1: Patient recruitment and assignment of controls.

Table 1 :
Items comprising each of the four scores.

Table 2 :
Differences between development data and validation data.
Swiss Medical Weekly • PDF of the online version • www.smw.chPublished under the copyright license "Attribution -Non-Commercial -No Derivatives 4.0".No commercial reuse without permission.See http://emh.ch/en/services/permissions.html.

Table 5 :
Discriminatory ability of the four scores analysed with c-statistics.
AUC = area under the receiver operating characteristic curve; TRST = Triage Risk Screening Tool

Table 6 :
Calibration of the four scores analysed with the Hosmer-Lemeshow χ 2 test.