Comparison of Standardized Mortality Ratios in seven Dutch EDs based on presenting complaints

Abstract

Background

Comparison of emergency departments (EDs) becomes more important, but differences are difficult to interpret because of the heterogeneity of the ED population regarding reason for ED presentation. The aim of this study was two-fold: First to compare patient characteristics (including diagnoses) across 7 EDs. Secondly, to compare Standardized Mortality Ratios (SMRs) across 7 EDs and in subgroups of ED patients categorized by presenting complaints (PCs).

Methods

Observational multicenter study including all consecutive visits of 7 Dutch (two tertiary care centre and 5 teaching hospitals) EDs. Patient characteristics, including PCs as part of triage systems, and SMRs (observed divided by expected in-hospital mortality) per ED and for the most common PCs (PC-SMRs) were compared across EDs and presented as funnel plots. The expected mortality was calculated with a prediction model, which was developed using multivariable logistic regression in the overall population and for PCs separately. Demographics, disease severity, diagnoses, proxies for comorbidity and complexity, and PCs (overall population only) were incorporated as covariates.

Results

We included 693,289 ED visits from January 1, 2017 to June 31, 2023, with a median age of 56 years, of which 47.9% were women and 1.9% died. Patient characteristics varied markedly among EDs. Expected mortality was similar in prediction models with or without diagnoses as covariate. SMRs differed across EDs, ranging from 0.80 to 1.44. All EDs had SMRs within the 95%-Confidence Intervals of the funnel plot apart from one ED, which had an higher than expected SMR. However, PC-SMRs showed more variation and more EDs had SMRs falling outside the funnel, either higher or lower than expected. The ranking of SMRs across EDs was PC-dependent and differences across EDs are present only for specific PC-SMRs, such as in “dyspnea” and “feeling unwell”.

Conclusion

In summary, patient characteristics and mortality varied largely across Dutch EDs, and expected mortality across EDs is well assessed in PC subgroups without adjustment for final diagnoses. Differences in SMRs across EDs are PC-dependent. Future studies should investigate reasons of the differences in PC-SMRs across EDs and whether PC-targeted quality improvement programs can improve outcomes.

Introduction

Background

The organization of acute care is evolving to provide patients with the best clinical outcomes. Part of this process is the development of quality registries, including those for emergency departments (EDs) [1,2,3,4]. Quality registries can be used to benchmark individual EDs, develop best practices, and evaluate whether implementation of guidelines and quality indicators leads to better patient outcomes in continuous Plan-Do-Check-Act (PDCA) cycles. One way to benchmark quality of ED care is by comparison of the Standardized Mortality Ratio (SMR), which is the observed mortality divided by expected mortality. The SMR has been used extensively for comparing hospitals and Intensive Care Units (ICU’s) and could also be valuable for comparing EDs, yet it is not regularly applied [5, 6]. Underlying is that an overall ED SMR may not reveal differences in quality of care in subgroups. Clearly, SMRs may be different in high-risk diagnoses like sepsis compared to a sprained ankle. Therefore, quality registries for EDs must take into account this substantial heterogeneity of ED visits and the resulting processes of care in EDs in order to give EDs helpful information about which subgroups to focus on if EDs are to improve their outcomes, i.e., SMRs. Many quality registries therefore benchmark per diagnosis [7].

However, definite diagnoses are mostly unavailable during ED presentation and in many patients not even after ED discharge. Sometimes, diagnoses that seem obvious at presentation prove to be wrong in retrospect [8, 9]. Moreover, in an ageing society, patients increasingly present to the ED with nonspecific complaints [10], with conditions that are oligo- or asymptomatic [11], and with multiple conditions simultaneously [12].

Presenting complaints (PCs) are available from the beginning of the patient’s journey through acute care and are used in ED triage systems to stratify patients by risk of death and expected resource utilization [13,14,15]. PCs help understand ED care among subgroups similar in disease severity, resource use, and prognosis [16,17,18]. Although PCs have been studied in triage, there has been no research on comparing SMRs in this context [19, 20]. Utilizing PC-specific SMRs may be more effective than overall SMRs, as it identifies distinct subgroups needing targeted improvement. Moreover, unlike diagnoses, PCs allow the entire ED process, from enrollment until discharge, to be included in the improvement initiative. The purpose of this study is two-fold: first, to compare patient characteristics (including diagnoses) across 7 EDs. Secondly, to compare overall SMRs using in-hospital mortality across EDs and SMRs of subsets of patients with the 6 most common PCs, in the Netherlands.

Methods

Study design and setting

This is a retrospective, observational study using data from the Netherlands Emergency Department Evaluation Database (NEED), the national quality registry for EDs in the Netherlands (www.Stichting-NEED.nl). Data was collected from January 1st 2017 until June 30th 2023 (see table S1, supplement 1 for inclusion periods per ED). At the time of study the NEED contained data from seven Dutch EDs located in different regions spread across the Netherlands: two tertiary care hospitals and five general hospitals. The Netherlands has a total of 80 EDs that are available 24/7. ED patients present themselves directly or are referred by a general practitioner, hospital specialist, or Emergency Medical Services (EMS). Only in the case of an anticipated neonatological- or obstetrics presentation or an ST-segment elevation myocardial infarction (STEMI), patients almost always bypass the ED because of specific expertise or immediate intervention. Presenting complaints were recorded as part of triage systems using a focused clinical assessment upon the patient’s arrival in the ED, performed by a specifically trained nurse for this purpose. The EDs included in the database used different triage systems to register PCs: Manchester Triage System (MTS), Netherlands Triage System (NTS) and Emergency Severity Index (ESI) [21,22,23]. Table S1 (supplement 1) also shows which triage system was used in which ED for what period. To use both MTS, NTS, and ESI presenting complaints in the analyses, we combined the MTS, NTS, and ESI presenting complaints into one combined list of 52 complaints, as shown in Table S2 (supplement 1). Diagnoses are recorded upon patient discharge from the hospital according to the International Classification of Diseases and Related Health Problems (ICD-10 codes) [24]. The need for individual informed consent was waivered by the medical ethics committee of the LUMC and registered in the Netherlands Trial Registry (NTR) with number NL8743.

Patients

All consecutive ED visits with a registered PC in the NEED database were included in the study unless patients objected to participate in the quality registry. Specifically, patients who died upon arrival or during their stay in the ED were excluded because these fatalities, such as out-of-hospital cardiac arrests, are predominantly indicative of pre-existing or untreatable medical problems and did not, or hardly reflect care provided in the ED.

Measurements

Details on the collection of data within the NEED registry have been published previously [25, 26]. In supplement 2 a comprehensive listing of all used variables from the NEED and their values can be found. The six most common PCs and the five most common diagnosis groups per PC from the collection of visits from all participating EDs were used for the analyses.

Outcomes

The primary outcome was the Standardized Mortality Ratio (SMR) as calculated by the observed in-hospital mortality divided by the expected in-hospital mortality. In-hospital mortality is defined as deaths in the hospital. For calculation of the expected mortality we developed a prediction model as described in the section below.

Statistical analysis

Sample size calculation

For the risk-adjustment models, we used the rule of thumb that approximately 5–10 events per variable are needed to prevent overfitting [27]. To adjust for 25 potential confounders (model without diagnosis groups) and 49 potential confounders (model with diagnosis groups) in the regression analyses would require 125–250 and 250–500 deaths, respectively, in each ED. At the time of the study, the NEED contained over 700,000 ED visits, which corresponds to around 100.000 ED-visits per ED. We accounted for in-hospital mortality of ~ 2% [28]. Therefore, per ED there will be approximately 2000 in-hospital deaths, and an appropriate number of events per ED was expected to prevent overfitting of the models.

Descriptive statistics

Baseline characteristics were summarized per ED and by PC. Data were presented as mean with standard deviation (SD) when normally distributed. Skewed data were presented as median with interquartile range (IQR). Categorical data were presented as number with percentages.

Main statistical analyses

Multivariable binary logistic regression analyses were employed to predict the risk for in-hospital mortality. All known potential predictor variables affecting mortality were included based on both the literature and common sense. Stepwise elimination was then applied to develop the final model. Separate models were developed for the overall population and for sub-populations by categorized by PC, necessary to assess the SMRs. The models used considered the following potential confounders: demographics (age, gender), urgency (triage category), disease severity (Glasgow Coma Scale, vital score – a categorical item composed of respiratory rate, O2 saturation, systolic and diastolic blood pressure, heart rate, and temperature; see figure S1, supplement 1, medication yes/no), complexity and comorbidity (number of specialist consultations during ED stay, additional investigations yes/no: blood tests, radiology imaging), and PCs (overall model only). Expected mortality is the mortality predicted for a population based on its case mix and the model applied. SMRs were computed by dividing the number of observed deaths by the number of predicted (expected) deaths in a given population.

To investigate the impact of diagnoses on the expected mortalities, we also developed the prediction model with the covariates as described above, but with the top 5 diagnoses per PCs added as covariates in the model. In this analysis, expected mortalities were calculated using the model with and without diagnoses.

An overall SMR and SMRs per PC were calculated per ED and reported in funnel plots. In a funnel plot, the SMR is plotted on the y-axis against the size of the study population on the x-axis. The funnel is constructed by lines representing the 95% confidence interval of the SMR in the complete population. Discrimination of the overall model was assessed with the area under the curve of the receiver operating characteristic (AUC-ROC) curve. Calibration was assessed with a calibration plot. Internal validation was conducted with bootstrap validation using 10 iterations.

Finally, a sensitivity analysis was performed to assess the effect of the COVID period on SMRs in EDs and PCs by repeating the analyses in a cohort without the COVID period (1 February 2020 until 1 June 2022). Data were analyzed using SPSS (SPSS, version 25.0, IBM, New York, USA) and R (R, version 4.3.1).

Results

Figure 1 shows patient flow through the study. Out of 728,902 ED visits in the total NEED cohort, 1,566 visits were excluded due to patient deaths before or at the ED, and 34,047 visits were excluded due to unregistered PCs. The analysis included 693,289 ED visits, of which 284,284 were hospitalized and 13,172 died. A subset of 370,266 ED visits, in which ICD-10 codes were fully available, was used to assess the impact of diagnoses on expected mortality across EDs. Inclusion periods for individual EDs are presented as Supplement 1, table S1.

Fig. 1

Patient flow through study. Study design and patient flow in study. The model uses the following potential confounders: age, gender, triage category, Glasgow Coma Scale, vitalscore- a categorical item composed of respiratory rate O2 saturation, systolic and diastolic blood pressure, heart rate,and temperature (See supplement) medication yes/no, number of consultation, addtional investigations yes/no: blood tests, radiology imaging and PCs. Models after stratification by PC do not include PCs. Abbreviations: NEED =Nederlandse Emergency department Evaluation Database,ED=emergency department,PC = presenting complaint, SMR = standardized mortality ratio

Patient characteristics

In Table 1, baseline characteristics per ED are shown. Crude in-hospital mortality for the separate EDs ranged from 1.2 to 2.5%. Overall, 70% of all ED visits were for the six most frequent PCs but their frequencies differed among EDs: “extremity problems”(range among EDs: 8.1–25.4%), “feeling unwell”(6.9–24.9%), “abdominal pain”(6.0-13.3%), “dyspnea”(6.8–11.4%), “chest pain”(1.9–40.5%), and “trauma”(0.7–8.4%). The urgency, disease severity and comorbidity and complexity of ED visits also differed across EDs. Table 2 shows the top 5 most common diagnosis groups by PC and their mortality across EDs as defined by the ICD-10 classification in the cohort in which ICD-10 codes were available.

Table 1 Patient characteristics in seven Dutch eds, total ED visits in the NEED

Table 2 Most common diagnoses groups according the ICD-10 underlying presenting complaints and their mortality across EDs

Expected mortalities

The bootstrap optimism-adjusted AUROC of our model for the overall population was 0.89. The prediction model had excellent calibration to an expected mortality of 40%, after which it underestimated the risk (see Supplement 3). Figure 2 shows that differences in expected mortality across the included EDs are PC-dependent. However, there are no differences between the expected mortality calculated with the model with or without diagnoses groups as covariates.

Fig. 2

Expected in-hospital mortality with and without ICD-10 codes as covariates in model in subgroups of patients by six most common PCs in five Dutch EDs The figure above is the result of the analysis in the cohort from which complete data on ICD-10 codes are available. This analysis assessed the effect of additional risk adjustment with diagnoses on the expected (predicted) mortality of PCs in participating EDs. In this cohort, data from five EDs (B, C, D, E, and G) were used, where complete diagnostic data on ED visits were available according to the International Classification of Diseases, 10th Revision (ICD-10) codes. No ICD-10 codes are known from ED A and ED F and therefore not presented in the figure. Expected mortality without adjustment for diagnosis groups (-) was calculated with multivariable binary logistic regression adjusted for: age, sex, triage category, vital score**, GCS-score, number of specialist consultations during ED stay, blood tests taken, radiology imaging performed, medication yes/no and PCs. For the expected mortality calculated including adjustment for diagnoses (+), in addition to the previously mentioned covariates for mortality, the 5 most common diagnosis groups as described in the ICD-10 coding system plus a residual group with other diagnoses were added as dummy variables for each PC. See Table 2 for most common diagnosis groups. Abbreviations: PC = presenting complaint, ED = Emergency Department, ICD-10 codes = International Classification of Diseases and health related problems 10th edition, GCS = Glasgow Coma Score. **vital score: a categorical item composed of respiratory rate, O2 saturation, systolic and diastolic blood pressure, heart rate and temperature; see figure S1, supplementary file 1

Overall SMR and SMRs per subgroups of the 6 most common presenting complaints

ED-SMRs were calculated for the overall populations of participating EDs, and they are visible in Fig. 3. This funnel plot shows that the ratio between observed and expected mortality for overall ED populations varies widely across EDs with SMRs outside the 95% CI for 4 EDs. Finally, Fig. 4 shows the calculated PC-SMRs across EDs in subgroups of patients based on the six most common PCs. On visual inspection, some PC-SMRs appear to have more variability than others across EDs. While all EDs had PC-SMR within the 95% CI funnel for “extremity complaints” and “chest pain”, several EDs had PC-SMRs outside the funnel for “feeling unwell” and “dyspnea”. For some EDs, PC-SMRs were not consistently high or low for all PCs. For example, while SMRs were high for almost all PCs for ED G, ED B also had a high SMR in the complete population and for some PCs, but a low SMR for “chest pain”. The results of the sensitivity analysis, without COVID period, can be found in Supplement 4.

Fig. 3

Funnel plot with standardized mortality ratios of seven Dutch EDs. The figure above is the result of the analysis in which the cohort includes seven EDs over the period from January 2017 through June 2023. On the Y-axis the SMR (Observed/Expected), on the X-axis (precision) the number of cases. The funnel lines represent the 95% confidence intervals for the SMRs, based on the total population. SMRs were computed by dividing the number of observed deaths by the number of predicted deaths in the given population. An SMR value above 1 was interpreted to mean that more deaths than expected occurred in an ED; conversely, a value below 1 was interpreted to mean that there were fewer deaths than expected. The expected mortality is calculated with multivariable binary logistic regression adjusting for: age, sex, triage category, vital score**, GCS-score, number of consultations, blood tests taken, radiology imaging performed and medication yes/no and PCs. Abbreviations: PC=presenting complaint, ED=Emergency Department, SMR=standardized mortality ratio, GCS= Glasgow Coma Score. **vital score: a categorical item composed of respiratory rate, O2 saturation, systolic and diastolic blood pressure, heart rate and temperature; figure S1, supplementary file 1

Fig. 4

Funnel plot with standardized mortality ratios of in subgroups of patients by six most common presenting complaints in seven Dutch EDs. The figure above is the result of the analyses in which the cohort includes seven EDs over the period from January 2017 through June 2023. On the Y-axis the SMR (Observed/Expected), on the X-axis (precision) the number of cases. The funnel lines represent the 95% confidence intervals for the SMRs, based on the total PC populations. SMRs were computed by dividing the number of observed deaths by the number of predicted deaths in the given population. An SMR value above 1 was interpreted to mean that more deaths than expected occurred in an ED; conversely, a value below 1 was interpreted to mean that there were fewer deaths than expected. The expected mortality in EDs per PC is calculated with multivariable binary logistic regression adjusting for: age, sex, triage category, vital score**, GCS-score, number of consultations, blood tests taken, radiology imaging performed and medication yes/no. Abbreviations: PC=presenting complaint, ED=Emergency Department, SMR=standardized mortality ratio, GCS= Glasgow Coma Score. **vital score: a categorical item composed of respiratory rate, O2 saturation, systolic and diastolic blood pressure, heart rate and temperature; figure S1, supplementary file 1

Discussion

The present study shows substantial variability of patient characteristics across Dutch EDs. Expected mortality across EDs is well assessed in PC subgroups using a multivariable model, without further improvement by adding the final diagnosis to the model. Differences in SMR across EDs were larger in subgroups by PC than in the overall population.

This study was the first to report SMRs per PC rather than an overall SMR per ED. Jeong et al. [6] developed a scoring system to predict mortality in ED patients and proposed using standardized W statistics based on data collected at the initial ED evaluation, including the cause of the visit to compare severity-adjusted mortality among institutions. Berthelot et al. [5] introduced a Hospital Standardized Mortality Ratio (HSMR) for ED care, using emergency-sensitive ICD-10 diagnoses in a multivariate model. These studies highlight the importance of accounting for severity adjustment and heterogeneity of conditions when comparing mortality among EDs, providing different methodologies for more accurate and meaningful comparisons.

Adjustment for reason of presentation to the ED is commonly performed by studying a model in a specific ED population based on diagnosis, e.g. in patients with COVID-19 or in patients with sepsis [29, 30]. Although Berthelot et al. showed that it is possible to develop a model for risk adjustment for overall ED populations based on a limited number of emergency-sensitive diagnoses, such a model may be of limited use in ED practice. In many ED patients, diagnoses are made after their ED presentation, not at all, or appear to be erroneous [31, 32].

As an alternative for diagnoses, we compared EDs by PCs. The advantage of PCs is that they are patient-centered, are already used in triage systems and are consequently available directly after registration at the ED. In addition, PCs align with the way EDs are organized and often determine which care pathway is followed and which resources are used. For example, in patients with “chest pain” different blood tests and radiological imaging will be acquired compared to patients with “dyspnea” or “abdominal pain”, and different specialties will be involved. Finally, PCs contain prognostic information. Although subgroups of PCs may include different diagnoses with dissimilar risks of death and may potentially lead to inadequate risk adjustment, we showed that additional risk adjustment for the most common diagnosis groups by PC caused no relevant change in expected mortality across EDs. This suggests that adjustments for severity and heterogeneity of conditions across EDs, as previous literature emphasized, are adequately accounted for using PCs alone. We found that SMRs varied across EDs. Importantly, the variation in SMRs across EDs is not present in all PC groups, but only in specific PCs. Differences were most marked in patients presenting with non-specific PCs “dyspnea” and “feeling unwell”, in contrast to PCs “extremity complaints” and “chest pain”. Also, ranking SMRs based on risk adjusted mortality is PC-dependent. Some EDs have unfavorable PC-SMRs in specific PCs, but favorable PC-SMRs in others. There are a number of possible explanations for these findings. Theoretically, there may be residual confounding, which our model does not adequately correct for. However, the discrimination of our model is excellent so we do not expect this. Also, several funnel plots for subgroups instead of one for the overall populations increase the probability of finding differences by chance as the number of tests rises, i.e. by multiple comparisons. Finally, differences in SMRs found in PC subgroups may have been caused by differences in the care provided to that specific subgroup. A high PC-SMR may indicate suboptimal diagnostic work-up, protocols, or staffing specific to this subgroup. These insights would not have been obtained if only SMRs for the overall population of EDs were reported, especially if subgroups are relatively small or if suboptimal outcomes of one subgroup are offset by optimal outcomes in another. In contrast to diagnoses, which are sometimes absent in the ED and would only allow for retrospective comparisons, PC-SMRs provide insight into which subgroups of patients EDs can focus their quality improvement initiatives on to improve care for the entire ED process, from enrollment to discharge.

Individual EDs can use PC-SMRs to improve outcomes. For example, if an ED identifies an elevated SMR for patients presenting with dyspnea, it should consider implementing a targeted quality improvement program for this patient group. Additionally, the ED could collaborate with other EDs that report significantly lower SMRs for patients with dyspnea to investigate differences in staffing, quality of care, or other factors contributing to better outcomes in dyspnea patients. This approach is similar to successful initiatives in the Dutch quality registry for esophageal cancer (DUCA), which have improved patient outcomes [33].

There are some limitations of this study which should be discussed. First, comorbidities were not registered. However, we believe that the use of proxies, such as number of consultations and performed additional investigations were good alternatives, as a previous study have shown that they are associated with comorbidities and complexity [34]. Second, only seven EDs participated in this study. In the next years many more EDs in the Netherlands are expected to participate in the NEED, which may influence to what extent SMRs fall outside the norm. The current results, however, already show that Dutch EDs are different with respect to SMRs and that some of these differences are found only in PC subgroups. Fourth, the model we developed has limitations, such as overestimating the number of deaths at very high death probabilities. However, this is not considered a major problem for this study since most ED visits have low mortality probabilities. Sixth, outcomes may have been influenced by patients who died after discharge to home or transfer to another hospital. Our study does not aim to explain differences found, but identifies patient groups that EDs can target regarding outcome differences, including evaluation of discharge or transfer policies. Seventh, although other relevant clinical outcomes exist to compare EDs, mortality (and the SMR) is the most robust, although most ED patients will not die. In most of the selected PCs, mortality is relatively high, and outcomes like hospital admission may not always reflect quality of care as logistic issues also influence them. In future studies, it would be valuable to investigate other standardized outcomes. Finally, the present study has limitations inherent to its retrospective nature such as errors of documentation and data entry, although this was largely automatized which minimalized accidental misregistration.

In summary, patient characteristics and mortality varied largely across Dutch EDs, and expected mortality across EDs is well assessed in PC subgroups without adjustment for final diagnoses. Variations in SMRs among EDs are PC-dependent, enabling practical benchmarking across EDs, despite their diverse and undifferentiated patient populations. Future studies should investigate reasons of the differences in PC-SMRs across EDs and whether PC-targeted quality improvement programs improve outcomes.