Measuring the crowding of emergency departments: an assessment of the NEDOCS in Lombardy, Italy, and the development of a new objective indicator based on the waiting time for the first clinical assessment

Abstract

Background

There is no ubiquitous definition of Emergency Department (ED) crowding and several indicators have been proposed to measure it. The National ED Overcrowding Study (NEDOCS) score is among the most popular, even though it has been severely criticised. We used the waiting time for the physician’s initial assessment to evaluate the performance of the NEDOCS and proposed a new crowding indicator based on this objective measure.

Methods

To evaluate the NEDOCS, we used the 2022 data of all the Lombardy EDs and compared the distribution of waiting times across the five levels of the NEDOCS at ED arrival. To construct the new indicator, we estimated the centre-specific relationship between the total number of ED patients and the waiting time of those with minor or deferrable urgency. We defined seven classes of waiting times and calculated how many patients corresponded to an average waiting time in the classes. These centre-specific cutoffs were used to define the 7-level crowding indicator. The indicator was then compared to the NEDOCS score and validated on the first six months of 2023 data.

Results

Patients’ waiting time did not increase at the increase of the NEDOCS score, suggesting the absence of a relationship between this score and the effect of ED crowding on the ED capacity of evaluating new patients. The indicator we propose is easy to estimate in real-time and based on centre-specific cutoffs, which depend on the volume of yearly accesses. We observed minimal agreement between the proposed indicator and the NEDOCS in most EDs, both in the development and validation datasets.

Conclusions

We proposed to quantify ED crowding using the waiting time for physician’s initial assessment of patients with minor or deferrable urgency, which increases in crowding situations due to the prioritization of urgent patients. The centre-specific cutoffs avoid the problem of the heterogeneity of the volume of accesses and organization among EDs, while enabling a fair comparison between centres.

Introduction

Emergency department (ED) crowding is a worldwide recognized problem, although without a precise and unique definition [1]. Crowding affects patients’ outcomes directly, as it has been linked to increased medical errors [2], inpatient mortality [3], and worse quality of care [4, 5]. Other consequences are related to patient management efficiency [6], as ED crowding is associated with longer pre-visit waiting times [7, 8], ED total length of stay [9] and an increase in the number of patients leaving without being seen by a doctor [10]. Among the several causes contributing to crowding conditions in ED, the most common are staff shortages, sudden increase in patients’ arrivals and the lack of hospital beds for patients needing hospitalization.

Due to the variability of definition, causes and effects, a wide range of indicators have been proposed to measure ED crowding [11]. Both simple indicators, directly measuring one of the aspects of ED crowding (e.g., occupancy rate or the number of patients in the ED) [12, 13], and composite indicators, which account for several factors describing the conditions of EDs, have been proposed. Among others [14,15,16,17,18,19,20], the National ED Overcrowding Study (NEDOCS) [21] is one of the most widely used composite indicators in practice [22]. It is currently adopted in the Lombardy region of Italy to monitor the crowding level of the 117 regional EDs. The index is based on data collected over 20 years ago in eight US academic EDs. It consists of a score based on the number of patients in the ED, both in the waiting room and in visit, the number of ED patients waiting for a hospital bed and those using ventilators, the waiting time from registration to visit of the last called patient and the longest hospitalization time, defined as the waiting time for a bed for patients needing hospitalization [21]. The score was developed to predict the subjective physicians’ perception of crowding. Although some studies showed the NEDOCS’s ability to discriminate across ED crowding levels [10, 22], results were contrasting in several other experiences [23,24,25,26,27,28]. In Italy, the NEDOCS is frequently criticized by emergency physicians, as its quantification of crowding often does not match the staff’s perceptions, as demonstrated by Strada et al. [29].

One of the most commonly described measurable effects of ED crowding is a delay in patient assessment and treatment [11, 30,31,32]. Hence, to assess the ability of NEDOCS to represent the ED crowding level, we evaluated the relationship between the score and the patients’ waiting time for the physician’s first assessment in all general EDs in the Lombardy region of Italy. Because the waiting time has consistently been reported as an objective consequence of ED crowding [33], it was also used to develop a new crowding indicator.

Methods

Data source

The Lombardy Regional Agency for Emergency and Urgency, namely AREU, coordinates a system of data centralization from all EDs in the region. For each ED, this system records the hourly information of the NEDOCS and, for each arriving patient, the triage priority code and the date and time of arrival, physician assessment and ED discharge. Lombardy’s triage system consists of four levels: red code identifies patients with compromised vital functions (immediate emergency care); yellow code is assigned to patients with evolving conditions or in acute pain (very urgent emergency care); green code is for patients with minor or deferrable urgency (urgent and less urgent emergency care), and white is assigned to non-urgent patients. The data centralization system also records the ED where each patient arrives. In Lombardy, EDs are classified into three groups according to the catchment area and the number of hospital specialties. Specifically, simple EDs (S-ED) have catchment areas with less than 150,000 inhabitants and are located in hospitals with essential services only; hospitals of Level-I Departments for Emergency and Admission (DEA-I) have a catchment area of 150,000–300,000 residents and offer more specialties, including at least a high dependency care unit. Level-II DEAs (DEA-II) are part of hospitals with complex structures, most services available 24–7, and a wider user base [34].

We extracted the hourly data of the NEDOCS from 2022 and the first 6 months of 2023 and the information about all patients arriving at one of the 91 general EDs in Lombardy in the same timeframe. We did not include specialized emergency departments, i.e., paediatric, obstetrics/gynaecologic, traumatological, orthopaedic and ophthalmic EDs. Patients with missing information on triage code, arrival and visit date were excluded. The data provided by AREU were anonymous as they do not identify patients and do not represent personal information. Hence, the study does not require the Ethics Committee’s approval or the signing of patient informed consent [35].

Association between NEDOCS and waiting time

We hypothesized the increase in waiting times for the arriving patients to be an objective measure of the progressive saturation of the ED material and human resources and, thus, of ED crowding. To verify this hypothesis, we studied the association between the time each patient waited for the first physician assessment and the number of patients who were present in the ED during the waiting time. The waiting time was computed as the time elapsed between arrival at ED and the start of the medical examination. Using the date and time of arrival and discharge from the ED, we calculated the hourly number of patients in the ED, including all waiting and in-visit patients. Since the number of patients in the ED could change over the waiting time, we computed, for each arriving patient, the average number of patients in the ED over the waiting time. The ED-specific association between the average number of patients in the ED (predictor) and the waiting time of the patients (response) was estimated with linear regression, separately for each ED. We opted for this family of regression models because the estimated association was found to be linear for all centres.

After verifying the consistency of using the waiting time as an indicator of ED crowding, we evaluated its association with the NEDOCS. Specifically, we grouped the patients by NEDOCS level upon arrival at the ED and compared the distribution of the waiting times across groups. Because in Lombardy the NEDOCS is stored as a five-level indicator, patients were grouped with the same classification, i.e., not busy or busy (score 0–50), very busy but not overcrowded (51-100), overcrowded (101–140), severely overcrowded (141–180) and dangerously overcrowded (> 180).

Development of the new crowding indicator

In parallel with the validation of the NEDOCS, we defined a new crowding index based on the association between the waiting time for physician assessment and the average number of patients in the ED. While a linear relationship described well the nature of this association for all triage codes, we restricted our analysis to green triage code patients (minor or deferrable urgency), as we verified that the waiting time of patients with red and yellow codes (extremely or very urgent) depended on the number of patients in the ED only marginally, while the size of the group of white-code patients (non-urgent) was very limited.

We established six thresholds of the waiting time, i.e., 15, 30, 60, 90, 150 and 210 min from triage. According to Italian guidelines [36], green-code patients should not wait longer than 90 min for the first clinical assessment. Following this definition, we considered 90 min as one of the thresholds. Then, we included other meaningful times, lower and greater than 90 min, to provide a more detailed characterization of the waiting time as a crowding-level consequence. Using the estimates of the models, we were able to identify, for each ED, the number of patients in the centre corresponding to the average expected waiting times equal to these six prespecified thresholds. We thus calculated six ED-specific cutoffs for the number of patients in the ED that define the seven crowding levels, from 1 (least crowded, average expected waiting time for patients with minor or deferrable urgency < 15 min) to 7 (most crowded, average expected waiting time ≥ 210 min).

Evaluation of the proposed indicator

The computed ED-specific cutoffs were retrospectively applied both to the 2022 data, which were used to construct the new indicator, and to the first 6 months of 2023, to assess the consistency of the indicator on external data. In both cases, we evaluated the hourly crowding condition of each ED. As a first step, we compared the overall distribution of the waiting times of patients with minor or deferrable urgency in 2022 and 2023, to verify whether any major change in the waiting times occurred in 2023. Then, to verify whether the ED-specific linear relationship between number of patients in the ED and waiting time in 2023 differed from the estimates performed in 2022, we estimated the 2023 ED-specific lines and compared the slopes to those estimated in 2022. Finally, we applied the cutoffs estimated on the 2022 data to 2023 and estimated the hourly crowding level using the developed indicator. For each ED, the distribution of the indicator over the six months of 2023 was compared to the NEDOCS and to the results on the development dataset, i.e., the 2022 data, to evaluate whether the estimated levels of crowding were robust on validation data. Moreover, we checked if the distribution of the waiting time of the 2023 patients was consistent with the distribution observed in 2022, both according to our indicator and the NEDOCS. All analyses were performed using R, version 4.2.1.

Results

Patients characteristics

Figure 1 provides a graphical representation of the locations of the 91 Lombardy EDs on a map of the region, showing the population density at the township level [37]. A total of 2,964,363 visits to Lombardy EDs were recorded in 2022. After the exclusions represented in Figure S1 (supplementary material), 2,754,968 patients were retained for the analyses, of which 1,938,458 (70.4%) received a green code at triage (minor or deferrable urgency). We compared the patients excluded because of the missing triage code to those retained for the analyses in terms of hospitalization (missing vs. nonmissing triage code: 12% vs 14%, standardized mean difference [SMD]: 5.9%) and ED mortality rate (0.1% vs 0.3%, SMD: 4.5%). Because SMDs smaller than 10% are generally considered negligible between-group differences [38], there is no evidence of a substantial difference between included and excluded patients.

Fig. 1

Geographic location of the 91 general EDs in Lombardy

Table 1 provides a description of the available data. In 2022, half of the arrivals (52%) were recorded by the 41 DEA-I EDs (45% of the 91 EDs in Lombardy). About one-fourth of the visits (25.7%) were directed to the 12 DEA-II EDs (13% of the EDs), while the remaining visits (22.3%) were recorded in the 38 S-EDs (42% of the EDs). Most arrivals were recorded on weekdays and in the morning. Patients with immediate urgent conditions at triage were the least frequent (2.3%), followed by non-urgent (6.9%) and very urgent patients (20.4%). As expected, the median waiting time from triage to the first clinical assessment was longer for patients with low-priority triage codes (white and green), with very high variability (IQRs wider than 100 min). On the contrary, the time between the first physician assessment and ED discharge or hospital admission was longer for patients triaged in immediate urgent conditions, with a median of more than 4.7 h with respect to a median of 24 min for non-urgent patients. Similar characteristics were observed for the 2023 data.

Table 1 Patients’ accesses characteristics

Association between NEDOCS and waiting time

The linear relationship between the number of patients and the waiting time, stratified by triage code, was confirmed for all EDs. Figure 2 shows an example of the estimated trends for a centre. While the waiting time of red-code patients, given their urgency, is usually not affected by the number of patients in the ED, the slope of the linear relationship for all other triage codes was positive, with the waiting time of green codes (minor or deferrable urgency) being the most affected by the increasing number of patients in the ED. In Supplementary Materials, we report the estimates obtained for each of Lombardy’s EDs (Fig. S3) and the distribution of the corresponding slopes separately for the triage codes (Fig. S4).

Fig. 2

Example for an ED of the estimated linear relationship between the number of patients in the ED and the waiting time stratified for the four triage codes

These results support the role of the waiting time for the physician initial assessment of patients with minor or deferrable urgency as an indicator of ED crowding. To evaluate the sensitivity of the NEDOCS to this indicator, we stratified all the 2022 patients according to the value of the NEDOCS computed at their arrival to the ED and compared the distribution of the waiting times across strata. Such a comparison is provided in Fig. 3. While we observed increasing median waiting times in the first two strata, the waiting time did not increase in the three highest NEDOCS levels, where the median times were approximately constant, around 50 min. Paradoxically, the waiting time of patients entering dangerously overcrowded EDs was lower than when entering severely overcrowded EDs. These results were confirmed when the same evaluation was repeated separately for the other triage priority codes (Supplementary Figure S12b).

Fig. 3

Distribution of the waiting times of all patients arrived in 2022, stratified by the levels of NEDOCS at ED arrival

Development of the new crowding indicator

The waiting times for physician initial assessment for patients with minor or deferrable urgency were well distributed into the seven categories defined by the six established threshold times (Suppl. Fig. S5). In particular, 23% of the patients waited less than 15 min from triage to the start of the visit, whereas 12% waited more than 3.5 h.

Figure 4 shows an example of the cutoffs definition for two EDs: an S-ED (Panel A) and a DEA-II ED (Panel B). The solid line represents the relationship between the number of patients and the waiting time of minor or deferrable urgency patients, estimated by linear regression. On the y-axis, the dashed lines represent the established threshold times, from which we could determine the corresponding, centre-specific, number of patients, which represent the cutoffs of the indicator. Specifically, the S-ED in the example (Fig. 4A) is considered in crowding level 3 whenever there are between 4 and 7 patients and critically overcrowded (level 7) with more than 24 patients. In the ED managing a greater volume of patients (Fig. 4B), the same crowding levels are attained whenever 11 to 27 patients (level 3) and more than 112 patients (level 7) are present, respectively. An example of the cutoffs definition for a DEA-I ED is provided in Supplementary Materials (Fig. S6).

Fig. 4

A Example of cutoffs’ definition for a S-ED; (B) Example of cutoffs’ definition for a DEA-II ED

Evaluation of the proposed indicator and comparison with the NEDOCS

The distributions of the waiting time of minor or deferrable urgency patients in 2023 and 2022 were very similar (Suppl. Fig. S8). Thus, there was no evidence of major events affecting the overall distribution of times and consequently preventing the use of the 2022 estimated cutoffs on 2023. The comparison of the 2022 and 2023 ED-specific lines representing the relationship between number of ED patients and waiting time resulted in very similar estimated slopes (Suppl. Figure S9), suggesting that the cutoff estimated in 2022 can be safely used to measure the crowding level in 2023.

When applying the crowding indicator to each ED, Fig. 5 reports the distribution of the crowding levels observed over 2022 and 2023 according to the proposed indicator and the NEDOCS. One ED was closed in 2023, so it only contributed to the development dataset. The distributions of the two indicators appear to be very discordant in both years. For example, four EDs that appear to be dangerously overcrowded according to the NEDOCS over 50% of the time do not seem very crowded according to our indicator (10th, 27th, 30th and 31st centres from the bottom) and, conversely, EDs that appear as not very crowded according to the NEDOCS are quite often classified as overcrowded with our indicator. Remarkably, the frequency of the levels of our indicator and the NEDOCS in 2023 was similar to 2022 for almost all EDs, confirming the consistency of our indicator on data that were not used for its development.

Fig. 5

Distribution of the crowding levels according to our indicator and the NEDOCS in 2022 in all general EDs in Lombardy, compared to the distributions of the first six months of 2023

Overall, according to our indicator, over the year 2022, the 38 Lombardy S-EDs were in crowding level 1 (corresponding to an expected average waiting time for visiting patients with minor or deferrable urgency of fewer than 15 min) 19.1% of the time. In the 41 DEA-I EDs, this proportion dropped to 5.2%, while the 12 DEA-II EDs were almost never in crowding level 1. On the other hand, DEA-II EDs were more frequently overcrowded, being in crowding level 6 or 7 (expected average waiting time of patients with minor or deferrable urgency of more than 2.5 h) almost 25% of the time, compared to around 6.5% in DEA-I EDs and 9.8% in S-EDs (Suppl. Figure S10). Considering 2023 (Suppl. Figure S11), the overall frequency of the levels of our indicator and the NEDOCS is remarkably similar to 2022, with S-EDs and DEA-I EDs being more frequently in lower crowding levels than DEA-II EDs, with both indicators.

Finally, we repeated the comparison of the distribution of the waiting times across strata of both our indicator and NEDOCS on 2023 data, to check the consistency with the development dataset. The results are shown in Supplementary Materials (Fig. S13, S14). As expected, according to our indicator, the median waiting time constantly increases with the ED crowding level, while, on the other hand, the trend for the NEDOCS was similar to what was shown for 2022.

Discussion

Our study verified that the waiting time of patients arriving to the ED with minor or deferrable urgency is directly linked to the total number of patients in the ED and found a lack of agreement between this measure and the NEDOCS, suggesting a poor performance of the broadly used crowding indicator in our large cohort. The waiting time was then used to propose a new objective indicator, which was developed to be easy to estimate in real time, clearly interpretable and centre-specific.

We considered the waiting time of ED patients as a measure of ED crowding because the delay in the first assessment of incoming patients, especially for those with non-severely urgent clinical conditions, is one of the most direct and reported consequences of the ED crowding [7, 9]. As expected, we found that the waiting time of urgent patients, who are assigned high priority codes, are minimally affected by ED crowding and, therefore, we focused on patients with minor or deferrable urgency. For these patients, we found a monotonically increasing relationship between the clinical assessment waiting time and the overall number of patients in the ED, confirming that delaying the clinical evaluation of patients with minor or deferrable urgency is a consequence of crowding. Thus, the waiting time of these patients can be profitably used both to validate existing scores of ED crowding and to develop new ones.

In our study, the waiting time did not increase at the increase of the NEDOCS level, in particular for high-level crowding conditions. This means that the score fails to capture the demonstrated consequence of overcrowding. There are several possible explanations for the poor performance of the NEDOCS in our cohort. First, the reliability of some of its items is not always guaranteed. As the structural variables cannot be collected from data flows generated by the ED software, they are periodically self-reported from hospitals. Hence, their reliability depends on the frequency of updates and their correct specification. Second, in variables such as the longest time in the ED for admitted patients, possible outliers could result in unreliable estimated levels of the NEDOCS. Third, the weights of all the NEDOCS score items were obtained by a model developed using data derived in the early 2000s from eight mid-to-large (40,000–80,000 yearly visits) US academic EDs to predict the physicians’ subjective perception of crowding. It is hard to think that the perceived crowding of US physicians in mid-to-large EDs in the early 2000s still represents the effective crowding of Western EDs nowadays.

Despite the universality of the effect of crowding on the clinical assessment waiting time, to the best of our knowledge, no index in the literature has quantified the crowding level on this phenomenon, which has the advantage of being an objectively measurable outcome variable. In contrast, other proposed indicators, like NEDOCS, were developed using the subjective physicians’ and nurses’ perceptions of crowding as outcome variable. We did not directly quantify crowding according to the actual patients’ waiting time as it cannot be estimated in real time. Indeed, in each moment, we could only know how long patients have been waiting, but not how long they still have to wait for the visit. For this reason, we based our indicator on the number of patients in the ED, which can be easily calculated in real-time from any ED electronic health record without the need for any additional data input.

In contrast to the NEDOCS, the proposed indicator can be easily computed at any given moment. Once the cutoffs are estimated, the crowding level can be calculated using only the number of patients in the ED, which can be effortlessly extracted from any ED electronic health record. The choice of centre-specific cutoffs enables a fair comparison of the crowding level between EDs of all dimensions and complexity. The interpretation of the crowding levels of the new indicator is indeed the same in all EDs: in the lowest crowding level, patients with minor or deferrable urgency wait for the visit, on average, less than 15 min in all EDs, while their expected waiting time is always more than 3.5 h in the highest crowding level. We believe the simplicity of interpretation of the levels of our indicator to be one of its major advantages.

Over the year 2022, the calculated crowding levels differed across Lombardy’s EDs, even within groups of EDs with the same complexity level (S-ED, DEA-I and DEA-II), confirming the ability of our indicator to capture the variability of crowding between centres with similar characteristics. The application of the ED-specific cutoffs to the first half of 2023 data provides a first external validation of our indicator. Unfortunately, no gold standard exists to measure the performances of the proposed indicator and the NEDOCS [12]. However, with our indicator, the median waiting time for the visit of patients of all codes increased at the increase of the crowding level. While this trend was expected by design for patients with minor or deferrable urgency, as the indicator was purposely developed to measure the crowding level based on the waiting time of this subgroup of patients, the fact that the same trend was observed for all other triage codes demonstrates that our indicator succeeds in detecting a true consequence of ED crowding. In contrast, the NEDOCS levels failed in discriminating the waiting times for the physician’s initial assessment, as their distribution was very similar across different crowding levels.

Our study is subject to limitations. First, unlike other proposed indicators, we did not account for the patients’ complexity when computing the total number of patients in the ED, so we attributed the same weight to all patients. Instead, some patients may require more human and technological resources because of their severity or care needs, and they should weigh more heavily on the crowding conditions of EDs. This problem should be investigated in future works. Additionally, our data did not include information on whether and when patients were admitted to an observation unit during their ED stay. In Lombardy, such units are part of the ED and dedicated to patients requiring short-term observation or monitoring in order to improve the appropriateness of the final decision to hospitalize them or not. As they are generally managed by dedicated physicians and nurses, further investigation is needed to assess whether separately accounting for the patients in observation units may improve the accuracy of the proposed crowding indicator.

Second, the indicator relies on centre-specific cutoffs, which we estimated on one year of data for each ED. These cutoffs must be updated regularly, as the ED organization changes over time, altering the relationship between the number of patients and waiting times. This is particularly relevant in case of specific events directly affecting the ED activity. For example, during the COVID-19 pandemic, both the volume of accesses and the internal organisation of the EDs were constantly evolving, thus requiring a frequent update of the cutoffs to provide a meaningful quantification of ED crowding. Nonetheless, this update process is easy to implement once the algorithm is defined. Furthermore, the data necessary to define the cutoffs are minimal and easily retrievable from any electronic health record system.

Third, we developed our indicator on data collected only in one year (2022) from the EDs in one Italian region. Nevertheless, the consequence of crowding on the waiting time is a widespread, common problem in all EDs; this indicator can be easily developed and applied to other contexts regardless of organizational differences. For instance, for each ED willing to use this indicator, we could easily estimate its specific cutoffs of patients on the retrospective data of the previous year, given the simplicity of the algorithm. Once the cutoffs are defined, the ED can apply them to estimate real-time crowding. The calculation of the cutoffs could be centralised, if the EDs are part of a regional or larger system of data centralisation, such as the one in Lombardy, or they can be individually calculated by centres using their own data.

Fourth, we evaluated our indicator by comparing it to the NEDOCS, which cannot be considered a gold standard for ED crowding. A future prospective validation in a clinical practice setting should be carried out for a more reliable evaluation of the indicator, for example by comparing the purposely recorded perceptions of the staff crowding with the developed indicator.

Such a further validation study could also overcome the fifth limitation of our indicator, which relates to the subjectivity of the classes of waiting time we chose to estimate the different levels of crowding. It is indeed possible that these do not capture operators’ perceptions of crowding well. It would be appropriate to finetune the threshold times to maximize their discrimination ability with respect to the feeling of the ED staff.

Finally, the indicator estimates the general level of crowding but cannot be used to establish its cause, so it cannot suggest strategies to address this problem (e.g., reducing the number of arriving patients through ambulance diversion, or speeding the hospitalizations from the ED). For this purpose, the indicator should be interpreted along with other measures that can directly quantify these problems.

In conclusion, to overcome the limitations of the NEDOCS, we proposed a new, objective, seven-level crowding indicator that enables the evaluation of the real-time crowding level in ED. The indicator, which should be prospectively validated in future studies, can be easily exported to different contexts and emergency care settings.