Emergency department (ED) crowding is a growing problem that can lead to the deterioration of the quality of health care. This concern is associated with a rise in medical errors made by clinicians [1,2], prolonged patient stay [3], morbidity [4,5], and increased mortality rates [6-8]. In some cases in the Netherlands, standards of health care were not met, resulting in the temporary closure of EDs [9]. The issue of overcrowding is expected to become even more evident in the coming years due to increased life expectancy and increased demand for complex care [10,11].

In recent years, studies have been extensively exploring the issue of ED crowding [12-14]. In a conceptual model, ED crowding is divided into three interdependent components: (1) the input component, (2) the throughput component, and (3) the output component [15]. Changes in one of these components can contribute to the ED length of stay and therefore ED crowding [16]. Each component comprises multiple factors that can influence the overcrowding problem both independently and through interaction with factors within or outside the component [13]. Focusing specifically on the throughput component, key factors have been identified, namely the experience level of staff [17], shortages of staff within the ED [18,19], availability of beds in the ED [20], delays in test results, and disposition decisions [21].

To address the problem of overcrowding, solutions can be pursued both within the ED and through broader changes at the hospital. Within the ED, particularly concerning the throughput component, studies have investigated, for example, the implementation of fast-track systems [22,23], adjustments in triage models [24], and the rising application of artificial intelligence (AI) solutions [25].

Currently, several studies are investigating the potential of AI solutions to mitigate the overcrowding problem in the ED. Some promising results have been reported in areas of patient admission to inpatient units and intensive care units or discharge from the ED, thereby impacting the duration of stay in the ED [21]. However, these AI models can vary significantly in their functioning, often using diverse parameters at various time points during ED admission [26-32]. While these studies tend to focus on the technical performance of the models, they often neglect to consider their practical relevance and applicability within health care settings [33-35].

In this study, an AI model was developed for decision support in the ED. Moreover, the retrospective model predictions were evaluated with updates occurring every 10 minutes based on the most current patient data. This AI model showed the health care professionals and residents the probability of admission to a hospital ward from the ED.

The clinical relevance of the AI decision support system was evaluated by analyzing the decision-making time. This evaluation involves retrospectively examining whether the model can reduce the time required for an admission decision, thereby potentially decreasing ED length of stay and alleviating the overcrowding problem.

In this retrospective study, anonymized patient records from the Dutch St. Antonius Hospital were collected from January 2018 to September 2023. The St. Antonius Hospital in the Netherlands has 2 different locations where emergency care is provided, and it is a level 2 trauma center in an urban setting. A total of 41,000 patients present to the EDs each year at the St. Antonius Hospital. The data up until May 2022 were earmarked for the development and assessment of our AI model.

As the emphasis of this study was on the clinical impact rather than the complexities of model development and evaluation, detailed information regarding the development of the model is provided in Multimedia Appendix 1, which includes a comprehensive overview of the preprocessing steps, feature selection, and models considered during the development phase. The development of the model was based on the study by De Hond et al [32]. Admissions in this study comprised patients treated in the EDs of St. Antonius Hospital. Patients who explicitly declined to provide consent for the use of their data in any research context were excluded. Additional exclusion criteria encompassed patients younger than 18 years.

The AI model predicts the admission probability as a percentage. If this percentage exceeds 50%, the model classifies the case as “admission.” Once the threshold is reached, the decision is final and cannot be reversed. This design choice was made to mimic clinical commitment, thereby reflecting a realistic clinical decision-making context in which a patient admission decision is typically irreversible once made.

Features were extracted from the dataset by performing several steps during the data cleaning and transforming phase. A detailed list of these features, along with a comprehensive explanation of the data-cleaning process, is provided in Table S1 in Multimedia Appendix 1. For model development, the dataset collected between January 1, 2018, and May 15, 2022, was split in an 80:20 ratio randomly; 80% (105,000/131,250) of the data was used for training the AI model, and 20% (26,250/131,250) was used for testing.

To evaluate the AI model’s performance in predicting ED admissions, a dataset including all ED visits from May 16, 2022, to September 1, 2023, was created. This dataset was designed to mimic real-world scenarios, allowing us to assess the model’s clinical performance in a controlled retrospective setting. This dataset was referred to as the evaluation dataset. Admission data were divided into 10-minute intervals, starting from the initial 0 minutes up to 3 hours. This segmentation reflected clinical decision-making by providing the model with the most recent information available at each time point. By checking new predictions every 10 minutes, we could analyze how changes in these predictions supported clinical decision of admission to the ward. This method tested the model’s ability to make accurate predictions with limited and progressively updated information, reflecting typical emergency settings.

Thus, 3 datasets were used: 1 for training the model, 1 for testing during the development phase, and 1 evaluation dataset to calculate the time saved by the AI model compared to admission decisions made by ED health care professionals.

This study was reviewed and approved by the St. Antonius Hospital’s local review committee (approval R&D/Z24.050). No informed consent from patients was required for this study, as it did not involve any additional risks or burdens for patients. Patients whose anonymized data were used for this study received no compensation. All patient data were processed anonymously and stored on a secure server with restricted access, in accordance with data protection laws and regulations.

The AI support model for decision prediction in the ED was an Extreme Gradient Boosting with an accuracy of 0.81, precision of 0.78, recall of 0.73, F₁-score of 0.75, and a receiver operating characteristic area under the curve of 0.89 on the test dataset. The final hyperparameters used for this model included a colsample_bytree of 0.7, γ of 0.0, learning rate of 0.1, max_depth of 15, and min_child_weight of 7.

The median durations of admission order placement by health care professionals were compared to the time required by the AI model on retrospective data to make equivalent decisions. Health care professionals required a median time of 151 (IQR 95-228) minutes to make admission decisions, decreasing to 131 (IQR 75-201) minutes for the admitted patient population. In contrast, the AI model achieved a median decision time of 20 (IQR 0-40) minutes for the correct prediction. This represented a time saving of 111 minutes per patient for correct predictions when using the AI model.

Figure 1 shows the performance metrics of the AI model. It illustrates that the quality of the admission predictions evolves over time. Initially, the model failed to capture many cases, but it eventually achieved a precision of 80% (24,696/30,870). Nevertheless, it still generated 30% (9261/30,870) to 35% (10,805/30,870) false positives, which may lead to excessive and unnecessary alarms for the medical staff. These metrics are slightly different from the general performance, since a stricter admission rule was applied; once the model admits a patient, this decision cannot be reversed. These results also showed that, later during the ED stay, the AI model made fewer mistakes than earlier.

Our study addresses the potential for integrating an AI decision model into clinical practice by not only developing an AI model using Extreme Gradient Boosting but also evaluating its clinical relevance through a 10-minute evaluation dataset. Many models rely on static inputs and technical performance, without addressing integration into clinical workflows [26-32]. Unlike previous studies, our model revises its admission decision at 10-minute intervals, enhancing clinical relevance and facilitating seamless integration into the clinical workflow.

It demonstrated that using AI to support the physicians in the ED has the potential to reduce time to an admission decision by 111 (IQR 59-169) minutes per correctly predicted patient, thereby improving the quality of care and reducing pressure on hospital resources. The model achieves an accuracy of 0.81, an F₁-score of 0.75, and a receiver operating characteristic area under the curve of 0.89. Nevertheless, these findings should be interpreted with caution, as both clinical relevance and performance are likely to be lower when implemented in an actual clinical workflow compared to existing literature [28,32]. Unlike previous work, our approach integrates iterative data updates every 10 minutes and checks the clinical impact of AI decision-making. This study provides a practically oriented contribution by demonstrating how AI can support timely decision-making, especially for less experienced clinicians.

Importantly, the model does not negatively impact patient safety. In the case of a true positive prediction, the patient is transferred to the correct department quickly and receives more specific treatment faster. In the case of a false negative prediction, the patient is transferred to the department as quickly as they would have without the AI model. In the case of a false positive prediction, it only leads to additional work for the employees. However, in this hospital’s case, this did not lead to clinically significant consequences.

The findings suggest that AI models can be effectively used to enhance the decision-making processes in the ED, leading to reduced time to admission decisions and potentially improving patient outcomes.

One data limitation of this study is that the model does not include radiological image results, blood gas, and free-text clinical notes, all of which are critical for a comprehensive patient assessment. Incorporating these data types could improve the model’s metrics and reliability. Additionally, the model does not account for data from previous appointments, which could provide valuable context and insights into a patient’s history and potential risks.

Another notable consideration is the potential consequence of faster admission decisions from the ED, namely a false admission prediction. This could lead to an unnecessary order being sent to the urgency coordinator to arrange a bed that is ultimately not required, resulting in wasted time for staff and disrupting operational workflows. In consultation with the urgency coordinators, it was agreed that this is a consequence that has minimal impact on the St Antonius Hospital.

While the AI model shows promise in an ideal scenario, real-world implementation will still face several challenges. Even with AI recommendations, delays in placing orders are likely to occur due to factors such as health care professionals being occupied or requiring additional time to assess patients. Additionally, factors such as the department’s workload, patient flow, and the need for patient transfers within the region can further affect the time to admission decision. In practice, the time difference may not be as significant as predicted by the model alone. A combination of health care professionals and AI models will need to work together, and this interaction should be explored in a prospective study, which is planned [35]. This combination of AI and health care professionals is also what could improve the false positive rate and thus the model, compared to just working with the AI model.

A prospective study is recommended to evaluate the actual impact of the model on ED length of stay in a real-world setting. To improve predictive accuracy, such a study should assess the model's performance in the clinical setting. In addition, incorporating additional data sources such as imaging results and patient history, could enhance the model’s applicability in a real-world setting.

ED overcrowding poses a challenge to health care systems, contributing to delays in treatment, increased medical errors, and compromised patient outcomes. This study was motivated by the urgent need to expedite the decision-making process within the ED to reduce patient ED length of stay.

To address this, we developed and evaluated an AI-based decision support model capable of predicting hospital admissions from the ED. Unlike previous studies that primarily focused on technical model performance, our work emphasizes clinical relevance through real-time decision-making via 10-minute interval updates, mimicking the dynamics of actual ED workflows.

The model achieved a precision of 0.78 and a recall of 0.73. In a retrospective dataset, the AI model was able to reduce the median time to admission order by 111 (IQR 59-169) minutes for correctly predicted admissions, potentially alleviating ED overcrowding and improving patient care. In addition, it offers the advantage of consistently providing weighted advice on admission, even when the ED is under pressure.

These findings demonstrate that integrating AI decision support into clinical workflows has the potential to speed up decisions, reduce ED overcrowding, and thus improve patient care. Future prospective studies are essential to validate these results in real-world settings.