Evaluating an AI Decision Support System for the Emergency Department: Retrospective Study

Introduction

Background

Emergency department (ED) crowding is a growing problem that can lead to the deterioration of the quality of health care. This concern is associated with a rise in medical errors made by clinicians [1,2], prolonged patient stay [3], morbidity [4,5], and increased mortality rates [6-8]. In some cases in the Netherlands, standards of health care were not met, resulting in the temporary closure of EDs [9]. The issue of overcrowding is expected to become even more evident in the coming years due to increased life expectancy and increased demand for complex care [10,11].

In recent years, studies have been extensively exploring the issue of ED crowding [12-14]. In a conceptual model, ED crowding is divided into three interdependent components: (1) the input component, (2) the throughput component, and (3) the output component [15]. Changes in one of these components can contribute to the ED length of stay and therefore ED crowding [16]. Each component comprises multiple factors that can influence the overcrowding problem both independently and through interaction with factors within or outside the component [13]. Focusing specifically on the throughput component, key factors have been identified, namely the experience level of staff [17], shortages of staff within the ED [18,19], availability of beds in the ED [20], delays in test results, and disposition decisions [21].

To address the problem of overcrowding, solutions can be pursued both within the ED and through broader changes at the hospital. Within the ED, particularly concerning the throughput component, studies have investigated, for example, the implementation of fast-track systems [22,23], adjustments in triage models [24], and the rising application of artificial intelligence (AI) solutions [25].

Currently, several studies are investigating the potential of AI solutions to mitigate the overcrowding problem in the ED. Some promising results have been reported in areas of patient admission to inpatient units and intensive care units or discharge from the ED, thereby impacting the duration of stay in the ED [21]. However, these AI models can vary significantly in their functioning, often using diverse parameters at various time points during ED admission [26-32]. While these studies tend to focus on the technical performance of the models, they often neglect to consider their practical relevance and applicability within health care settings [33-35].

In this study, an AI model was developed for decision support in the ED. Moreover, the retrospective model predictions were evaluated with updates occurring every 10 minutes based on the most current patient data. This AI model showed the health care professionals and residents the probability of admission to a hospital ward from the ED.

Aim

The clinical relevance of the AI decision support system was evaluated by analyzing the decision-making time. This evaluation involves retrospectively examining whether the model can reduce the time required for an admission decision, thereby potentially decreasing ED length of stay and alleviating the overcrowding problem.

Methods

Study Design

In this retrospective study, anonymized patient records from the Dutch St. Antonius Hospital were collected from January 2018 to September 2023. The St. Antonius Hospital in the Netherlands has 2 different locations where emergency care is provided, and it is a level 2 trauma center in an urban setting. A total of 41,000 patients present to the EDs each year at the St. Antonius Hospital. The data up until May 2022 were earmarked for the development and assessment of our AI model.

As the emphasis of this study was on the clinical impact rather than the complexities of model development and evaluation, detailed information regarding the development of the model is provided in Multimedia Appendix 1, which includes a comprehensive overview of the preprocessing steps, feature selection, and models considered during the development phase. The development of the model was based on the study by De Hond et al [32]. Admissions in this study comprised patients treated in the EDs of St. Antonius Hospital. Patients who explicitly declined to provide consent for the use of their data in any research context were excluded. Additional exclusion criteria encompassed patients younger than 18 years.

The AI model predicts the admission probability as a percentage. If this percentage exceeds 50%, the model classifies the case as “admission.” Once the threshold is reached, the decision is final and cannot be reversed. This design choice was made to mimic clinical commitment, thereby reflecting a realistic clinical decision-making context in which a patient admission decision is typically irreversible once made.

Data Collection

Features were extracted from the dataset by performing several steps during the data cleaning and transforming phase. A detailed list of these features, along with a comprehensive explanation of the data-cleaning process, is provided in Table S1 in Multimedia Appendix 1. For model development, the dataset collected between January 1, 2018, and May 15, 2022, was split in an 80:20 ratio randomly; 80% (105,000/131,250) of the data was used for training the AI model, and 20% (26,250/131,250) was used for testing.

To evaluate the AI model’s performance in predicting ED admissions, a dataset including all ED visits from May 16, 2022, to September 1, 2023, was created. This dataset was designed to mimic real-world scenarios, allowing us to assess the model’s clinical performance in a controlled retrospective setting. This dataset was referred to as the evaluation dataset. Admission data were divided into 10-minute intervals, starting from the initial 0 minutes up to 3 hours. This segmentation reflected clinical decision-making by providing the model with the most recent information available at each time point. By checking new predictions every 10 minutes, we could analyze how changes in these predictions supported clinical decision of admission to the ward. This method tested the model’s ability to make accurate predictions with limited and progressively updated information, reflecting typical emergency settings.

Thus, 3 datasets were used: 1 for training the model, 1 for testing during the development phase, and 1 evaluation dataset to calculate the time saved by the AI model compared to admission decisions made by ED health care professionals.

Model Evaluation

General Performance

Different analyses were conducted using the test and the evaluation datasets. General performance metrics such as accuracy, precision, and recall, were calculated using the test dataset.

Primary Outcome

The primary outcome measure was the difference in decision-making time between the AI model and the admission decision made by the clinician. The decision of the clinician was defined as the duration from a patient’s arrival at the ED to the time an admission order was placed or when the patient was informed by a health care professional that admission was not required, and discharge was appropriate. This outcome was evaluated against the AI model’s prediction when the predicted admission probability exceeded 50%. When the probability of admission exceeded the threshold, the model predicted an admission and could not revert to “discharge from ED” prediction. This influenced the metrics over time. The health care professional’s admission order served as the benchmark for this comparison. This analysis was performed using the evaluation dataset.

Secondary Outcome

The secondary outcome focused on the full patient group and true positive patients predicted by the AI model across various subcategories and baseline calculations. The patients correctly assessed by the model as admitted were classified as the true positive category. These baseline calculations represented the majority class assumption. These subcategories included age groups, medical specialties, triage categories, and different parts of the day. These results were obtained using the evaluation dataset, including the constraint that once the model predicts admission, it cannot be revised to discharge. Each subcategory was analyzed to determine its impact on admission decision time. These categories were also cross analyzed to assess coherence. These categories were chosen based on the assessment of importance by an ED clinician.

The ages of patients upon their arrival at the ED were analyzed. Patient ages were recorded and categorized into 10-year intervals such as 18 to 27, 28 to 37, and other age groups.

The medical specialty assigned at the time of ED arrival was examined, as different specialties could lead to varied outcomes. In this study, specialties included the top 10 presented specialties in the ED.

Upon arrival, patients were assigned a triage category indicating the urgency of their condition. The triage categories in the Netherlands range from U0 to U5, with U0 being the highest critical state of health.

The data were analyzed based on the arrival time of day, which may influence outcomes due to varying compositions of staff present in the ED. The data were segmented into 4 periods: night (midnight to 6 AM), morning (6 AM to noon), afternoon (noon to 6 PM), and evening (6 PM to midnight).

Baseline calculations were also performed for each subcategory to provide a reference point. This baseline represented the scenario in which all cases were assumed to result in admission (ie, if most patients are admitted, then a “yes, admit this patient” prediction for all patients is made). For each subcategory, the percentage of correct predictions under this baseline assumption was calculated and compared against the AI model’s performance. This showed us how the current situation could potentially be adapted and improved.

In addition, the feature importance of the model was examined to assess the impact of individual features on the model’s inclusion decision. A higher value indicated a greater contribution to the final decision of the model. These findings can inform the selection of features for future models and support clinical decision-making.

Ethical Considerations

This study was reviewed and approved by the St. Antonius Hospital’s local review committee (approval R&D/Z24.050). No informed consent from patients was required for this study, as it did not involve any additional risks or burdens for patients. Patients whose anonymized data were used for this study received no compensation. All patient data were processed anonymously and stored on a secure server with restricted access, in accordance with data protection laws and regulations.

Results

General Performance

The AI support model for decision prediction in the ED was an Extreme Gradient Boosting with an accuracy of 0.81, precision of 0.78, recall of 0.73, F₁-score of 0.75, and a receiver operating characteristic area under the curve of 0.89 on the test dataset. The final hyperparameters used for this model included a colsample_bytree of 0.7, γ of 0.0, learning rate of 0.1, max_depth of 15, and min_child_weight of 7.

Primary Outcome

The median durations of admission order placement by health care professionals were compared to the time required by the AI model on retrospective data to make equivalent decisions. Health care professionals required a median time of 151 (IQR 95-228) minutes to make admission decisions, decreasing to 131 (IQR 75-201) minutes for the admitted patient population. In contrast, the AI model achieved a median decision time of 20 (IQR 0-40) minutes for the correct prediction. This represented a time saving of 111 minutes per patient for correct predictions when using the AI model.

Figure 1 shows the performance metrics of the AI model. It illustrates that the quality of the admission predictions evolves over time. Initially, the model failed to capture many cases, but it eventually achieved a precision of 80% (24,696/30,870). Nevertheless, it still generated 30% (9261/30,870) to 35% (10,805/30,870) false positives, which may lead to excessive and unnecessary alarms for the medical staff. These metrics are slightly different from the general performance, since a stricter admission rule was applied; once the model admits a patient, this decision cannot be reversed. These results also showed that, later during the ED stay, the AI model made fewer mistakes than earlier.

‎

Secondary Outcomes

Age

Figure 2 and Table 1 show that younger patients (aged 18-27 y) had a median current time of 137 minutes in the ED, with a substantial improvement in time saved per patient (100 min). The precision of 0.51 and the recall of 0.46 suggested that younger patients were often misclassified. In contrast, the older adult population (aged 78-87 y and ≥88 y) presented the greatest clinical impact. Their admission times (for the true predicted patients) were reduced by 120 and 110 minutes per patient, respectively, and the model achieved a strong precision of 0.75 and 0.78 and a recall of 0.90 and 0.91.

For the 18- to 27-year age group, the model’s accuracy (84%) matched the baseline of assuming no admissions (84%). For all other age groups, the model consistently outperformed the baseline.

‎

Medical Specialty

Table 2 shows that pulmonology and gastrointestinal and liver disease cases showed a recall rate of higher than 0.90 and a precision value of more than 0.7. With this balance between recall and precision, the model ensured that the clinical risks of missed admissions (false negatives) were minimized, while the clinical impact of unnecessary admissions (false positives) on hospital capacity remained manageable. By contrast, specialties such as neurology, surgery, otorhinolaryngology, and internal medicine demonstrated greater challenges. For example, neurology had a recall of 0.82 but a lower precision of 0.52, with a significant number of false positives (1438/4659, 30.9%). This resource burden reflected the difficulty in assessing neurological symptoms. Figure 3 shows that across all specialties, the model consistently outperformed the baseline accuracy. It also shows that in neurology and cardiology, the admission decision time was 0 minutes for the AI model.

‎

Triage Categories

Figure 4 and Table 3 show that the model performs particularly well for the most critical patients (U0 and U1 categories), where the precision and recall ensure that almost all high-risk admissions are caught in time, saving crucial minutes. The system saves 80 minutes for the U0 category and 100 minutes for the U1 category per true positive predicted patient. However, challenges emerged in the U3 and U4 categories, where lower precision and recall indicated a notable number of false positives and false negatives. In such cases, unnecessary admissions could burden resources, while missed cases could endanger lives, indicating that improvements in midtier urgency cases could significantly impact ED efficiency. The baseline slightly outperformed the model for the U0 category, whereas the model excelled in all other triage categories. Figure 4 shows that the admission decision time was 0 minutes for the U0 and unknown categories when using the AI model.

‎

Part of the Day

Table 4 shows that performance varied depending on the time of day. Between noon and 6 PM, when the ED experienced its highest patient volume, the model achieved its best overall performance. A recall of 0.79 indicated that most patients needing admission were accurately flagged, saving, on average, 100 minutes per (true positive predicted) patient. Conversely, during quieter night shifts (midnight to 6 AM), the model’s precision decreased to 0.67. However, it exhibited a higher recall rate compared to the afternoon shift. Figure 5 shows that across all time periods, the model outperformed the baseline.

‎

Feature Importance

The results of the feature importance are shown in Multimedia Appendix 2. Table S1 in Multimedia Appendix 2 shows the top 20 most influential features, demonstrating that orders for inflammation, orders for kidney function, orders for blood count, and orders for blood cultures had the strongest influence.

Subcategory Coherence

Further analysis was conducted to evaluate the coherence of subcategories within the datasets. This additional layer of analysis aimed to ensure consistency in the results and provided a deeper understanding of the underlying patterns (Figures 6-11). This is presented in Multimedia Appendix 3.

‎

‎

‎

‎

‎

‎

Discussion

Principal Findings

Our study addresses the potential for integrating an AI decision model into clinical practice by not only developing an AI model using Extreme Gradient Boosting but also evaluating its clinical relevance through a 10-minute evaluation dataset. Many models rely on static inputs and technical performance, without addressing integration into clinical workflows [26-32]. Unlike previous studies, our model revises its admission decision at 10-minute intervals, enhancing clinical relevance and facilitating seamless integration into the clinical workflow.

It demonstrated that using AI to support the physicians in the ED has the potential to reduce time to an admission decision by 111 (IQR 59-169) minutes per correctly predicted patient, thereby improving the quality of care and reducing pressure on hospital resources. The model achieves an accuracy of 0.81, an F₁-score of 0.75, and a receiver operating characteristic area under the curve of 0.89. Nevertheless, these findings should be interpreted with caution, as both clinical relevance and performance are likely to be lower when implemented in an actual clinical workflow compared to existing literature [28,32]. Unlike previous work, our approach integrates iterative data updates every 10 minutes and checks the clinical impact of AI decision-making. This study provides a practically oriented contribution by demonstrating how AI can support timely decision-making, especially for less experienced clinicians.

Importantly, the model does not negatively impact patient safety. In the case of a true positive prediction, the patient is transferred to the correct department quickly and receives more specific treatment faster. In the case of a false negative prediction, the patient is transferred to the department as quickly as they would have without the AI model. In the case of a false positive prediction, it only leads to additional work for the employees. However, in this hospital’s case, this did not lead to clinically significant consequences.

Limitations

The findings suggest that AI models can be effectively used to enhance the decision-making processes in the ED, leading to reduced time to admission decisions and potentially improving patient outcomes.

One data limitation of this study is that the model does not include radiological image results, blood gas, and free-text clinical notes, all of which are critical for a comprehensive patient assessment. Incorporating these data types could improve the model’s metrics and reliability. Additionally, the model does not account for data from previous appointments, which could provide valuable context and insights into a patient’s history and potential risks.

Another notable consideration is the potential consequence of faster admission decisions from the ED, namely a false admission prediction. This could lead to an unnecessary order being sent to the urgency coordinator to arrange a bed that is ultimately not required, resulting in wasted time for staff and disrupting operational workflows. In consultation with the urgency coordinators, it was agreed that this is a consequence that has minimal impact on the St Antonius Hospital.

Considerations (of Implications) for Implementation

While the AI model shows promise in an ideal scenario, real-world implementation will still face several challenges. Even with AI recommendations, delays in placing orders are likely to occur due to factors such as health care professionals being occupied or requiring additional time to assess patients. Additionally, factors such as the department’s workload, patient flow, and the need for patient transfers within the region can further affect the time to admission decision. In practice, the time difference may not be as significant as predicted by the model alone. A combination of health care professionals and AI models will need to work together, and this interaction should be explored in a prospective study, which is planned [35]. This combination of AI and health care professionals is also what could improve the false positive rate and thus the model, compared to just working with the AI model.

Future Directions

A prospective study is recommended to evaluate the actual impact of the model on ED length of stay in a real-world setting. To improve predictive accuracy, such a study should assess the model's performance in the clinical setting. In addition, incorporating additional data sources such as imaging results and patient history, could enhance the model’s applicability in a real-world setting.

Conclusions

ED overcrowding poses a challenge to health care systems, contributing to delays in treatment, increased medical errors, and compromised patient outcomes. This study was motivated by the urgent need to expedite the decision-making process within the ED to reduce patient ED length of stay.

To address this, we developed and evaluated an AI-based decision support model capable of predicting hospital admissions from the ED. Unlike previous studies that primarily focused on technical model performance, our work emphasizes clinical relevance through real-time decision-making via 10-minute interval updates, mimicking the dynamics of actual ED workflows.

The model achieved a precision of 0.78 and a recall of 0.73. In a retrospective dataset, the AI model was able to reduce the median time to admission order by 111 (IQR 59-169) minutes for correctly predicted admissions, potentially alleviating ED overcrowding and improving patient care. In addition, it offers the advantage of consistently providing weighted advice on admission, even when the ED is under pressure.

These findings demonstrate that integrating AI decision support into clinical workflows has the potential to speed up decisions, reduce ED overcrowding, and thus improve patient care. Future prospective studies are essential to validate these results in real-world settings.