We conducted a single-center, multisite retrospective cohort analysis of nonpregnant adult (aged ≥18 years) ED patients who had a urinalysis and urine culture performed during their ED visit and were ultimately discharged from the ED. We excluded pregnant patients because of distinct guideline recommendations on the treatment of asymptomatic bacteriuria in this population [36]. We limited our study to discharged patients as this group represents relatively lower-risk patients less likely to receive empiric antibiotics for undifferentiated suspected infections. Data were collected between June 2013 and August 2021 from 9 EDs in a single northeastern US regional health network.

All data for model development and assessment were obtained from the system-wide electronic health record (EHR) data warehouse (Epic). Extracted variables included patient demographics, presenting concerns, ED diagnoses, prescriptions, ED dispositions, urinalysis results, and clinical notes to identify UTI signs and symptoms through natural language processing (NLP) [37]. We included missing values without further imputation. Over the course of the study, various study sites used several different scales and conventions to report components of urinalysis results. We reconciled various ordinal (qualitative and semiquantitative) and quantitative scales into a single ordinal scale for each urinalysis result component.

We considered a urine culture to be positive if it grew 10,000 or more colony-forming units per milliliter of a pathogenic bacterial organism [38-41] (see the organism list in Multimedia Appendix 1). This threshold is consistent with a recent multidisciplinary Delphi consensus study [39] and is our health system laboratories’ cutoff for reporting organisms in urine cultures.

In each of the analyses described below, we used a “strict” UTI definition that, consistent with society guidelines and interdisciplinary consensus statements [39,42,43], required both culture positivity (see the definition above) and the presence of at least one UTI sign or symptom (see the list in Multimedia Appendix 1) as identified through NLP of the ED clinician’s note from the ED visit in question (see the brief UTI definitions in Table 1). By requiring the presence of a UTI sign or symptom, we distinguished true UTI from asymptomatic bacteriuria, which typically does not require treatment.

Using methods previously described [37], we used a transformer-based large language model (Clinical-Longformer [44]) fine-tuned on ED notes to identify signs and symptoms of UTI through the NLP task of named entity recognition. In our prior work, this model showed excellent performance in identifying the presence of any UTI signs or symptoms at the note level, with precision of 0.97 (95% CI 0.93‐0.98), recall of 0.99 (95% CI 0.96-0.99), and F₁-score of 0.98 (95% CI 0.95-1.0) when evaluated in the same setting [37]. The included UTI signs and symptoms were selected based on literature review [15,45,46], society guidelines [42,43], and expert opinion.

As a sensitivity analysis, we also used a “liberal” UTI definition requiring only urine culture positivity (based on the aforementioned definition) using the same urine culture prediction model as that in our “strict” UTI definition (Table 1). This liberal definition was used to account for variances in patient presentations, clinician documentation, and NLP performance and to give clinicians the benefit of the doubt assuming that a urine culture was only ordered if a positive result would warrant treatment.

Because physicians vary in their documentation of visit diagnoses and antibiotics are prescribed for many diagnoses apart from UTIs, we created a composite proxy outcome to capture presumed physician UTI diagnosis. We considered a physician to have diagnosed their patient with a UTI in one of two conditions: (1) the patient was given an explicit diagnosis of a UTI (cystitis, pyelonephritis, unspecified UTI, or a synonym) or (2) they received a prescription for an antibiotic that could presumably treat a UTI and were given a nonspecific diagnosis that could likely be attributed to a UTI and were not given an alternative diagnosis that would explain their antibiotic prescription. Lists of included antibiotics, UTI symptoms, and alternative diagnoses were created based on society guidelines [45], published literature [38-41], reference texts [47,48], and expert consensus between study authors (MI and RAT) and can be found in Multimedia Appendix 1.

We used an intersectional lens [8] to examine equity in UTI diagnosis across patient groups. In defining intersections, patients were grouped by sex [49,50] and age [50,51] based on known epidemiologic risk factors between these groups as well as evidence on varied UTI treatment guideline adherence based on age [11,14] and race and ethnicity given some evidence on disparities in guideline adherence across these axes [14,52]. Age was binned into 4 groups: 18 to 45 years, 45 to 64 years, 65 to 84 years, and 85 years or above. Race and ethnicity were obtained from the EHR database and were based on patients’ self-reported responses to a prompt at health system registration, with options of “Hispanic or Latina/o/x,” “non-Hispanic,” and “unknown” for ethnicity and “American Indian/Alaska Native,” “Asian,” “Black or African American,” “multiracial,” “Native Hawaiian/Pacific Islander,” “other/not listed,” and “White or Caucasian” for race. Sex data were also obtained from the EHR database based on patients’ self-reported responses to a prompt at health system registration.

All patients were included in model development and overall model evaluation. However, in examining model performance across intersectional groups, we limited our analyses to the 19 intersectional groups with at least 2000 patients to ensure adequate sample size for meaningful comparison.

Patients were not directly involved in research question or outcome measure development.

The Yale School of Medicine Institutional Review Board approved this research and waived the need for informed consent (1602017249). All data was deidentified prior to analysis and stored on secure servers to maintain confidentiality. Participants were not compensated.

A total of 149,449 encounters met the inclusion criteria, of which 22,521 (15.1%) had a positive urine culture, meeting our liberal UTI definition; of these, 20,080 (89.2%; 20,080/149,449, 13.4% of total cases) had at least one UTI sign or symptom as identified through NLP, meeting our strict UTI definition. Median age was 50 years (IQR 31-67); 71.2% (106,380/149,449) of the patients were female; 56.2% (84,005/149,449) were White individuals; 22.1% (33,070/149,449) were Black individuals; and 25.6% (38,231/149,449) were Hispanic or Latina, Latino, or Latinx. Full baseline patient characteristics, UTI symptoms as identified through NLP, and urine culture results are shown in Table 2. Model feature completeness and missingness are provided in Multimedia Appendix 1.

A total of 19 intersectional groups of age bracket, sex, race, and ethnicity met our threshold of 2000 encounters (see the tree map in Figure 1). Physician and model diagnostic performance by group is shown in Figure 2.

A UTI diagnosis was made in 63.4% (12,735/20,080) of total cases with a UTI. Physicians had an overall false-positive rate of 9.1% (11,807/129,369) and an overall false-negative rate of 36.6% (7345/20,080).

The XGBoost classifier model achieved an ROC-AUC of 0.93 (95% CI 0.93-0.93). The area under the PR curve was 0.74 (95% CI 0.74-0.74). The receiver operating characteristic curve, PR curve, calibration curve, and ROC-AUC by intersectional group are shown in Figure 3. Model performance across intersectional groups using the decision threshold of 28% predicted the probability of a positive culture, the highest integer threshold at which the model’s false-negative rate for each intersectional group was lower than that of physicians (see the Selection of Decision Threshold section), is shown in Figure 2. At this decision threshold, the model also had a lower false-positive rate than physicians for each intersectional group; overall model false-positive rate at this threshold was 7.8% (95% CI 7.6%-7.9%). and the false-negative rate was 32% (95% CI 30.9%-32.1%). Overall model accuracy was 0.89, and DOR was 25.8. The “optimal” decision threshold based on the aforementioned definition was identified at 15%. Model performance according to the “liberal” UTI definition (requiring culture positivity alone) and at other decision thresholds can be found in Multimedia Appendix 1. Multimedia Appendix 1 also shows rates of culture positivity for each decile of model-predicted likelihood.

The CV for model performance across intersectional groups at a decision threshold of 28% was 0.039 (95% CI 0-0.36) for accuracy and 0.48 (95% CI 0.14-0.81) for DOR. Physicians had a CV of 0.080 (95% CI 0-0.40) for accuracy and 0.33 (95% CI 0.004-0.66) for DOR.

A SHAP value waterfall plot depicting the features’ importance on model predictions and the directionality of that impact is shown in Figure 4.

In this retrospective observational study of nonpregnant adult patients discharged from the ED, we found that an XGBoost machine learning model trained solely on urinalysis results and the clinical site performing the urinalysis had excellent performance in predicting urine culture positivity. When combined with NLP to detect UTI signs and symptoms, the model had lower over- and underdiagnosis rates than our composite proxy for physicians’ real-world UTI diagnoses for each intersectional group defined by age, sex, race, and ethnicity, with similar variation in performance across groups. These findings suggest that an AI model has the potential to augment ED care by decreasing both over- and underdiagnosis of UTI with comparable variation in diagnostic equity to that of current physician performance. Our predictive model performed at the high end of the range reported in a recent meta-analysis of 14 UTI prediction models, which found a pooled area under the curve of 0.89 (95% CI 0.86-0.92) [61], and to our knowledge, our study is the first to report performance across intersectional groups.

In contrast to some prior examples of instances in which AI has been shown to introduce or propagate bias, we suspect that our model had a relatively stable performance across groups because (1) it was trained on an objective gold standard (ie, a laboratory result rather than one involving physician judgment) and (2) the most impactful predictors as indicated by SHAP values—urinalysis bacteria, nitrite, leukocyte esterase, and white blood cells—were not only objective but also biologically related to the outcome of interest. In a widely cited 2019 study by Obermeyer et al [3], a commercial algorithm used to identify patients with complex health needs was found to exhibit substantial racial bias, largely because the model was trained using health costs as a proxy for health need and costs are not equitable across demographic groups. In contrast, this study used a direct, objective outcome (urine culture results) rather than a proxy, reducing the risk of bias. A 2022 study by Juhn et al [62] examining the performance of a machine learning model to predict pediatric asthma exacerbations found that the model performed worse in children with lower socioeconomic status, which the authors postulated was due to greater missingness in EHR data pertaining to past medical history. In a 2023 study examining the effects of simulated missing data in a cohort of intensive care unit patients, Getzen et al [63] found that missingness more negatively impacted disease prediction model performance in groups with less health care access. Because the various components of the urinalysis are routinely performed and reported together, there was relatively low missingness in our study and no reason to suspect inequitable distribution of missing data based on health care access or other factors, decreasing the risk of missingness bias.

While our model performed strongly in all intersectional groups, similar to physicians, it did not perform equally across groups. In particular, we noted that overdiagnosis was more common in older adult patients; for both the model and physicians, the highest rates of overdiagnosis were observed in male and female White, non-Hispanic or Latina, Latino, or Latinx patients aged 85 years or older, the only 2 groups in this age bracket who met the sample size threshold for inclusion in the intersectional analysis; these 2 groups also had the lowest rates of underdiagnosis for both physicians and the model. Conversely, the 3 groups with the lowest rates of overdiagnosis for both physicians and the model comprised male individuals aged 18 to 44 years. Potential explanations for these age- and sex-related disparities include baseline differences in UTI rates; difficulty obtaining a clean-catch specimen in older adults [16,64]; varying rates of bacterial colonization [31,65]; and, in the case of physician decision-making, diagnostic biases and varying risk-benefit calculations guiding testing and treatment thresholds. Prior literature has similarly demonstrated decreased model performance in older adult populations in other domains [66]. Future work should explore the creation of a predictive model specific to this high-risk group.

In our sensitivity analysis using a “liberal” UTI definition that relied solely on urine culture results and did not require the presence of UTI signs or symptoms, the model also had lower over- and underdiagnosis rates when compared to the proxy for physician diagnosis for each intersectional group, although the difference between model and physician overdiagnosis rates was less pronounced than for the strict UTI definition. This finding illustrates that the model’s strong performance relative to physicians was driven not just by physicians’ misdiagnosis of asymptomatic bacteriuria as UTI but also by physicians’ incorrect predictions of urine culture positivity. By removing the requirement for UTI signs or symptoms, we also eliminated the possibility that our analysis incorrectly classified patients as not having had a UTI due to NLP errors, incomplete documentation, or case-specific clinical nuances. The decision to include or exclude the NLP component of the model in future real-world applications would depend on the use case. We believe that NLP is valuable for disease and treatment surveillance as it can help establish the gold-standard diagnosis; however, its utility may be lower in real-time clinical decision support as clinicians are likely already familiar with the details of their patients’ presentations and the documentation required for NLP often lags behind clinical decision-making and diagnosis. Looking ahead, as ambient technologies are increasingly used in clinical care, real-time incorporation of clinical data may become more feasible.

A novel observation in this study is that variation in the model’s diagnostic performance across intersectional groups differed depending on both the decision threshold selected and the test characteristic evaluated such that, for some test characteristics and decision thresholds, the model demonstrated less cross-group variance than physicians and, for others, it demonstrated increased variance (Multimedia Appendix 1). These findings suggest that evaluations of algorithmic performance and equity should carefully consider the clinical situation in which a model will be applied, taking into account anticipated decision thresholds and which test characteristics to prioritize, among other factors. This calculation requires an understanding of the relative risks and benefits associated with correct classification and misclassification for positive and negative cases and may vary by patient and scenario, often including information that is not captured in the EHR.

From the standpoint of resource use, our model’s performance suggests that one potential application could be avoidance of urine culture testing in very low-risk patients. Of note, 60% (89,741/149,449) of patients had predicted likelihoods of urine culture positivity below 10%; in this population, the rate of urine culture positivity was 1.1% (979/89,741), which is likely below the threshold for testing for most clinicians in many clinical scenarios. Future applications could consider using AI to help guide decision-making on urine culture testing, potentially reducing low-value testing.

Our study has several limitations. First, we should note that the decision threshold we analyzed was a policy-constrained threshold that does not reflect an analysis of the relative benefits and risks associated with appropriate diagnosis and treatment, overdiagnosis, and missed or delayed diagnosis. The ideal threshold likely varies with clinical and patient factors, and future work could explore decision analysis to better guide population- and patient-level care.

Other limitations relate to the generalizability of our findings. While we included data from 9 EDs over nearly a decade, all sites were part of a single regional health network, and our findings, particularly regarding physician practice patterns, may not be applicable to all care settings; external validation would be necessary before broader implementation. An additional limitation to generalizability is that we only studied discharged patients with a urine culture ordered. Performance in admitted patients has not yet been evaluated, and clinicians may have different thresholds for treating these patients as they await urine culture results. We cannot directly comment on performance in patients who had a urinalysis but no urine culture as they lack outcome data.

Three additional limitations arise from the challenge of retrospectively identifying diagnoses from EHR data. First, in certain situations, clinicians may have identified and acknowledged the possibility of a UTI but elected to defer treatment (and formal diagnosis) until they had the results of the urine culture due to patient-specific risk-benefit considerations, such as concern regarding overprescription of antibiotics and associated downstream harms. Second, because physicians do not always explicitly code their UTI diagnoses, we developed a composite proxy outcome of explicit or presumed diagnosis, as detailed in the Methods section; while the large majority of patients diagnosed with UTI (22,574/24,542, 92%) had an explicit UTI diagnosis, some of those with a presumed diagnosis may have been misclassified if they were in fact prescribed antibiotics for an alternative diagnosis that was not explicitly made or was not among the alternative diagnoses that we included (Multimedia Appendix 1). Third, as discussed above, the “strict” UTI definition used in our primary analysis relied on NLP to identify signs and symptoms of UTI; while the model we used showed excellent performance in our prior evaluation [40], it may have misclassified some cases, including ones in which patients with UTIs had atypical symptoms or, conversely, had symptoms potentially attributable to UTI that had an obvious alternative etiology (eg, abdominal pain due to trauma). However, as noted above, the model also performed strongly in our sensitivity analysis using a “liberal” UTI definition that did not involve NLP.

This proof-of-concept study demonstrates that an AI algorithm trained on a small set of objective features that do not include patient demographics can have consistently strong performance across demographic intersectional groups and has the potential to augment clinicians’ decision-making. Nonetheless, performance varied across groups, and any real-world application should take that variance into account. Moreover, appropriate clinical application of AI in UTI diagnosis requires consideration of the case-specific diagnostic probability threshold at which treatment is warranted, as well as the presence or absence of clinical signs or symptoms that distinguish UTI from asymptomatic bacteriuria.