In this retrospective study, we evaluated the performance of ChatGPT, Gemini, and ECG Buddy in classifying MI from ECG images. A publicly available 12-lead ECG image dataset compiled by the Ch. Pervaiz Elahi Institute of Cardiology in Multan, Pakistan, was used [19]. The dataset includes ECG images categorized into the following 4 groups: patients with MI (239 images), patients with abnormal heartbeats (233 images), patients with a history of MI (172 images), and healthy controls (284 images). This publicly available, fully deidentified ECG image dataset was chosen to enable reproducible benchmarking without privacy constraints and is frequently referenced in prior studies. The dataset does not provide additional patient information beyond these labels. It lacks metadata such as infarct territory or cardiac biomarker data needed to differentiate STEMI from non-STEMI (NSTEMI). Therefore, further analyses by infarct location or NSTEMI status were not possible.

This study was designed and reported in accordance with the TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis specifically tailored for LLM) guidelines, a comprehensive reporting framework for studies involving LLMs in health care, to ensure that every step, from data processing and image-to-text conversion to AI querying and performance evaluation, was transparently and reproducibly documented [20]. To ensure consistency in data processing, extraneous areas of the images, including any supplementary text not related to patient information or diagnosis, were cropped, retaining only the waveform regions. No raster-to-signal conversion was applied, and all analyses were performed directly on the image data. For classification, only the images labeled as “patients with myocardial infarction” were designated as MI-positive, representing active MI cases. The remaining 689 images, comprising abnormal heartbeats, history of MI, and healthy cases, were classified as MI-negative.

The identical workflow was applied to 2 multimodal LLMs—GPT-4o (OpenAI, version November 20, 2024) and Gemini 2.5 Pro (Google, May 2025 release). To assess the ability of LLMs to classify MI from ECG images, we designed a structured prompt aimed at systematically capturing the model’s diagnostic rationale and confidence in detecting MI. Before querying, ECG images were converted into base64 format, a lossless binary-to-text format required by both application programming interfaces (APIs) and one that preserves every pixel and ensures no loss of ECG signal fidelity.

Each LLM received the base64-encoded images and prompted explicitly to analyze them, determine the likelihood of MI, and select from 5 predefined response categories—unknown, unlikely, possible, probable, and definite—representing increasing diagnostic certainty. Both LLMs were queried with the prompt shown in Textbox 1, which presents the ChatGPT-4o version; the Gemini 2.5 Pro prompt was identical, differing only in that the model name was replaced with “GPT-4o.”

To define a positive MI diagnosis based on this likelihood measure, the Youden index was applied to determine an optimal cutoff. Specifically, ChatGPT was instructed to identify and specify which ECG leads exhibited abnormalities, describe the changes observed (such as ST-segment elevation, T-wave inversion, or pathological Q waves), and provide detailed reasoning supporting its diagnostic conclusion. In cases where no abnormalities were noted or the ECG findings were ambiguous, the model was prompted to discuss potential alternative diagnoses or clearly explain why the ECG appeared normal. All queries were conducted using GPT-4o via the ChatGPT API. An example of a typical ChatGPT response is illustrated in Figure 1. All inferences were conducted as single-turn prompts, since each case consisted only of a deidentified ECG image with no ancillary clinical or serial ECG data that could support further interaction, and our study aimed to evaluate the models’ final diagnostic performance.

In addition, a qualitative assessment of the diagnostic explanations of LLMs was performed to further evaluate the ability to accurately interpret ECG images. For GPT-4o, 40 cases were randomly selected: 10 true-positive, 10 true-negative, 10 false-positive, and 10 false-negative. For Gemini 2.5 Pro, the same procedure yielded 37 cases (10 true-positive, 10 true-negative, 10 false-positive, and 7 false-negative) owing to the model’s smaller false-negative pool. Two board-certified clinicians, an emergency medicine specialist and a cardiologist, each with more than 10 years of ECG interpretation experience, independently reviewed the diagnostic explanations to assess whether the model provided clinically appropriate rationales for their classification. They then reconciled any discrepancies by consensus, and the final consensus ratings, together with per-reviewer tallies, are reported in the Results section.

ECG Buddy is a deep learning–based ECG analysis platform designed for 12-lead ECG image interpretation. The software is available for both smartphones and Microsoft Windows–based desktop personal computers. In this study, ECG Buddy for Microsoft Windows [21] was used to perform bulk analysis of ECG data (Figure 2). It is approved by the Korean Ministry of Food and Drug Safety and freely available for download in Korean app stores and can analyze 12-lead ECGs by taking pictures of ECG outputs to produce 10 digital biomarkers. The software automatically detects the ECG image displayed on the desktop and provides the analysis results within 10‐15 seconds. Figure 2A shows the operating screen of ECG Buddy for Microsoft Windows, while Figure 2B shows the ECG image analysis output. ECG Buddy generates 10 digital biomarkers that assess a range of cardiac conditions, including STEMI, acute coronary syndrome (ACS), myocardial injury (MyoInj), critical condition, pulmonary edema, pericardial effusion, left ventricular dysfunction, RV dysfunction, pulmonary hypertension, and severe hyperkalemia. This study analyzed only the STEMI, ACS, and MyoInj biomarkers owing to their direct relevance to MI classification.

Model performance was evaluated using accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). The Youden index was used to determine optimal classification thresholds. In addition, model performance was evaluated using the area under the receiver operating characteristic curve (AUROC), and the AUC values were compared using the DeLong method, with statistical significance set at P<.05. All analyses were conducted using R software version 4.1.0 (RStudio) [22], with ChatGPT API responses obtained using Python (Python Software Foundation).

The study design was approved by the Institutional Review Board of Seoul National University Bundang Hospital (IRBX-2504-966-902). Given the public availability of the dataset, the Institutional Review Board of Seoul National University Bundang Hospital granted a waiver for the requirement of informed consent.

In total, 928 ECG recordings (239/928, 25.8% MI-positive cases) were analyzed, and all were successfully processed by both AI models. ChatGPT demonstrated limited discriminative ability in MI detection, achieving an AUC of 57.34% (95% CI 53.44‐61.24). Using the Youden index, the optimal cutoff was determined as the category “definite.” At this cutoff, the model’s sensitivity, specificity, PPV, and NPV were 36.40% (95% CI 30.30‐42.85), 76.20% (95% CI 72.84‐79.33), 34.66% (95% CI 28.79‐40.90), and 77.55% (95% CI 74.21‐80.64), respectively (Figure 3A and Table 1). Gemini 2.5 Pro showed even weaker overall discrimination, with an AUC of 51.63% (95% CI 50.22‐53.04). Applying the same Youden index procedure, the optimal cutoff corresponded to the “definite” category. At this threshold, sensitivity rose to 97.07% (95% CI 94.06‐98.81) but at the expense of specificity, which fell to 6.24% (95% CI 4.55‐8.31); the resulting PPV and NPV were 26.42% (95% CI 23.53‐29.47) and 86.00% (95% CI 73.26‐94.18), respectively (Figure 3A and Table 1).

The dedicated ECG AI software ECG Buddy exhibited highly accurate MI classification across the STEMI, ACS, and MyoInj markers. The AUC for detecting MI-positive cases for the STEMI biomarker was 98.87% (95% CI 98.30‐99.43), for the ACS biomarker was 98.78% (95% CI 98.05‐99.50), and for the MyoInj biomarker was 98.88% (95% CI 98.24‐99.51). Using the STEMI biomarker, ECG Buddy achieved the best accuracy of 96.98% (95% CI 95.67‐97.99), with a sensitivity of 96.65% (95% CI 93.51‐98.54), specificity of 97.10% (95% CI 95.55‐98.22), and F₁-score of 94.27% (95% CI 91.86‐96.28). DeLong test confirmed that ChatGPT (AUC 53.63%) performed significantly worse than ECG Buddy across all biomarkers (all P<.001; Figure 3B and Table 1).

Two board-certified clinicians independently reviewed every explanation generated by the 2 LLMs. For GPT-4o, reviewer 1 judged 5% (2/40) explanations fully correct, 40% (16/40) partially correct, and 55% (22/40) completely incorrect. Reviewer 2 judged 5% (2/40) fully correct, 37.5% (15/40) partially correct, and 57.5% (23/40) completely incorrect. The reviewers concurred in 87.5% of GPT-4o cases (35/40; weighted κ=0.76). After consensus, GPT-4o explanations were fully correct in 5% (2/40), partially correct in 37.5% (15/40), and completely incorrect in 57.5% (23/40; Table 2).

For Gemini 2.5 Pro, the 2 clinicians showed high interrater agreement (91.9%, 34/37; weighted κ ≈ 0.67) while following the identical review procedure. Consensus ratings were 32.4% (12/37) fully correct, 13.5% (5/37) partially correct, and 54.1% (20/37) completely incorrect. Although Gemini produced a higher proportion of fully correct statements than GPT-4o, more than half of its explanations remained completely inaccurate, underscoring the need for expert oversight. The detailed per-rater counts are provided in Multimedia Appendix 1.

This study directly compared ChatGPT and Gemini, general-purpose multimodal LLMs, with ECG Buddy, a specialized deep-learning tool for ECG analysis. While ECG Buddy achieved high accuracy in detecting MI from ECG images, ChatGPT’s performance was significantly inferior. Gemini 2.5 Pro, the latest vision-language model from Google, showed even lower overall accuracy than GPT-4o, reinforcing the conclusion that current general-purpose LLMs remain unsuitable for primary ECG interpretation.

The considerable performance gap between the dedicated ECG Buddy and LLMs underscores a difference in their architecture and training methodologies. LLMs are primarily optimized for textual understanding and general visual recognition tasks and lack the specific training necessary for detailed ECG waveform interpretation. As a result, it may generate contextually plausible yet inaccurate responses, which could lead to potentially dangerous diagnostic errors if relied upon in clinical settings. Moreover, the performance of LLMs is highly sensitive to prompt design and the specific model version used, resulting in inconsistent outcomes. Clinical studies have reported that while LLMs perform moderately well in common clinical cases, they deviate significantly from evaluations in critical scenarios [10-12]. Conversely, ECG Buddy is optimized through targeted domain-specific training, resulting in superior performance and reliability compared to LLMs.

The findings of this study indicate that current general-purpose multimodal LLM architectures may primarily rely on textual annotations or explicit labels rather than on directly analyzing waveform patterns. Due to these structural limitations, current LLMs cannot be considered reliable as primary diagnostic tools for detecting STEMI. Accordingly, LLMs should be confined to a supplementary decision-support role, where they can supply guideline-based contextual information, augment the interpretations generated by specialized tools such as ECG Buddy, and propose clinically appropriate follow-up options. Although recent advances in LLMs have broadened the applications of medical AI, domain-specific models remain indispensable, as the inherent limitations of general-purpose LLMs still compromise clinical utility and reproducibility.

This study has some limitations. First, the study was conducted retrospectively using a publicly available ECG dataset, which lacked detailed clinical context or patient demographic information. The absence of comprehensive clinical data may limit the generalizability of these findings to diverse patient populations or different clinical settings. In addition, the dataset comprised only deidentified ECG images lacking infarct-territory labels and lab results, location-specific and STEMI or NSTEMI analyses were not possible. Second, the qualitative assessments were performed independently by 2 board-certified clinicians—1 emergency physician and 1 cardiologist. While providing useful insight, interpretations by multiple clinicians across various specialties might yield different assessments of diagnostic appropriateness or accuracy. To address this limitation and confirm its clinical utility, we plan to initiate prospective validation studies that will evaluate ECG Buddy’s diagnostic accuracy and workflow integration in emergency department settings across hospitals.

Our findings are consistent with those of prior research, highlighting that ECG-specialized AI models regularly outperform general-purpose models. Previous studies have demonstrated that deep learning models trained extensively on ECG-specific datasets accurately detect subtle waveform changes indicative of asymptomatic ventricular dysfunction and cirrhosis [23,24]. Similar to these specialized ECG models, ECG Buddy undergoes targeted optimization tailored specifically to ECG images, ensuring consistent predictive performance and stable error margins essential for clinical reliability. Notably, users only need to provide an ECG image or screenshot; both the smartphone and desktop versions feature an intuitive interface that requires no extra training or expertise. In addition to MI, ECG Buddy demonstrates robust diagnostic capabilities across diverse cardiac conditions, including STEMI, hyperkalemia, and RV dysfunction, validating its efficacy in various clinical settings [15-18]. Moreover, it has outperformed human experts in diagnosing STEMI and hyperkalemia.

To our knowledge, this study provides the first direct comparative assessment between ChatGPT, Gemini, and ECG Buddy for detecting MI from ECG images. Our findings reveal that, despite the accelerating use of LLM-based AI, current LLMs do not meet the clinical performance and accuracy requirements for ECG interpretation. Herein, the ability of a general-purpose multimodal LLM (ChatGPT and Gemini) to detect ECG abnormalities fell short of that achieved by board-certified emergency physicians, rendering it insufficient for use in interpreting critical ECG readings in clinical practice. In contrast, the specialized ECG AI tool (ECG Buddy), which has been trained and validated, achieved high diagnostic accuracy and reproducibility, suggesting its utility in clinical settings. These results, consistent with those of several other studies, underscore the superiority of medical domain-specific AI programs over general-purpose LLMs for ECG analysis and interpretation and emphasize the importance of specialized models in the field of medical AI development.