To systematically assess the performance of LLMs in the medical field, we adopted a multidimensional evaluation framework under MedBench, encompassing objective multiple-choice questions (single-select and multiselect) and subjective open-domain questions. Accuracy and robustness were used as core metrics for objective tasks, while key information recall and hierarchical error taxonomy were used for subjective tasks (Table 1).

For objective questions, model accuracy was calculated by directly matching LLM-generated options against ground-truth answers. The 10 LLMs (specifically AntAngelMed [13], Citrus-2.0 [14], INF-Med [15], WHU_Med, zhuomuniao-Med, TeleChat2 [16], hunyuan-med [17], UNI-GPT, fusiontech-Med, and GPT-4 [18]) evaluated in this study comprise both publicly available models and proprietary models submitted by developers. For publicly accessible models (eg, GPT-4 and hunyuan-med), we accessed them via their official APIs or open-source platforms (detailed in Table S1 in Multimedia Appendix 1). For proprietary or competition-specific medical LLMs (eg, INF-Med and WHU_Med), model developers interacted directly with our MedBench platform via our standardized evaluation API. To ensure evaluation consistency across both access methods, we applied uniform prompt templates (detailed in Table S2 in Multimedia Appendix 1). These templates explicitly instructed the models to output their final answers in a standardized format. We then used rule-based text parsing with regular expressions to extract the specific choices (eg, A, B, C, and D) from the models’ raw output generations. Any responses that failed to yield recognizable options or deviated entirely from the instructed format were handled as invalid outputs. To rigorously validate robustness, answer choices for each multiselect question were shuffled across 10 permutations. A response was deemed valid only if the model consistently identified the correct answer or answers across all permutations, ensuring resistance to positional bias.

Subjective responses were evaluated via macro-recall based on coverage of predefined key information points. To ensure evaluation objectivity, reproducibility, and analytical rigor, we systematically translated qualitative error descriptions into quantitative mathematical formulations. Errors were classified into 8 categories through a 3-expert consensus protocol using these formalized criteria: omissions (failure to address critical content), hallucination (fabricated claims), format mismatch (deviation from structured guidelines), causal reasoning deficiency (flawed logical chains), contextual inconsistency (contradictory statements), unanswered (no valid output), output error (technical failures), and deficiency in medical language generation (nonclinical phrasing). The detailed qualitative definitions, mathematical formulations, and illustrative examples are presented in Table 2.

This study involved the analysis of publicly available data and computational models (LLMs). It did not involve human participants, human data, human tissue, or animal subjects. Therefore, in accordance with the policies of JMIR AI, ethical approval from an institutional review board or research ethics board was not required. All methods were carried out in accordance with relevant guidelines and regulations.

From June to December 2024, standardized evaluations of 44,711 question responses generated by the top 10 models (AntAngelMed, Citrus-2.0, INF-Med, WHU_Med, zhuomuniao-Med, TeleChat2, hunyuan-med, UNI-GPT, fusiontech-Med, and GPT-4) across MedBench’s 5 competency dimensions revealed distinct capability patterns. As shown in Table 3, models achieved peak accuracy (0.86) in medical knowledge question answering, demonstrating robust comprehension of foundational concepts. Health care safety and ethics emerged as a secondary strength (accuracy 0.80), yet its robustness score (0.79) under shuffled permutations highlighted persistent inconsistencies in ethical decision-making. Aligned with our 8 newly defined error metrics, critical gaps persisted in medical language understanding (challenges in parsing specialized terminology, often leading to deficiency in medical language generation) and complex medical reasoning (limited multistep diagnostic inference, frequently resulting in causal reasoning deficiency and omissions).

A granular dissection of incorrect responses identified omissions of critical answer points as the dominant failure mode (4723/11,909 errors, 39.7%), reflecting systemic deficiencies in comprehensive reasoning despite superficially coherent outputs. Causal inference breakdowns (2037/11,909, 17.1%) and hallucinatory content generation (1913/11,909, 16.1%) constituted secondary yet consequential flaws, while format mismatch accounted for 1307 out of 11,909 errors (11%; Figure 1), underscoring the need for standardized output frameworks.

Domain-specific failure profiles further elucidated model limitations (Figure 2). In medical knowledge question answering, omissions dominated (1644/3703 errors, 44.4%), followed by hallucinations (868/3703, 23.4%), exposing weak knowledge boundary safeguards. For medical language understanding, 1984 out of 5784 errors (34.3%) stemmed from unaddressed contextual constraints, while 1303 out of 5784 errors (22.5%) arose from fractured causal reasoning chains, indicating deficits in clinical narrative modeling. Medical language generation exhibited severe hallucination (979/1530 errors, 64%), revealing semantic-clinical pragmatics disconnects. Strikingly, complex medical reasoning errors were overwhelmingly dominated by omissions (858/891 errors, 96.3%), fundamentally undermining reasoning reliability. As documented in Table 4, a granular error taxonomy delineates the classification schemas and clinical manifestations of these incorrect responses.

Leveraging the authoritative medical LLM evaluation system MedBench, we conducted an analytical assessment of current mainstream models in the health care field and compiled the incorrect responses from the top 10 models in the evaluation rankings.

The robustness score of 0.79 for health care safety and ethics under shuffled options permutations reveals concerning inconsistency in ethical decision-making scenarios. This could potentially be attributed to inadequate safety mechanisms or insufficiently comprehensive datasets concerning drug contraindications and medical ethics information [19]. For example, the model may generate incorrect drug recommendations, ignore the patient’s allergy history or drug contraindications, and directly threaten patient safety. At the same time, the model may lack an understanding of medical ethics, such as patient privacy protection or fairness, resulting in output that does not meet specifications. Therefore, enhance model safety protocols by integrating rigorously curated medical cases and ethical scenarios, particularly those involving privacy protection, patient rights, and health care equity [20-24]. Embed international medical ethics guidelines and local legal frameworks as hard constraints for content generation. Strengthen pharmacological safety [24-26] through expanded datasets covering drug contraindications, interactions [27], and duplicate medication risks. Medical domain-specific models should prioritize these safety-critical considerations [23] during development and deployment.

In medical knowledge question answering, issues such as information omission (1644/3703, 44.4%) and hallucinations (868/3703, 23.4%) reveal gaps in the training corpus, which lacks comprehensive coverage of key medical scenarios. This leads to incomplete or fabricated outputs [28,29], such as omitting critical details or generating nonexistent content, posing risks in high-stakes medical contexts. To address this, models need domain-specific datasets, including medical textbooks and clinical guidelines, to improve reliability. In medical language understanding, omissions (1984/5784, 34.3%) and causal reasoning (1303/5784, 22.5%) deficiencies show models struggle to fully grasp questions, resulting in overinterpretation or insufficient inference [30,31]. For example, they may miss diagnostic criteria or fail to connect symptoms to diseases. Enhancing this requires better training data and architectural improvements, such as integrating causal reasoning modules or external knowledge graphs. In complex medical reasoning, omissions severely hinder performance. Tasks such as diagnosing patients require synthesizing multiple data points, but models often fail to integrate information effectively, leading to fragmented outputs [32]. For instance, they might overlook critical diagnoses or prioritize unlikely conditions. Improving this demands hierarchical reasoning frameworks or hybrid systems combining neural networks with symbolic logic alongside training on annotated case studies and real-world scenarios.

The improvement suggestions for medical LLMs can be structured into 4 levels based on technical complexity and expected impact, progressing from simpler, high-yield enhancements to more advanced, long-term innovations. At the first level, low-cost, high-return optimizations, the focus is on refining “surface-level” elements such as data quality, prompt engineering, and parameter fine-tuning [33,34]. For instance, cleaning noisy medical text, designing specialized prompt templates [34,35] for clinical scenarios, and using techniques such as Low-Rank Adaptation (LoRA) for efficient fine-tuning can significantly enhance baseline performance with minimal model modifications. Moving to the second level, domain-specific adaptation, the goal is to bolster the model’s medical expertise through methods such as knowledge-augmented retrieval [36], multitask joint training [37], and ethical constraint integration. This includes linking to authoritative medical databases [38-40] for real-time evidence retrieval and training on diverse tasks such as diagnostic reasoning and medical record generation [41] to improve overall competency. At the third level, architectural advancements, the focus shifts to upgrading the model’s structure or introducing hybrid systems to tackle complex medical challenges. Examples include combining symbolic logic with neural networks [42] to enhance diagnostic interpretability or designing modular reasoning frameworks [43-45] to minimize error propagation, thereby addressing tasks requiring advanced cognition or extended reasoning chains. Build real-time validation [46,47] pipelines combining rule-based filters and API-driven fact-checking. Finally, at the fourth level, cutting-edge exploration, the emphasis is on long-term technological innovation, such as multimodal pretraining that integrates text [48], imaging, and genomic data, transitioning from correlation-based learning to causal reasoning models [43,44], and leveraging digital twin technology to simulate virtual patient cohorts for training. While these efforts are technically demanding and may yield limited immediate benefits, they lay the groundwork for future breakthroughs in medical AI. This tiered improvement framework, progressing from simple to complex and from high-impact to foundational changes, not only addresses current model limitations but also charts a clear path toward developing reliable and trustworthy medical AI systems.

While MedBench provides a standardized evaluation framework, relying solely on this benchmark may not fully capture the unstructured noise and ambiguity inherent in real-world clinical environments, potentially limiting the generalizability of our findings to diverse health care settings. Furthermore, this study represents a static snapshot of the top 10 performing models, excluding emerging architectures and specialized open-source tools that might exhibit different behavior. The manual development of the error taxonomy also introduces a degree of subjectivity, particularly when distinguishing between overlapping error types, such as causal reasoning deficiencies and hallucinations, in complex medical scenarios.

Additionally, the current evaluation is primarily restricted to textual modalities, leaving the models’ capabilities in processing imaging or genomic data, critical components of modern diagnostics discussed in our future outlook, largely unexplored. Finally, the proposed 4-level optimization strategy remains a theoretical framework derived from our diagnostic insights. While this work identifies systemic weaknesses and prescribes an actionable road map, the empirical implementation and quantitative validation of these specific interventions fall outside the current scope, necessitating future research to verify their efficacy in enhancing clinical robustness and safety.

The current mainstream Chinese medical models face challenges across multiple dimensions, including medical knowledge question answering, language generation, and complex reasoning, while demonstrating room for improvement in safety mechanisms and ethical constraints. Based on a hierarchical improvement framework, we propose a progressive optimization pathway spanning from foundational data optimization and domain-specific knowledge enhancement to architectural innovation. These findings not only provide actionable directions for enhancing the clinical applicability of medical LLMs but also validate the benchmarking value of MedBench in advancing medical AI technologies. Future research should focus on exploring multimodal medical data integration and constructing causal reasoning mechanisms, thereby facilitating the leapfrog development of medical LLMs from knowledge association to clinical decision support systems.