This was a controlled evaluation study with a 4×2 factorial design: 4 LLMs were evaluated as clinical decision support systems, and 2 LLMs served as independent raters. The primary research question concerned the scoring behavior of the AI rater. Clinical LLM performance was a secondary characterization. All 40 vignettes were prespecified before any model was accessed. The scoring rubric was finalized before any response was scored. The statistical analysis pipeline was documented prior to data extraction.

This is an exploratory proof-of-concept study. All findings should be interpreted as preliminary observations and hypothesis-generating results, rather than confirmatory results. The distinction between the “reasoning-quality” and “context-fidelity” scoring paradigms described in Structured Scoring Differences: Criterion Interpretation Analysis section was identified post hoc from the pattern of scoring differences, not preregistered as a hypothesis before scoring began. This taxonomy is proposed here as a framework for future prospective testing.

Forty clinical vignettes were developed across 5 domains: peripheral artery disease (PAD), abdominal aortic aneurysm (AAA), chronic venous insufficiency (CVI), coronary in-stent-restenosis (ISR), and cerebrovascular/neuroendovascular medicine (NEURO; 8 vignettes per domain). The domains were selected to encompass the peripheral vascular, cardiovascular, venous, and neurovascular subspecialties. This allowed for the evaluation of the effects of domain-specific training data on the evaluated model performance and rater scoring behavior.

Within each domain, vignettes were stratified according to clinical demand: vignettes conforming to standard guidelines (n=3 per domain), anatomically or pharmacologically complex cases (n=3), cases requiring explicit contraindication recognition (n=1), and cases requiring conservative or noninvasive management (n=1). Each vignette specified patient demographic information, presenting symptoms, investigational results, and anatomical imaging findings sufficient to guide a specific device selection decision.

A sentinel vignette (V-NEURO-05: fusiform middle cerebral artery aneurysm 3 wk after intracerebral hemorrhage) was selected beforehand as an internal validity anchor. This vignette was chosen because it contained an objectively verifiable contraindication: Dual antiplatelet therapy required by flow-directing devices is contraindicated in a patient at week 3 postintracerebral hemorrhage (ICH), and the correct recognition of this contraindication does not depend on clinical opinion. The correct administration of V-NEURO-05 requires not only the recognition of the contraindication (C3) but also the formal evaluation of an alternative strategy (C6 reflection). This makes it the most demanding vignette in the dataset and a direct stress test for reflective clinical reasoning. Gold-standard consensus expert responses for all 40 vignettes are provided in Multimedia Appendix 1. The complete set of 40 clinical vignettes submitted to the evaluated models is provided in Multimedia Appendix 2.

A 6-criterion scoring system was developed and finalized before any model was evaluated. Each criterion was scored as 0 (none), 1 (partial), or 2 (complete), resulting in a maximum vignette score of 12 and a maximum model score of 480 (40×12). The 6 criteria were as follows:

Four LLMs were evaluated as clinical decision support systems. Each received all 40 vignettes in a single session via the standard chat interface of the respective platform. No system prompting, few-shot examples, or data retrieval augmentation was applied. The models were accessed in standard configuration to reflect real-world clinical use cases.

Two LLMs served as independent raters. Rater 1 was Claude Sonnet 4.6; Rater 2 was GPT-4o. Both raters received the same assessment guidelines, including full criterion definitions, predetermined C6 scoring anchors (Table S1.1 in Multimedia Appendix 3), and the complete vignette text accompanying each model’s response. Raters independently scored each criterion for each vignette-response pair without access to the other rater’s scores. Rater 1 (Claude Sonnet 4.6) was also assessed as M2; rater 2 (GPT-4o) was also assessed as M1. This design feature was intentional and constituted the primary experimental manipulation of the study. It enabled examination of three related questions: (1) whether each rater’s scoring behavior when scoring their own responses differed from their behavior when scoring the responses of other systems (self-scoring patterns); (2) whether high-performing models systematically assigned high scores to one another (mutually high-scoring patterns); and (3) whether the observed scoring behavior reflected criterion-application approaches despite identical assessment guidelines. Complete criterion-level and vignette-level scoring reports for M1, M2, M3, and M4 are provided in Multimedia Appendices 4-7, respectively.

This study did not include human expert scoring. This was a conscious design decision: adding human assessment would have shifted the study question from rater disagreement to rater accuracy. In this context, AI rater scores constitute the data, which are the phenomenon under study, rather than measurements that need to be validated against human reality. This design precludes claims of absolute clinical performance, and all comparisons are made against the implemented AI rater framework. Absolute clinical performance validation requires human expert assessment and is a stated priority for future studies.

Three internal validity analyses were conducted. First, ordinal rank agreement: whether raters agree on the model rank direction despite cardinal score differences, assessed using Spearman rank correlation on total scores. Second, intercriterion correlation: whether scores within each rater logically vary across criteria (eg, C1 and C2 should show positive correlation; high C5 should be seen concurrently with high C1 in the absence of illusory guidelines). Third, sentinel vignette analysis: interrater agreement on V-NEURO-05 scoring across all four models, where widely recognized high-risk contraindications provide a rubric accuracy test independent of clinical opinion.

Interrater reliability was quantified using ICC(2,1) (two-way random effects model, absolute agreement, single measures [18,19], with vignette total scores (0‐12) as the unit of assessment). ICC interpretation followed Koo and Li [20]: <0.50=weak, 0.50‐0.75=moderate, 0.75‐0.90=good, and ≥0.90=excellent agreement. Spearman rank correlation was used to assess ordinal agreement in model rankings. All analyses were performed using SciPy 1.13 and NumPy 1.26 in Python 3.12. The observed low ICC requires careful contextual interpretation. Low agreement between 2 AI raters applying different criterion-application approaches is not necessarily equivalent to random scoring inconsistency. Instead, the disagreement pattern may reflect systematic differences in how the identical scoring criteria are operationalized among evaluators. The complete statistical analysis pipeline is provided in Multimedia Appendix 8.

This study used deidentified and anonymized clinical scenarios constructed by the authors based on published case archetypes. No patient data, personal health information, or identifiable clinical records were accessed or used at any stage. All vignettes were fictitious and prepared solely for the purpose of model evaluation. Under these circumstances, a formal ethics committee review was not required in accordance with institutional and JMIR AI editorial policies. This assessment reflects the authors’ own determination based on the nature of the study; no formal Institutional Review Board (IRB) exemption letter was obtained.

Table 1 presents the scores assigned by each rater to each evaluated model. Rater 1’s scoring ranked the models as follows: M1 GPT-4o (455/480, 95%)>M2 Claude (441/480, 91%)>M3 Gemini (431/480, 89%)>M4 Llama (270/480, 56%). Rater 2’s scoring ranked the models as follows: M2 Claude (446/480, 93%)>M1 GPT-4o (409/480, 85%)>M4 Llama (301/480, 63%)>M3 Gemini (273/480, 57%). Both raters agreed on a tier separation between M1/M2 (both ≥85%) and M3/M4 (both ≤63%). The full 4-model Spearman rank correlation was ρ=0.60 (P=.40), reflecting agreement on tier separation but differing within-tier ordering.

The combined ICC(2,1) value for all 4 models was 0.153 (95% CI 0.007‐0.295; F_159,159=1.39; P=.02; Table 2), which is classified as poor according to Koo and Li [20]. The ICC(2,1) values per model ranged from −0.149 (M4) to +0.048 (M2), all of which are poor. The low reliability between the 2 AI raters whose scoring patterns are most consistent with inconsistent criterion-interpretation approaches is not surprising but an expected result: it shows that the scoring results are sensitive to which rater’s approach is applied. This does not mean that low ICC in general is acceptable; it means that this particular low ICC model should be interpreted in conjunction with the structured disagreement analysis in Structured Scoring Differences: Criterion Interpretation Analysis section, not in isolation.

Negative ICC(2,1) point estimates (M1: −0.123 and M4: −0.149) arise when the estimated between-vignette variance component is smaller than expected under the two-way random-effects model, specifically, when within-vignette rater disagreement exceeds any consistent vignette-level signal. This occurs here because both raters placed M1 and M2 at ceiling (≥85%), leaving minimal variance for the ICC to detect. It does not indicate agreement “worse than chance,” but rather that consistent vignette-level discrimination was undetectable at this performance level.

Analysis of the scoring matrices revealed 2 internally consistent, but mutually inconsistent criterion interpretation approaches applied by the raters. These models are defined in terms of the observed scoring outcomes; they are not intended to characterize internal model states or decision processes. Table 3 presents the criterion-level scores for all 4 evaluated models under both raters. This confirms that scoring divergence between the raters was concentrated on criteria C1, C2, and C5 for models with context-tracking errors (M3 and M4 under rater 2), whereas the magnitude and direction of divergence varied across the remaining criteria.

Rater 1’s scoring was most consistent with the reasoning quality approach: responses received higher scores when they contained clinically advanced reasoning, and partial points (C1=1, C2=1, and C4=1) were given for responses demonstrating advanced clinical knowledge, even when applied to the wrong vignette. Under this approach, M3 Gemini scored 431 out of 480 (89%) despite 15 confirmed vignette-context errors across AAA, CVI, ISR, and NEURO domains (Table S1.5 in Multimedia Appendix 3).

Rater 2’s scoring was most consistent with the context fidelity approach: Responses received near-zero scores (C1=0, C2=0, and C5=0) when they addressed the wrong vignette, regardless of the quality of reasoning they contained. Under this approach, M3 Gemini scored 273 out of 480 (57%), a difference of 158 points from rater 1’s scores for the same responses. This difference is the primary reason for the low pooled ICC.

Both criteria application models offer consistent answers to a real ambiguity in clinical AI assessment: what is the appropriate unit of clinical quality? The reasoning-quality approach assesses transferable clinical expertise and acknowledges partial competence. The context-fidelity approach assesses patient-specific reliability and operational safety. Neither is wrong in principle; they answer different questions about the utility of clinical AI, and researchers using LLM-reviewed methods should consider which question their study is designed to answer.

Contrary to the self-preference hypothesis, no rater’s scoring supported their own responses. Rater 1’s scoring placed GPT-4o (M1) in the highest position at 94.8%, which was higher than the score assigned to its own response (M2, 91.9%). Rater 2’s scoring placed Claude (M2) at 92.9%, higher than their own response (M1, 85.2%). This antiself-preference pattern challenges simple self-preference explanations and instead suggests a high-tier concordance pattern: both raters assigned scores ≥85% to the highest-performing models (M1 and M2) and ≤63% to the lower-performing models (M3 and M4; Table 4).

We note that with only 2 raters, the high-tier concordance is suggestive rather than definitive. For this to be considered a robust fact, a consistent finding across a larger group of AI raters from different model families would be needed. Here, we report it as an observation requiring further investigation.

The only scoring model inconsistent with simple high-tier concordance was the M4 Llama asymmetry: rater 2’s scores placed Llama higher than rater 1’s scores (301/480, 63% vs 270/480, 56%). Criterion-level analysis shows that rater 2’s context fidelity approach heavily penalized Gemini’s vignette context errors while being more tolerant of Llama’s responses in the PAD domain (handling incorrect context but being artificially consistent). Rater 1’s reasoning quality approach heavily penalized Llama’s complete lack of reflection (C6=0/80 across all 40 vignettes) more than context errors. This asymmetry is the clearest evidence that the 2 criterion application models penalize different error modes: the context fidelity approach penalizes context loss while tolerating lack of reflection; the reasoning quality approach penalizes lack of reflection but tolerates partial context errors.

C6 (PARM reflection) showed a notable but evaluator-dependent scoring pattern (Table 5). Under rater 1, C6 was the criterion that received the lowest score across all 4 models. Under rater 2, however, C6 scores varied substantially across the models, including marked divergence for M2 (C6=75/80 for rater 2, whereas C6=41/80 for rater 1). Several criterion-level observations were nevertheless noteworthy: (1) rater 1 assigned a C6 score of 0 to M4 across all 40 vignettes; (2) V-NEURO-05 was the only vignette to receive a C6=2 from both raters; and (3) C6 scores were similar between raters for M1 and M3.

These findings do not establish an evaluator-independent reflection deficit. Rather, they show that C6 scoring is particularly influenced by evaluator-specific criterion application, while selected observations, including similar M1 and M3 C6 totals and the V-NEURO-05 sentinel finding, were more consistent across raters. PARM reflection should therefore be considered a criterion requiring further investigation.

V-NEURO-05 (12 mm fusiform middle cerebral artery aneurysm, phases score 8, 3 wk after intracerebral hemorrhage) was the only vignette in which both raters independently assigned C3=0 to the same model, M4 (Llama 3.3 70B), on an objectively verifiable contraindication. This is the only criterion score of 0 on an objectively verifiable item in the entire dataset, as the model’s response failed to identify the need for dual antiplatelet therapy as a contraindication in a patient 3 weeks post-ICH. This scenario represented a high-risk contraindication with relatively limited scoring ambiguity.

M2’s response was the only one in the full dataset that reached C6=2 by both raters: it identified the dual antiplatelet therapy contraindication, formally presented 2 alternative strategies (surgical extracranial-intracranial bypass with aneurysm trapping; delayed flow redirection with phased antiplatelet readministration after 3‐6 mo), specified an antiplatelet scheduling protocol, required multidisciplinary team documentation, and calculated the phases 8 rupture risk as the basis for strategy decision.

The distinctive value of V-NEURO-05 lies in what it requires. The standard information-based response (flow redirection for fusiform morphology) is factually correct, but it becomes clinically inappropriate for this patient. Correct management requires acknowledgment that the standard response is contraindicated, followed by formal evaluation of alternatives—precisely the reflection dimension of the PARM framework. A model that correctly responds to the general situation but fails to recognize the patient-specific contraindication exhibits a clinically concerning capability profile: a high level of knowledge, low contextual safety judgment.

Figure 2 illustrates the model and vignette level distribution of signed interrater score differences, revealing significant heterogeneity in both the magnitude and direction of the evaluator disagreement.

C6 (PARM reflection) showed a scoring pattern that was dependent on the rater but distinct. Under rater 1, C6 was the lowest-scoring criterion for each of the 4 models evaluated, whereas rater 2 showed greater evaluator-dependent variation, particularly for M2. Nevertheless, cross-rater C6 scores were similar for M1 and M3, while sentinel-vignette analysis identified a specific case where reflective reasoning was clinically consequential. These findings support further investigation into spontaneous reflective output but do not establish an evaluator-independent reflection deficit.

The 0.153 ICC reported in this study should not be read as a simple methodological failure. However, it is indeed low and requires explanation rather than being ignored. The explanation offered here is that 2 AI raters applying inconsistent criterion interpretation approaches to the same evaluation criterion will, by definition, produce a low agreement when scoring the responses that trigger these differences. The more important observation is that the disagreement is not random. It is largest in models with confirmed vignette context errors (M3 Gemini: Δ=−158; M4 Llama: Δ=+31), though M1 also shows a substantial divergence (Δ=−46) attributable primarily to rater-specific scoring thresholds across C1-C5 rather than context-tracking failures, a distinction discussed below.

The M1 divergence (Δ=−46) warrants a separate explanation. Although GPT-4o (M1) had no vignette context errors, rater 1 scored it 46 points higher than rater 2. Criterion-level analysis reveals that this gap is distributed across C1-5. Rater 1 assigned GPT-4o a mean C6 of 75% (60/80), while rater 2 assigned 75% (60/80), near-identical C6 totals. The M1 gap is therefore driven primarily by threshold differences across C1-C5 between the raters on high-quality responses. This reflects the same criterion interpretation divergence documented for other models but without a context-tracking-failure component. This further supports the interpretation that the 2 raters’ scoring approaches represent distinct criterion application philosophies, rather than simply quality detection.

Structured, directional disagreement can be interpreted in a way that random disagreement cannot be interpreted. This situation is consistent with the known instability of LLM under different evaluation framings [21,22]. In these studies, the scoring outcomes systematically vary according to question structure and option order. The important point is that a low ICC value remains evidence of weak absolute agreement among evaluators, even when the disagreement pattern itself is informative. We do not argue that low reliability strengthens evaluator validity. Rather, we argue that the observed disagreement pattern provides insight into the evaluator-dependent criterion application.

Agreement on broad tier separation (both raters’ scores place M1/M2 above M3/M4), together with large numerical differences (158 points on the Gemini scale), reveal that the 2 scoring approaches share a rough model hierarchy but apply different criteria to translate this hierarchy into numbers. The Spearman correlation (ρ=0.60; P=.40) is reported here as a descriptive summary of the ordinal agreement across the 4 models. The small sample precludes inferential interpretation at the model ranking level. Therefore, the choice of an AI rater determines not only the magnitude but also the basis of the benchmark results, which is a key finding of the study.

The 2 scoring models identified (reasoning quality and context fidelity) are described as behavioral characterizations of the observed scoring outcomes, not as statements about the intrinsic properties of the model. Throughout this discussion, we deliberately and consistently use phrases such as “most consistent” and “the scoring was most consistent.” Alternative explanations for the observed models cannot be ruled out, and future studies using controlled stimulus designs may reveal them more definitively.

An antiself-preference pattern emerged; both raters assigned higher scores to the other high-performing model than to their own responses. First, both AI raters may apply more tolerant scoring to recognized high-quality clinical reasoning, regardless of its source (epistemic generosity). Second, GPT-4o and Claude share large portions of training datasets (medical literature and clinical guidelines), which can be expressed as overlapping training data. This can produce convergent reasoning patterns that each rater’s scoring recognizes and rewards.

The observation of high-tier concordance, whereby both raters consistently assigned scores of 409 out of 480 (85%) or higher to the highest-performing models, may have practical implications for reinforcement learning from AI feedback (RLAIF) pipelines in which the same model family functions as both the evaluator and target system. If high-performing raters systematically favor similarly high-performing models, preference learning processes may generate inflated performance estimates and amplify apparent capability differences that do not fully correspond to meaningful differences in clinical quality. This hypothesis warrants evaluation in future multirater studies.

C6 (PARM reflection) demonstrated an evaluator-dependent rather than uniformly convergent scoring pattern. Under rater 1, C6 was the lowest-scoring criterion for each of the 4 models evaluated. Under rater 2, this pattern was not uniformly replicated, with the clearest divergence observed for M2. Nevertheless, cross-rater scores for C6 were similar for M1 and M3, and the sentinel-vignette analysis identified a specific instance where reflective reasoning was clinically consequential.

A clinical decision support system that offers sound advice but fails to address uncertainty and offers no alternatives where the standard approach might be problematic. It ignores limitations of evidence and can be dangerous even when it seems most confident [4]. The PARM Reflection gap is not a minor assessment deficit. It operationalizes the gap between high clinical knowledge and safe clinical reasoning under uncertainty. This finding is consistent with the PARM framework’s emphasis on reflective clinical reasoning as a distinct component of the safe clinical decision-making process [23]: the reflection dimension is what separates clinical intelligence from active clinical reasoning. The clinical relevance of reflection remains plausible. However, the evaluator-dependent C6 scoring observed here precludes interpreting the pattern as a stable characteristic of the evaluated models. Independent replication with additional AI raters and validation by human experts is required.

This study contains 4 specific implications for clinical AI researchers using LLM-based evaluation approaches. First, single-evaluator benchmark tests can produce results dependent on criterion interpretation. A study using GPT-4o as a single reviewer will produce the most consistent results with a context fidelity approach; a study using Claude will produce the most consistent results with a reasoning quality approach. Without knowing which approach is valid, benchmark scores cannot be compared between studies using different AI reviewers.

Second, the identity of the AI reviewer should be disclosed, justified, and recorded in advance along with the evaluation criteria. This is similar to the requirement to disclose the identity and training of human reviewers in traditional reliability studies. This is not currently standard practice in clinical AI benchmarking.

Third, the high-tier agreement model, if replicated, suggests that using the same model as both subject and judge, as in RLAIF pipeline [24], can generate systematically enhanced preference predictions for highly skilled models. This is a particular prediction that RLAIF methodology studies should directly examine. More broadly, undetected AI evaluator divergence risks permeating clinical AI training processes [25]. This makes the characterization of systematic evaluator divergence not only a methodological issue but also a concern from a patient safety perspective.

Fourth, the C6 findings illustrate the importance of reporting criterion-level agreement rather than relying solely on aggregate benchmark scores. Criteria showing substantial evaluator-dependent variation should be distinguished from observations that replicate across evaluators, and multievaluator studies should report these patterns separately.

A note on the C6 finding as a rubric-conditional: the models were not explicitly prompted to acknowledge uncertainty, propose alternatives, or state limitations of the evidence. The observed C6 scoring pattern therefore reflects spontaneous reflective output in the absence of an explicit reflection instruction. Whether an explicit system prompt would elicit C6=2-level reflection is a testable question deferred to future work (Future Directions section). The C6 finding should be interpreted as a characterization of default model behavior under standard prompting conditions, rather than a statement regarding the maximum achievable reflection performance.

This study has 6 limitations. First, human expert scoring was not performed, which prevents absolute clinical performance claims. All comparisons were made according to the AI rater scoring approach applied. Second, only 2 AI raters were used; the findings regarding self-scoring models and high-tier concordance are suggestive rather than conclusive and need to be replicated with additional raters from different model families. Third, the vignettes were not validated by an independent clinical panel prior to use. Fourth, model access was provided via standard chat interfaces without fixed temperature or seed parameters, leading to potential response variability. Fifth, the extracted criterion-interpretation patterns, including reasoning quality and context fidelity, are not direct measures of model processing but rather behavioral characterizations derived from scoring outcome patterns. Sixth, the scope of this study’s claims is intentionally narrow. With 4 evaluated models and 2 AI raters, no findings can be generalized to LLM evaluator behavior in general. All results are observations from one study configuration and should be interpreted as preliminary evidence warranting prospective investigation. The reasoning-quality and context-fidelity paradigm taxonomy proposed in Structured Scoring Differences: Criterion Interpretation Analysis section is a post hoc hypothesis generated from this data, not a preregistered theoretical framework. Future studies should test whether this taxonomy replicates across different model families, clinical domains, and scoring rubrics. In addition, each model response was scored once by each rater; intrarater reliability was not measured. Under stochastic decoding conditions, a single scoring pass cannot fully separate stable criterion interpretation differences from intrarater response variability. Quantifying intrarater reliability through independent rescoring sessions is a priority for future work and would strengthen the claim that the 2 observed scoring patterns are internally consistent. The PARM Reflection findings should be interpreted as preliminary observations requiring independent validation rather than confirmatory evidence for the framework. Additionally, rater 1 (Claude Sonnet 4.6) was also evaluated as M2, and rater 2 (GPT-4o) was also evaluated as M1. This design feature is intentionally disclosed. The structural overlap between rater identity and evaluated model identity is a study limitation that cannot be resolved by disclosure alone and requires replication with independent evaluators.

Four research directions emerge from this study. First, scoring pattern characterization should be repeated with additional AI raters to determine whether the distinction between reasoning quality and context fidelity is stable across model families or specific to this pair of raters. Second, PARM Reflection performance should be investigated under explicit prompting conditions to determine whether the observed pattern reflects a capability limitation or a default behavior that can be modified through prompt engineering. Third, the high-tier concordance pattern should be investigated in RLAIF pipelines. Fourth, a “sentinel vignette methodology,” consisting of a small set of short histories that are widely recognized, in high demand, and allow for interrater validity testing without the need for full human assessment, should be developed as a standard component of clinical LLM assessment.

In this exploratory study, 2 AI raters applied to LLM-based clinical benchmarking did not behave as interchangeable scoring instruments. Their scoring is most consistent with model-specific criterion interpretation approaches that systematically produce different results. The low interrater reliability (ICC=0.153) observed in this study reflects a structured, directional disagreement concentrated in models with vignette context errors, not random inconsistency. This requires contextual interpretation. It should not be dismissed as a methodological failure but interpreted as evidence of structured, model-dependent evaluator disagreement.

The antiself-preference pattern and high-tier concordance in the scoring of both raters call into question simplistic accounts of AI rater scoring behavior and suggest that the relationship between the AI rater and the model being evaluated is more complex than characterized by the current LLM-peer-reviewed literature.

Under rater 1, C6 was the lowest-scoring criterion for each of four evaluated models, whereas this pattern was not uniformly replicated under rater 2, where C6 showed substantial model-dependent variation. These findings therefore identify PARM Reflection as a criterion warranting further investigation rather than an evaluator-independent deficit. The V-NEURO-05 case study illustrates the potential clinical relevance of reflective reasoning, with only 1 of the 4 evaluated models receiving a perfect C6 score from both raters. More broadly, the identity of the AI rater and its criterion-application approach are not methodological footnotes when LLMs are used as judges. They can materially influence benchmarking results and should therefore be explicitly reported, prespecified where feasible, and critically examined.