We used an existing dataset that was used in a previous paper from our group consisting of 250 RCTs drawn from 7 major medical journals (British Medical Journal, JAMA, JAMA Oncology, Journal of Clinical Oncology, The Lancet, The Lancet Oncology, and New England Journal of Medicine) published between 2005 and 2023 [16]. In the aforementioned publications, eligible trials were restricted to designs with exactly 2 arms and a single primary end point, and abstracts were retrieved via PubMed and parsed from text to create the study corpus. For the present analysis, we reused the released dataset and corresponding ground-truth labels.

Ground-truth labels were adopted from the original dual-annotation procedure performed by 2 authors (PW and CK), in which trials were classified as positive if the primary end point was met and as negative otherwise. The annotation workflow included an initial calibration phase followed by independent labeling and consensus resolution of discrepancies. Full texts or protocols were consulted only when the abstract did not clearly report the primary end point and its results. We did not modify labels or repeat manual annotation for the present analysis.

To distinguish evidence use from identifier-driven recognition, we generated 5 input variants for each RCT. The baseline condition contained the trial title and abstract (title+abstract). A title-only condition contained only the trial title. A DOI-only condition contained only the DOI string. A counterfactual “fake results” condition contained the potentially edited title and an edited abstract in which the reported primary end point outcome was flipped (positive to negative or negative to positive) while keeping all other content maximally unchanged. Counterfactual abstracts were created by identifying the sentences reporting primary end point attainment (typically in the Results and/or Conclusions sections) and minimally editing the outcome language (eg, reversing whether the primary end point was met or reversing the direction of statistical significance statements tied to the primary end point). Background, design, population, interventions, and secondary outcome text were left unchanged unless modification was required to maintain internal coherence. The original and counterfactual texts are provided in the GitHub repository that is referenced in the Data Availability section. Finally, we input the counterfactual title and abstract together with the original DOI to create a direct conflict between the supplied counterfactual evidence and identifier-conditioned prior information.

The task for the LLMs was to determine, from the provided information, whether the trial met its primary end point. While this is essentially impossible from a DOI with the exception of a slight bias of high-impact journals to publish positive trials, titles sometimes report results and usually those related to the primary end point. Abstracts usually but not always state which end point was the primary one and explicitly provide the results per end point. To align with the previously published workflow and minimize ambiguity, we used an explicit instruction format requiring the model to output exactly 1 token-level label: “POSITIVE” or “NEGATIVE,” all capitalized.

The following system prompt was used: “You will be provided with information about a randomized controlled oncology trial. Your task will be to classify if the trial was positive, i.e. if it met its primary endpoint, or negative, i.e. if it did not meet its primary endpoint. Your response should be either the word POSITIVE (in all caps) or NEGATIVE (in all caps). Do not output anything else.” The user prompt consisted of the respective input variant (title+abstract, title only, DOI only, counterfactual title+abstract, and DOI and counterfactual title+abstract).

We did not use few-shot examples, retrieval augmentation, or chain-of-thought prompting as the aim was to evaluate default single-label behavior rather than prompt-engineered mitigation strategies.

Three commercial LLMs were evaluated via their vendor APIs in a local pipeline (Claude [Anthropic], Google Gemini, and GPT-5.2 [OpenAI]). The evaluated model IDs were gpt-5.2-2025-12-11, gemini-3-flash-preview, and claude-opus-4-5-20251101. Models were queried under default vendor settings to reflect typical end user deployment conditions. No additional decoding parameters were set, and no seed was specified. No vendor-specific reasoning control was requested.

Responses were considered valid only if the returned text matched exactly the “POSITIVE” or “NEGATIVE” label after trimming white space. Any other output was recorded as invalid for downstream summaries.

The primary objective was to quantify how model behavior changed as informational content was removed and when counterfactual results created a direct conflict between the provided abstract and the original trial outcome potentially associated with the identifier. Performance for the baseline, title-only, and DOI-only conditions was summarized using F₁-score (and accuracy) against the original ground-truth label. For the counterfactual condition, the expected label was defined as the inverse of the original ground truth, and performance was evaluated against this counterfactual target. In addition to per-condition performance, we quantified counterfactual sensitivity by calculating the proportion of trials for which the predicted label differed between the baseline and counterfactual inputs (flip rate), and we summarized invalid-format outputs per model and condition. For the calculation of the F₁-score, only valid (ie, correctly formatted predictions) were considered. The 95% CIs were estimated using normal approximation intervals. We did not analyze token-level probabilities or calibration because comparable logits were not consistently available across the evaluated vendor APIs.

This study used publicly available abstracts from published clinical trials. It did not involve human beings, material of human origin, health-related personal data, deceased persons, or human embryos. Therefore, per the applicable regulation (ie, the Swiss Human Research Act and its ordinances), ethics approval was not required [17].

All 3 commercial models achieved high performance when given full title and abstract information (accuracy and F₁-score clustered around 0.96-0.97), with only minor rates of invalid-format outputs. Performance decreased stepwise as informational content was removed: with title-only inputs, accuracy dropped to a range of 0.79 to 0.88, and with DOI-only inputs, it fell further to a range of 0.63 to 0.67. In contrast, when the abstract’s primary outcome statement was counterfactually flipped, the models followed the altered evidence and achieved near-ceiling performance against the inverted labels (accuracy and F₁-score=0.96-0.99). Reintroducing the real DOI alongside the counterfactual abstract minimally affected GPT-5.2 but led to modest performance reductions for Gemini and Claude, primarily via reduced sensitivity, suggesting occasional identifier-driven interference when DOI and results text conflicted.

Interpreting these results in the context of prior work, the strong baseline performance is consistent with broader evidence that LLMs can perform competitively on biomedical language tasks [1,3]. However, high task performance alone does not establish that the prediction is grounded in the provided document rather than influenced by parametric knowledge or learned associations, an issue long discussed in the framing of language models as implicit knowledge bases [4]. The stepwise ablation results speak directly to that distinction. Title-only performance well above chance can plausibly arise from a mixture of (1) genuine, weakly informative cues in titles (eg, disease setting, intervention class, and end point hints); and (2) recognition of well-known trials from pretraining exposure. DOI-only inputs are informative because the DOI string contains minimal semantic content for end point attainment. Above-chance performance under DOI-only conditions is therefore compatible with identifier-triggered recall, learned metadata correlations, or other DOI-associated regularities.

However, the counterfactual results indicate that the models’ predictions were usually dominated by the supplied counterfactual outcome evidence rather than by any identifier-associated prior signal. When the outcome-bearing sentences in the abstract were minimally edited to flip the primary end point conclusion, the models overwhelmingly produced the inverted label, indicating sensitivity to the supplied text even when that text contradicted the original outcome potentially associated with the trial identifier. At the same time, the small but consistent decrement for some models when the real DOI was appended to the counterfactual abstract suggests that identifiers can still exert a measurable pull on the model’s decision, echoing findings that models do not integrate context and prior knowledge uniformly and can privilege priors when they are “familiar” with an entity [9]. It is important to emphasize that our counterfactual flips created direct, high-salience evidence conflicts. In more realistic settings, users may supply incomplete or subtly contradictory evidence, where LLMs can struggle with conflict resolution in biomedical contexts [12]. Taken together, the findings support a nuanced conclusion: the models can robustly follow explicit outcome statements in abstracts (strong context-grounded behavior under explicit outcome evidence) yet still display some degree of susceptibility to identifier-driven priors when content is sparse or when identifiers are introduced.

Several strengths support the interpretability and practical relevance of these findings. First, the design operationalizes the contrast between context-grounded evidence use and identifier-conditioned prior signal using deterministic, logged perturbations (progressive content removal plus counterfactual outcome edits), which makes the diagnostic logic transparent and reproducible. Second, the evaluation used a corpus of real oncology RCT abstracts spanning multiple high-impact journals and many publication years, aligning the test distribution with how LLMs are used in evidence synthesis and clinical research workflows rather than relying solely on synthetic benchmarks. Third, the single-token output constraint (“POSITIVE” and “NEGATIVE”) reduced ambiguity in scoring and minimized the confounding role of verbose explanations, whereas the high valid output rates indicate that the API-based setup was stable across conditions. Fourth, comparing multiple commercial models under default settings improved external validity for typical end user deployment, where practitioners rarely tune decoding parameters or implement specialized grounding interventions.

Several limitations should temper overgeneralization. First, this study focused on a single binary task (primary end point met vs not met) within oncology RCTs. The balance between context-grounded evidence use and identifier-conditioned prior signal may differ for tasks requiring finer-grained extraction, multi-label judgments, or synthesis across multiple documents. Second, DOI-only above-chance performance could partly reflect indirect correlations encoded in DOI structure (publisher prefix, journal family, and year) rather than memorized trial-specific outcome associations. Additional controls (eg, DOI shuffling across papers and synthetic DOIs matched on prefix and year) would help isolate the mechanism. Third, counterfactual editing was designed to be minimal, but any manual or rule-based editing procedure risks introducing artifacts (lexical cues, unnatural phrasing, or consistency breaks) that models could exploit. Even though edits were concentrated where outcomes were stated, future work could quantify the detectability of edits or use blinded human review to ensure counterfactual naturalness. Fourth, the models were queried via vendor APIs with default settings and without a fixed seed. While this reflects realistic use, it limits ANOVA due to decoding stochasticity and complicates strict reproducibility across future model snapshots. Finally, we did not evaluate model calibration, abstention behavior, or uncertainty reporting—properties that may be crucial when deploying such systems in safety-critical evidence workflows.

Future research could try to strengthen causal attribution of identifier effects by adding negative controls: swapping DOIs between trials, adding semantically irrelevant identifiers, or using “matched” synthetic DOIs, which would help separate true memorized mapping from metadata correlations. A second direction is to broaden task coverage: applying the same progressive ablation and counterfactual conflict framework to other biomedical natural language processing tasks (population, intervention, comparator, and outcome extraction; effect direction and magnitude; toxicity end points; and comparative effectiveness statements) would test whether strong counterfactual sensitivity generalizes beyond this relatively explicit classification problem. A third direction is temporal and contamination-robust evaluation: constructing time-split test sets consisting of trials published after known model training cutoffs (or using newly published or embargoed material where feasible) would reduce the plausibility of trial-specific parametric associations and better isolate context-grounded task performance.

In conclusion, this study suggests that the evaluated models were highly sensitive to explicit outcome statements in oncology RCT abstracts, as shown by near-ceiling performance on counterfactual abstracts that directly contradicted the original results. At the same time, above-chance performance with titles—and especially with DOI-only inputs—indicates that identifiers can carry predictive signal for these models, consistent with identifier-associated predictive signal that is at least partly independent of the provided abstract text. The modest degradation in performance observed when reintroducing real DOIs into counterfactual abstracts further implies that identifier-triggered priors can occasionally compete with textual evidence.