Institutional Review Board submission and approval were not required for this study because it contains no human subjects research.

We extracted 106 case vignettes written by 133 discussants across 19 primary DSM-5-TR (Diagnostic and Statistical Manual, Version 5, Text Revision) diagnostic categories from the DSM-5-TR Clinical Cases book. Each vignette is a multi-paragraph description of a particular case; the vignettes are written for educational purposes; the author’s intended final diagnoses are also provided separately from the vignette. The standardized task was to generate diagnoses from the case vignette, with the author-designated diagnoses used as the gold-standard comparison.

For this study, we selected one LLM and one LRM from two major vendors from whom an enterprise-grade LLM application programming interface (API) was available, and which had agreed terms of use that were (1) compatible with the intellectual property rights of the study researchers and rightsholders associated with the study data, and (2) prohibited the use of study data for model training by vendors. The vendors and models used for this study were: OpenAI (LLM: gpt-4.1-2025-04-14, LRM: o3-pro-2025-06-10), and Google (LLM: Gemini 2.5 Flash, LRM: Gemini 2.5 Pro). The LLMs were used at their default settings, with an output token limit of 2000. LRMs were provided with 16,000 reasoning tokens, and OpenAI models were also set to medium reasoning effort (Google models do not provide this parameter). Reasoning was disabled for both LLMs. Safety settings were disabled or set to the minimum level of filtering. Models were constrained to respond in a specified machine-parsable output format (JavaScript Object Notation, or JSON) either through the use of application programming interface (API) flags or by reprompting until the output was in valid JSON. All prompts and reprompts were independent without inclusion of prior input, output, or logs.

We compared two inference approaches previously developed by the authors [4] using each of the available models. In the first approach (the “Basic” approach), the model was directly prompted to assign diagnoses to the vignette using a standardized prompt (Textbox 1). In the second approach (the “SV” approach), diagnoses generated by the Basic approach were evaluated using a sequential pairwise elimination procedure. Each pair of candidate diagnoses was presented to the model using a standardized prompt asking whether both diagnoses were necessary or whether one was better explained by the other (Textbox 1); each of these prompts was done independently without including context from any other prompt. If both were retained, the procedure continued; if only one was selected, the other was removed from further evaluation. Because this procedure requires at least two candidates, vignettes for which the Basic approach produced a single diagnosis were not modified by SV, and if a candidate diagnosis list was reduced to only one candidate, the procedure was terminated. This approach serves as a templated unidirectional filter that can only narrow the list of diagnoses.

We successfully processed 106 psychiatric case vignettes using the selected approaches and found that both simulated reasoning and self-verification yielded statistically significant improvements in the PPV, compared to the Basic approach alone, without significant differences in the sensitivity. The best overall performance of models was found in the o3-pro LRM with the additional SV prompt, yielding a sensitivity of 0.782 and a PPV of 0.779. The sensitivity ICC of 0.77 suggests that much of the variance found in our experiments was explained by the intrinsic difficulty of the individual vignettes, a finding also clear in the per-vignette results depicted in Figure 1. In contrast, the PPV ICC of 0.49 indicates that the approach and the underlying vignette differences contributed approximately equal variance, which is consistent with the effects we found in PPV and the underlying mechanics of the SV method (since, for example, the SV prompt acts as a unidirectional filter, only removing diagnoses). The substantial case-level variability is consistent with past findings, such as the variation in inter-rater reliability across categories found in the DSM-5 field trials [10,11].

Interestingly, we found no statistically significant interaction effect between SR and SV, suggesting that the use of both yielded no additional benefit over any one alone. This finding may, however, be due to a lack of sufficient power from our study to detect these differences and future evaluation is required. Thus, the addition of explicit reasoning suggestions in prompts (such as that in the SV prompt) could continue to be helpful even when using automated simulating reasoning systems. Similarly, we found no significant impact of the use of either approach on sensitivity; this is unsurprising for SV given that this approach can only remove diagnoses, but was surprising for SR which theoretically could use reasoning tokens to expand diagnostic range.

First, the semiautomated simplification and matching process we used was intended to give models the best opportunity to be scored as correct, and this may have the impact of overestimating the performance of the models. Second, the vignettes and diagnoses used for our experiment may be in the training set of these models (though they have not been licensed for such use); our test results are reassuring against memorization and the vignettes are explicitly not licensed for this use, but it is still possible that training exposure may have biased our evaluation towards better performance, constituting a contamination risk. Even in this case, however, we believe that the relative findings between methods in our study would be valuable. Third, since the vignettes were written for teaching, the prevalence of diagnoses is higher in the dataset than in the general population and more comprehensive information is provided in the vignettes than is generally available in the clinical setting, limiting the ecological validity of our sample. This may serve to inflate the performance metrics that we found and our results are not intended to suggest clinical utility at this stage. Additionally, the amplified illness prevalence in a vignette-based educational dataset enables methodological exploration, but findings may not generalize to low-prevalence real-world populations (such as self-diagnosing internet users). Fourth, not all diagnostic categories are equally or proportionally covered by our dataset, and our study was not powered to detect inter-category differences. Additionally, because we specifically collapsed neurocognitive disorders into two diagnoses (Multimedia Appendix 1), performance on these cases may be artificially inflated by masking potential challenges in differentiating between specific neurocognitive pathologies. Fifth, we have focused in our study on evaluating whether there is an improvement in the outcomes of the pipeline for analysis, but we did not directly evaluate the accuracy of the reasoning output generated by the LRMs, or the individual-level decisions made after SV prompts. Further, the actual reasoning output generated may not be causally linked to the final outcome [14].

Future efforts could make use of larger datasets, potentially including enhanced coverage of each diagnostic category and making use of either novel vignettes or real-world clinical data, considering full differential diagnosis, and comparison to human performance. This would enhance the power of future studies and minimize the risk of training bias. Additionally, future studies could examine the impact of model selection and prompt optimization to better elucidate how best to configure language models for psychiatric use, including the potential blending of SR and SV-like approaches, such as the use of pretemplated prompts that call for generating additional reasoning prompts. The addition of probabilistic confidence estimates would also be helpful, as real-world diagnostic processes require the generation of differential diagnoses; though estimates obtained by prompting are not reliable [3], recent efforts have explored the use of confidence metrics like LLM perplexity for psychiatric reasoning [24]. Finally, further studies could directly assess the accuracy and efficacy of the actual reasoning tokens generated by LRMs during inferencing, and whether there may be methods to improve the generated tokens.

In this study, we found that both simulated reasoning and self-verification improved the positive predictive value of LLM-based psychiatric diagnosis from case vignettes, without significantly affecting sensitivity. The impact of these two approaches suggests that manually crafted verification prompts can continue to provide value even when automated reasoning capabilities are available. Future work should validate these findings using real-world clinical data in order to inform clinical system design.