In this section, we review related work on user simulation, patient simulation, and behavioral and linguistic modeling for conversational AI. User simulation provides scalable methods for evaluating conversational agents across domains, and patient simulation adapts these methods for clinical dialogue and diagnostic reasoning.

User simulation enables controlled, scalable evaluation of interactive systems without requiring human trials. Foundational surveys span information access, dialogue modeling, and recommendation systems [14,25-27], while recent work leverages LLMs to generate context-aware user behavior through dual-model architectures combining generators and verifiers [28,29] and session-level search simulation [30,31]. In health care, synthetic users with clinical profiles evaluate decision support and coaching systems [32], establishing user simulation as a critical method for assessing reliability and safety in AI systems. Mental health conversational AI research has historically split between technical evaluation of response quality and clinical evaluation of patient outcomes, with limited integration across the two [33].

Patient simulation involves the construction of virtual patients that engage in interactive clinical dialogues with human or automated clinicians [34]. Frameworks vary widely in patient state representation, from hand-crafted profiles to EHR-grounded models, and in how they control disclosure, tone, and clinical accuracy [1,35,36]. Prior work has often treated patient simulation as a single unified problem, but it involves two distinct technical challenges: (1) constructing clinically valid patient representations and (2) simulating interactive dialogue that expresses those representations over time [36]. Recent systems have integrated LLMs to enhance expressiveness and flexibility [37], constructed structured knowledge bases from clinical notes to support medical intake tasks [38], and embedded risk-aware feedback mechanisms [34].

Early systems used deterministic state machines or probabilistic sequence models to generate medical histories. Synthea [39] and SynSys [40] produce scalable, transparent timelines but reflect population-level distributions rather than individual patient data. EHR-grounded approaches improve realism. Generative Adversarial Network (GAN) variants [41,42] address missing features and mixed data types, achieving higher fidelity than rule-based methods on MIMIC-III/IV benchmarks [43], while SimSUM [44] combines Bayesian networks with prompted LLMs to synthesize clinical notes. Systems integrating EHR grounding with conversational retrieval, including ophthalmology simulators [38] and multiagent knowledge graph pipelines [45], advance clinical accuracy but treat the encounter as a transactional data exchange rather than a dynamic, rapport-dependent interaction.

Beyond medical profile construction, systems must express profiles through natural dialogue with appropriate communication style and conversational dynamics [35,46]. Cognitive and persona-based approaches model psychological states to generate behaviorally coherent conversation. PATIENT Ψ [47] uses expert-designed cognitive schemas with GPT-4 to reproduce emotional fluctuations and resistant behaviors, while SFMSS [48] embeds Big Five traits to shape dialogue tone and coordinates patient, nurse, and supervisor agents to enforce outpatient triage workflows. PAL [49] simulates emotionally nuanced palliative care patient interactions with NURSE-framework feedback, illustrating affect-grounded simulation in another condition-specific clinical context. These systems remain condition-specific (eg, mental health and palliative care) and lack medical grounding for clinical evaluation. Prompt-based approaches [34,37] offer scalability but lack persistent patient state tracking and systematic control over linguistic and behavioral variation.

A third class prioritizes procedural fidelity and clinical workflows through discrete agent roles and actions. iPDG [50] enforces clinical plausibility through manually defined domain rules. Such systems achieve procedural coherence but sacrifice conversational flexibility and behavioral depth.

Recent systematic reviews of mental health chatbots found that LLM-based systems concentrate in early stage technical validation, with most evaluations focused on conversational quality or adherence to specific prompts in controlled settings rather than rigorous testing for clinical benefit [51]. Existing patient simulation systems exhibit 2 critical gaps that constrain trustworthy AI evaluation. First, no existing framework integrates medical realism, behavioral variation, and linguistic diversity within a unified architecture. Recent systems pair medical grounding with behavioral modeling or behavioral depth with linguistic variation, but this fragmentation creates evaluation blind spots. Existing systems cannot systematically test whether agents maintain diagnostic accuracy across complex comorbidities, varied health literacy, and adversarial behavior simultaneously. Multiturn failures like context loss and inconsistent safety emerge precisely at these intersections.

Second, existing approaches prioritize medical outcomes over systematic risk management aligned with frameworks like the NIST AI RMF. They lack explicit risk mapping connecting simulation parameters to risk categories, auditability with traceable lineage to source data, and controllable risk probing through systematic variation. This prevents the detection of consequential failures: agents may recommend different treatments when patients express identical conditions at different health literacy levels, or safety mechanisms may fail under adversarial conditions that only systematic variation can expose. We address both gaps through a unified framework whose medical, linguistic, and behavioral profiles align with the NIST AI RMF Map and Measure functions, enabling risk assessment across medical accuracy, communication appropriateness, and behavioral robustness.

Both the patient simulator medical profiles and the AI decision aid prediction models were derived from the All of Us Research Program Registered Tier v8 dataset, a national longitudinal study providing structured EHR data, including conditions, medications, procedures, and demographics [17].

We extracted data from All of Us participants with major depressive disorder (Systematized Nomenclature of Medicine Clinical Terms [SNOMED CT] code 370143000 and its descendants). The resulting cohort included 58,446 participants, contributing 466,752 antidepressant trials, with 18,471 diagnoses, 2642 medications, and 5001 procedures. Outcomes captured responses to 14 antidepressants and 1 category covering all remaining antidepressants (dataset cutoff: October 1, 2023). Antidepressant response was defined as taking the antidepressant for at least 10 weeks without switching or augmenting with another antidepressant [52].

For patient simulator medical profiles, the dataset provided feature distributions, RR calculations for outcome-relevant predictor selection, and demographic distributions for age and gender initialization. The simulator was evaluated on the 4 most common antidepressants: fluoxetine, sertraline, trazodone, and duloxetine [53]. For the AI decision aid, the same data trained prediction models generating treatment recommendations across all 14 antidepressants. All data use complied with All of Us dissemination policies.

The patient simulator design was grounded in the NIST AI RMF, which defines four core functions: (1) Govern (accountability and oversight), (2) Map (risk identification), (3) Measure (risk assessment), and (4) Manage (risk mitigation) [16]. We aligned with Map and Measure across 2 complementary phases. Simulator construction supported Map by structuring risk identification through 3 profile dimensions, each addressing a distinct category of risk: medical profiles targeting clinical accuracy, linguistic profiles targeting communication-dependent risks, and behavioral profiles targeting interaction-dependent risks. Simulator-AI interaction supported Measure by surfacing how AI performance varies across that structured profile space, facilitating downstream risk characterization.

Medical profiles provided the clinical foundation for risk assessment, requiring realistic and diverse medical contexts to evaluate performance across clinical heterogeneity. Generating such profiles from structured EHR data faces interconnected challenges: high-dimensional feature spaces (10⁵-10⁶ concept codes) with sparse task-relevant signals, data quality issues (missingness and inconsistent coding), and medically implausible feature combinations [54-56]. Our design adhered to five AI RMF–aligned trustworthiness requirements: (1) controllability (emphasize task-relevant features while maintaining outcome diversity), (2) coherence (maintain clinical plausibility across diagnoses, treatments, and temporal events, including rare but valid scenarios), (3) variability (capture heterogeneous comorbidities and contextual factors to expose brittleness and subgroup bias), (4) efficiency (balance clinical completeness with computational tractability for large-scale evaluation), and (5) transparency (maintain traceable feature lineage to source distributions for targeted risk analysis). The mapping of these requirements to data challenges and mitigated risks is provided in Multimedia Appendix 1.

Medical profiles establish clinical context but cannot capture communication-dependent and interaction-dependent risks, requiring 2 additional profile dimensions, consistent with the safety taxonomy proposed by Lim et al [57]. These are needed because accurate medical retrieval can still fail due to how a patient communicates or how the conversation unfolds. The 2 dimensions map onto the patient input types and hazardous scenarios mentioned by Lim et al [57], informing the communication-dependent and interaction-dependent categories used here.

Communication-dependent risks emerge from heterogeneity in patient expression. Agents must maintain safety and comprehensibility across health literacy levels, condition-specific patterns, and vernacular variations [58]. Linguistic profiles enable controlled assessment grounded in health literacy and psycholinguistic research [59].

Interaction-dependent risks emerge when patients manipulate information, test boundaries, or withhold details [60]. Because evaluation cannot assess safety mechanism robustness without systematic behavioral variation, behavioral profiles are grounded in clinical and human-computer interaction research to probe agent resilience across conditions.

The patient simulator integrated 3 profiles: medical profiles grounded in EHR data, linguistic profiles capturing health literacy and condition-specific variation, and behavioral profiles representing engagement and interaction patterns. Profiles were operationalized through structured LLM prompting, where the medical profile assigned hierarchical indices to each clinical fact (eg, [3.2] Individual Psychotherapy). For each intake question, the model identified relevant indexed facts; applied linguistic style transfer (eg, [3.2] “talked to someone”); and constructed a JSON response containing the original facts, style-transferred equivalents, and a final natural language response with inline span markers. Simulator-provided markers were removed before being passed to the AI decision aid to ensure independent extraction. The complete prompt is provided in Multimedia Appendix 1.

Phase 1 generated outcome-relevant, coherent, and diverse patient profiles. Phase 2 applied probabilistic selection using a binomial distribution derived from All of Us antidepressant response data, ensuring that the final cohort reflects realistic population-level variability.

Profile generation used three core design principles [61,62]: (1) prioritize outcome-relevant features, (2) enforce statistical independence among selected features, and (3) inject controlled diversity from residual features, implemented through 4 stages (relevance filtering, demographic initialization, independence screening, and diversity expansion), as outlined in Figure 2.

The algorithm restricted feature candidates to the top K predictors of the antidepressant response outcome, e₀ (K=500), improving sample efficiency and aligning with the max-relevance component of minimum redundancy maximum relevance feature selection [62].

Each profile S was initialized with age and gender from All of Us demographic distributions.

Features were added iteratively, with each candidate v screened for statistical independence from already-selected features. The RR RR(s,v) quantifies the association between features s and v with respect to antidepressant response:

where values near 1 indicate independence, values >1.5 indicate positive association, and values <0.67 indicate negative association. Each candidate was required to satisfy:

This symmetric band excluded near-deterministic couplings (RR >high) and strong anticorrelations (RR <1/1.5), ensuring statistically independent and clinically coherent feature combinations [61,62]. The upper threshold (high) was set to 7.

After forming a coherent feature set, residual features outside the top-K set were added if they satisfied:

This captured latent contextual variables enriching patient heterogeneity without compromising coherence [61,62]. The complete pseudocode is provided in Multimedia Appendix 2.

Phase 2 selected patients whose response probabilities reproduce the All of Us population-level distribution by dividing the probability range into 7 sigma bands of a binomial distribution B(n,p), with sampling weighted to match expected binomial frequencies. For example, with n=100 and p=0.4 (μ=40; σ≈4.9), approximately 64% of patients fall within (μ-σ, μ+σ), 15% in each adjacent band (μ±1σ to μ±2σ), and <3% in the tails beyond μ±2σ. This stratified sampling preserves central tendency and natural variability while avoiding over- or underrepresentation of extreme responders. Detailed sigma-band allocations appear in Multimedia Appendix 2.

For AI RMF alignment, controllability and coherence were achieved through outcome-relevant feature selection and RR screening, captured by the proportion of outcome-related features and the average RR. Variability was assessed via the number of unique concept codes, efficiency was determined by the average features per profile, and transparency was evaluated through explicit feature lineage enabling auditable profile construction.

The framework had 2 tunable hyperparameters governing profile density: the high threshold from stage 3 and the number of residual features added in stage 4 (set here to 3‐5 per profile). Lowering high narrowed the set of candidate concepts eligible for inclusion. Varying it across {4,5,6} yielded 5‐8 features per profile on average versus 8‐9 at high=7, a gradual rather than sharp change indicating limited sensitivity to the exact threshold within this range. Either hyperparameter can be adjusted to target a desired profile density.

Conversational agents must maintain safety and comprehensibility across diverse patient expression styles [63,64]. Systematic linguistic variation exposes blind spots and failure modes [65,66]. The simulator implemented a dual-axis linguistic framework with profiles along two independent dimensions: (1) a health literacy gradient capturing variation in comprehension, terminology, and discourse structure [19,59,67,68], and (2) condition-specific communication reflecting linguistic patterns characteristic of depression and anxiety disorders derived from Linguistic Inquiry and Word Count (LIWC)–based clinical analyses [69]. Health literacy generalizes across clinical tasks as comprehension barriers affect patient-agent interaction regardless of medical condition. Condition-specific profiles capture diagnostic patterns (depression and anxiety) adaptable to other conditions. Table 1 specifies the 5 linguistic profiles.

The linguistic profiles align with NIST requirements through literature-grounded linguistic variation [19,59,67-69], enabling controllable assessment of communication-dependent risks.

Conversational agents struggle when patients go off-topic, respond vaguely, or withhold information [70,71]. Systematic behavioral variation is essential for stress-testing agent robustness and assessing recovery from conversational breakdowns [72,73]. We organized the 13 patient behaviors documented by Simpson et al [74] into 4 behavioral categories (complete mapping is provided in Multimedia Appendix 2), with structured and cooperative as an additional fifth category. This study operationalized 3 profiles: distracted and unfocused and adversarial and combative to capture challenging dynamics, with structured and cooperative as the baseline. We selected this subset to bound experimental scope while preserving the contrast that matters for risk assessment: a cooperative baseline against 2 profiles that diverge in distinct ways. Each profile varied along 4 dimensions: conversational adherence, engagement, topical focus, and adversarial behavior, as detailed in Table 2.

The behavioral profiles align with NIST requirements through empirically grounded behavioral variation [74], enabling controlled assessment of interaction-dependent risks including adversarial scenarios with transparent lineage.

Together, these 3 profile dimensions provide comprehensive Map and Measure coverage for conversational health care AI risk assessment.

The evaluation framework examined patient simulator performance across medical, linguistic, and behavioral dimensions: medical profiles through human and LLM-based annotation using a 3-label schema (accurate, inaccurate, and unsupported), and linguistic and behavioral profiles through human annotation, quantitative metrics, and visual clustering.

All annotations were performed by 2 annotators with complementary domain expertise: one had an undergraduate degree in psychology, and one was a registered nurse. Each sample was independently annotated by both raters, with disagreements resolved through adjudication.

The patient simulator CoT process (1) identified relevant medical concepts, (2) rephrased each concept according to the linguistic profile, and (3) constructed a natural language response based on linguistic and behavioral characteristics. For each conversational turn, the simulator outputted (1) relevant concepts referenced numerically (eg, [2.3]) and (2) a natural language response with inline spans marking where each concept is expressed. These spans served as annotation units labeled using a three-category schema as follows: (1) accurate, where the medical fact is correctly expressed with minor colloquialisms (eg, “happy pills” for “Prozac”) permitted if core clinical meaning is preserved; (2) inaccurate, where the fact is present but misrepresented, distorting critical clinical details or using implausible phrasing; and (3) unsupported, where content does not correspond to the patient’s profile, capturing hallucinated or fabricated details outside tagged spans. This approach evaluated expression fidelity conditioned on correct concept retrieval. Concept recall was computed programmatically from the simulator’s traced outputs, with 95% of profile concepts appearing at least once in the generated conversations.

We validated the annotation and judging pipeline itself through controlled error injection. The patient simulator expressed nearly all medical concepts accurately, so annotators could achieve artificially high interannotator agreement (IAA) by labeling all concepts as accurate, and the LLM judge validation would lack discriminative power. To enable rigorous evaluation, we introduced controlled semantic perturbations by replacing clinical concepts with semantically similar but clinically distinct alternatives (eg, “hypertension,” “prehypertension,” “diabetes mellitus,” and “prediabetes”), selected via a semantic search over SNOMED CT and Current Procedural Terminology, 4th edition (CPT-4) that identified the top 20 candidates, which were randomly shuffled before ontology-based filtering removed hierarchical ancestors or descendants and enforced minimum hierarchical distance. The first qualifying candidate was selected, ensuring variation rather than defaulting to the nearest semantic neighbor; if none qualified, the threshold was relaxed or the pool expanded. The resulting perturbed profiles maintained linguistic realism while introducing subtle clinical inaccuracies for evaluating both human annotation quality and LLM judge performance.

Linguistic profiles were evaluated through human annotation, where annotators classified each conversation into 1 of 5 predefined linguistic profiles. Five automated metrics were also used:

Annotators classified each conversation into 1 of 3 behavior profiles. Three automated metrics quantified behavioral patterns:

LLM-based judges have demonstrated strong performance in health care evaluation tasks [81-83]. Automated natural language processing similarity metrics, by contrast, have not been shown to correlate with expert human evaluation of generative LLM outputs on EHR data [84]. We used a separate LLM, Claude Opus 4.6, as an automated judge, applying the same annotation schema across medical, linguistic, and behavioral dimensions. The judge prompt was developed on 45 held-out conversations without human annotations. Judge alignment was then validated against human annotations on the full evaluation set, with results presented in the Results section.

This section evaluates AI decision aid performance across 500 simulated conversations.

Overall concept recall reached 93% (n=2622) across 2819 reference concepts, indicating that approximately 7% (n=197) were not identified by the AI decision aid. Recall was the highest for diagnoses (1958/2077, 94.3%) and procedures (518/569, 91.0%), while medications showed the lowest recall (146/173, 84.4%), identifying medication retrieval as a relative vulnerability. The system additionally introduced 151 concepts outside predefined profiles, predominantly medications (n=84) and procedures (n=57), reflecting conversational elaboration.

The retrieval performance of concepts identified by the AI decision aid was reported on the full set of simulator-generated reference concepts (n=2819), with unidentified concepts contributing to overall outcomes. Overall rank-1 accuracy was 65.9% (1857/2819); however, 80.2% (2261/2819) of concepts appeared within the top 20 candidates, indicating ranking imprecision rather than missing semantic coverage. Retrieval accuracy increased with health literacy level. Rank-1 accuracy ranged from 47.6% (limited) to 81.9% (proficient). Condition-specific profiles fell in between, with depression at 62.0% and illness anxiety disorder at 63.7%. Vague or affect-heavy language reduced ranking accuracy (Multimedia Appendix 3).

Patients with limited health literacy or depression used colloquial, fragmented, or affect-heavy language that misaligned with the system’s canonical concept vocabulary. Because top-20 retrieval did not fully close this gap, the system introduces a structural bias: retrieval is reliable for patients who use precise medical terminology but degrades for populations expressing equivalent clinical content through less structured language (Multimedia Appendix 3).

Recommendations were produced by DEMI over 15 antidepressants plus a no recommendation category (probability <0.1), making downstream performance sensitive to information loss during intake. Performance improved with higher health literacy, with proficient achieving the highest F₁-score (0.73) under structured and cooperative and limited achieving the lowest F₁-score (0.48). Behavioral profiles further modulated this risk: structured and cooperative yielded more stable outcomes, while distracted and unfocused and adversarial and combative were associated with performance shifts that varied by linguistic profile, as shown in Figure 6.

The medical profile generation process produced clinically coherent profiles with explicit feature lineage traceable to source EHR data. To validate both annotator and LLM judge performance, we applied controlled error injection. Semantically similar but clinically distinct perturbations (eg, hypertension → prehypertension, diabetes mellitus → prediabetes, major depressive disorder moderate → major depressive disorder mild, and generalized anxiety disorder → adjustment disorder with anxiety) tested annotator and LLM judge sensitivity. The injection was challenging by design. Each perturbation is semantically close to the original concept, the kind of subtle substitution an LLM is likely to produce, and once rendered in a patient’s communication style, it is easily mistaken for an accurate expression. Agreement on perturbed concepts was lower than overall agreement (human-human: κ=0.30 vs κ=0.73; human-LLM: κ=0.24 vs κ=0.78). This partly reflects the intended difficulty of the perturbations, which approximate real-world concept ambiguity, but also signals a real detection limit relevant to deployment monitoring. If human raters and the LLM judge both struggle to catch subtle clinical substitutions under these controlled conditions, comparable errors are unlikely to be easier to catch in live use, where similar ambiguity could go undetected without dedicated monitoring.

The simulator produced measurably distinct profiles across both dimensions. Behavioral profiles formed categorically different clusters, while linguistic profiles formed a continuous health literacy gradient. The functional and proficient overlap reflects their genuine similarity along this continuum. Cross-dimensional analysis confirmed profile independence with 2 interaction effects: distracted and unfocused consistently increased response length, and adversarial and combative increased toxicity regardless of linguistic profile.

The linguistic profile substantially affected AI decision aid performance, with monotonic degradation across the health literacy gradient: rank-1 retrieval ranged from 47.6% (limited) to 69.6% (functional) and to 81.9% (proficient), with the gradient persisting in downstream antidepressant recommendation accuracy. The system relied on precise terminology: colloquial expressions like “my morning pill for my nerves” force the retrieval system to resolve a larger semantic distance than “20 mg of fluoxetine for generalized anxiety disorder.” These failures point to specific Manage-function interventions: clarification prompting, terminology normalization, or multipass retrieval. Condition-specific profiles showed intermediate retrieval performance (depression: 62.0%, illness anxiety disorder: 63.7%), suggesting that affective expression also degrades performance, though less severely than low health literacy.

The LLM judge aligned strongly with human annotators on medical concept agreement (overall F₁-score=0.93) and showed lower agreement on perturbed concepts (F₁-score=0.77), mirroring divergence among human annotators on those same items. The judge also flagged unsupported content beyond the perturbations, related to incidental medications and minor symptoms not present in the profiles. These results support LLM judge use for large-scale screening, with human review reserved for edge cases.

Performance differences across health literacy and behavioral profiles raise clinical, equity, and governance concerns that should be addressed before the AI decision aid is deployed.

Rank-1 retrieval fell from 81.9% (proficient) to 47.6% (limited). Because antidepressant trials typically last 6-12 weeks before effectiveness can be assessed [88,89], retrieval errors may not surface until after weeks of ineffective or harmful treatment. Limited health literacy is more common among patients with severe depression, lower socioeconomic status, advanced age, and less formal education [90,91], which are the same groups that bear the heaviest burden of untreated mental illness [92,93]. The performance gap is concentrated in the population most in need of accessible support. This is not a technical limitation to be addressed in a later release; it is an immediate patient safety risk requiring active mitigation before deployment.

The distracted and unfocused profile mirrors the psychomotor slowing, impaired attention, and disorganized thought of moderate-to-severe depression [94,95], which are core symptoms of the illness rather than mere variations in communication style. The adversarial and combative profile often reflects trauma history, prior negative health care experiences, or treatment frustration rather than a difficult personality [96,97]. The system is least reliable for patients whose illness severity itself makes communication difficult. These clinical interpretations explain why the observed performance gaps carry safety significance; the behavioral profiles were derived from a general interaction taxonomy [74] and have not been validated against symptom presentations aligned with the Diagnostic and Statistical Manual of Mental Disorders (DSM). Thus, the mapping to specific psychiatric phenotypes should be read as illustrative rather than diagnostic.

Provider-supervised use makes the identified gaps manageable, and direct-to-patient deployment, particularly to underserved populations with limited provider access, amplifies them. Deployment plans should specify the oversight model, review criteria, and escalation pathways for low-confidence recommendations. Under the NIST AI RMF, these governance structures correspond directly to Manage-function commitments and should be specified before deployment approval. Where the system informs treatment selection, providers should also interpret its outputs as patterns of medication persistence, recognizing that persistence and symptom remission can diverge in real-world treatment, with prognostic implications [98]. The Manage-function interventions (clarification prompting, terminology normalization, and adaptive retrieval) should be developed with input from patients with limited health literacy and providers experienced in psychiatric communication, and validated in the same simulation framework before rollout.

The performance gradient documented here is a familiar pattern in health care AI. Tools designed and validated on data representing dominant communication styles reinforce existing disparities at deployment scale rather than reducing them [99]. Antidepressant selection compounds this risk. Comorbidity, pharmacogenomics, and social determinants all shape outcomes, and patients with limited health literacy already face diagnostic and treatment gaps in standard psychiatric care [100,101]. A system that underperforms for these populations risks widening those gaps, and technical correction alone will not address this. The identified Manage-function interventions should be co-designed with patients carrying limited health literacy and with clinicians who serve them. Psychiatric care has developed practices for this work, including teach-back protocols, plain language standards, and culturally adapted patient education [102]. Conversational AI for these populations should be built on the same foundation.

Simulated conversations have not been validated against real patient interactions. Ecological validity will be evaluated in a planned real-world trial. The evaluation covers a single clinical task and 4 antidepressants. The framework itself does not assume a deployment model, but the harm potential of the performance gaps we report depends on how the AI decision aid is deployed. Future evaluations should match the oversight model under which it will be used.

Antidepressant response was operationalized as 10-week medication persistence without switching or augmentation, a surrogate influenced by formulary access, provider inertia, and barriers to follow-up, which does not capture clinical remission [103,104]. Future work should anchor response in patient-reported outcomes or validated symptom scales (eg, Patient Health Questionnaire-9 and Hamilton Rating Scale for Depression) [105,106].

Medical profiles relied exclusively on structured EHR data, omitting the unstructured clinical narrative where clinicians document symptom severity, treatment history, and psychosocial context. The independence screening criterion may exclude valid coupled comorbidities, underrepresenting patients with strongly correlated conditions. Behavioral profiles follow the taxonomy by Simpson et al [74] and are not validated against DSM-aligned presentations: the distracted and unfocused profile resembles depressive neurovegetative symptoms but is not clinically verified, and the adversarial and combative profile does not distinguish oppositional engagement from trauma- or frustration-driven presentations. Validation against psychiatrically characterized populations remains a priority.

The patient simulator and AI decision aid both used GPT-4.1, though information leakage between them was structurally prevented. The simulator’s internal profile and concept markers were stripped before its response reached the AI decision aid, which extracted clinical information from natural language text alone, as it would from a real patient. A shared model family could still produce correlated blind spots not present with 2 independently trained systems. The LLM judge (Claude Opus 4.6) was a different model, mitigating this risk at the evaluation layer but not for the simulator-decision aid pairing. Future work should test decision aid performance against simulator output using an alternative model family on a subsample.

All annotations were performed by 2 raters. While their complementary clinical backgrounds (psychology and nursing) and adjudicated agreement provide some assurance, a 2-rater design limits the generalizability of the reported agreement estimates and cannot capture the full range of expert interpretation. Future evaluations should incorporate a larger, more diverse annotator pool.