For each hospital visit v_i, we collect physician-authored clinical notes and extract entities like conditions, procedures, and medications. To enrich context, we use PubMed literature parsed into knowledge triples (subject-relation-object) via an LLM-based extraction pipeline. We retain only triples that appear across patient visits. These triples form a biomedical KG, serving as an auxiliary source to support LLM reasoning and diagnosis.

The first step of the framework is generating patient context. We use EHR data, including physicians’ notes, patient conditions, prescribed medications, and procedures in natural language. Since physician notes can be lengthy and exceed the context window of a small, locally fine-tuned LLM, we summarize them using an LLM. This approach addresses Challenge 2 (Long Context).

To introduce domain-level medical knowledge, we build a biomedical KG by combining the UMLS [28], PubMed abstracts, and LLM-generated entity-relation triples. UMLS provides standardized biomedical concepts and relationships, and the entity-relation triples are structured facts extracted by the LLM in the form (entity₁, relation, entity₂), capturing semantic connections. We apply the Leiden algorithm for community detection [30], which partitions the KG into semantically coherent and well-connected subgraphs. After clustering, we use a separate LLM to generate a textual summary for each cluster (Figure S8 in Multimedia Appendix 1). These summaries are produced by reasoning over the relationships among entities within each cluster, capturing the latent biomedical semantics encoded in the graph. Each summary serves as a high-level abstraction of the biomedical concepts and interactions within a subgraph. We embed these summaries using SentenceTransformer (MiniLM-L6-v2) [31] and retrieve the most relevant ones for each patient by computing semantic similarity with the embedded patient context. This process directly addresses Challenge 3 (Specialized Medical Domains) by enriching patient context with structured, domain-specific knowledge, improving the model’s understanding of specialized medical terminology. Figure 2 illustrates a partial biomedical KG constructed from PubMed data, which is used to retrieve domain-relevant knowledge associated with each patient’s clinical context.

We provide additional context by retrieving semantically similar patient visits using precomputed visit-level embeddings and a similarity index using the Facebook AI Similarity Search [32] library with inner-product search on L2-normalized embeddings, which effectively approximates cosine similarity. For each target patient visit, we retrieve the top 50 most similar patients while excluding self-matches and other visits from the same individual. Each retrieved patient is scored by similarity, and we filter them into positive and negative cohorts based on matching or nonmatching ground truth labels (eg, readmission vs no readmission). The final output includes the top-k positive and negative similar patients (with k=1,2). Unlike KARE [15], we also provide the physician notes of the retrieved similar patients, enabling the language model to leverage more clinical context when assessing patient risk. To prevent data leakage, we maintain patient-level separation between training and testing sets. The similarity index was constructed exclusively using training-set patient embeddings. During testing, each test patient retrieved similar cases only from the training-set hub.

In this module, we prepare inputs to fine-tune the LLM for clinical prediction. For each patient visit, we create a prompt with the patient’s context with the top-k most similar cases retrieved earlier. These similar cases guide the model by highlighting patterns in clinically comparable scenarios. We also add biomedical knowledge summaries from clustered subgraphs of a PubMed KG, providing literature-based context. Combining patient data, historical cases, and domain knowledge, we fine-tune the LLM to produce task-specific predictions with interpretable reasoning, supporting each outcome.

We fine-tune a LLaMA-3 8B model using the Unsloth framework [33,34], which enables memory-efficient training via 4-bit quantization and low-rank adaptation [35]. Prediction tasks are framed as instruction-following using Alpaca-style prompts with task description, patient context, and optional justification. Each prompt combines clinical notes, retrieved similar cases, and biomedical knowledge summaries. The model is trained via supervised learning to generate both predictions and reasoning. We use limited training steps with gradient accumulation and sequence lengths up to 8192 tokens. Unlike KARE, which trains larger models with higher compute, our method uses smaller, quantized models to reduce computational cost while maintaining interpretability and performance. Algorithm 1 outlines the training and inference procedures of M₁. Additionally, an example prompt and its overall structure are provided in Multimedia Appendix 1.

We extract structured features from the patient’s visit history during each stay, including:

While both structured and unstructured models use information about conditions, medications, and procedures, they access this information from different data modalities. M₁ processes the clinical narrative and reasoning about these elements while M₂ processes the structured codes and standardized entries.

The structured data are first transformed using a discretization step to enforce uniform temporal resolution and impute missing values. This is followed by normalization using precomputed mean and SD statistics over the continuous variables.

To augment the structured input, we include the LLM’s prediction and its tokenized reasoning. For each patient visit, M₁ generates (1) a prediction probability and (2) a textual reasoning explanation. The reasoning text is embedded using SentenceTransformer (all-MiniLM-L6-v2) [36], resulting in a 384-dimensional vector. This embedding, together with the scalar prediction probability, is concatenated with structured features and provided as input to the final classifier in M₂.

The LLM-derived vector is merged with structured input features to create a unified representation, directly addressing Challenge 1 (Multimodal Information). To reduce dimensionality and suppress noise from high-dimensional embeddings, we apply principal component analysis to the combined feature vector.

Readmission within 30 days is a highly imbalanced task, with only about 4% (403/10,031) positive cases in the dataset. This severe imbalance is reflected in the results, where most models achieve high accuracy and precision on the negative class but struggle with recall for the positive (readmitted) class.

As shown in Table 3, our framework with a balanced random forest (KAMELEON-BalancedRF) classifier achieves the highest AUROC (0.845) and notably improves recall on positive cases to 0.79, a crucial metric since identifying patients at risk of readmission is clinically imperative. The KAMELEON-MLP model, while achieving the highest overall accuracy (0.91) and macro F₁ (0.58), still attains a sensitivity of 0.28 on positive cases, illustrating the persistent challenge in detecting rare events. Unstructured LLM-based baselines such as Claude-3.7-Sonnet, MedGemma, LLaMA3-Med, and KARE show substantially lower sensitivity for positives (below 0.3), suggesting that these models struggle to identify the minority class without further fine-tuning or domain-specific adaptation.

In Table 3, KAMELEON-BalancedRF achieves a precision of 0.13, meaning that about 1 in 8 patients flagged as high-risk were actually readmitted. Recall (sensitivity) is 0.79, indicating that the model correctly identifies nearly 8 out of 10 true readmissions—a clinically critical result. The F₁-score of 0.55 reflects the balance between precision and recall. The NPV is 0.99, showing that almost all patients predicted as low risk were indeed not readmitted. Specificity is 0.80, meaning the model correctly classifies 8 in 10 patients who were not readmitted as low risk.

To better understand this model’s behavior, we perform Shapley additive explanations (SHAP) analysis to identify feature importance, and Figure 3 indicates that the model relies primarily on prediction embeddings (59.3%) and laboratory/vital features (40.4%) for predicting 30-day readmission, highlighting the importance of multimodal inputs.

While prior studies like Morgan et al [45] reported AUROCs up to 0.81 for readmission, and general models typically ranged from 0.61 to 0.73 [46,47], our multimodal approach effectively captures complex clinical nuances.

We conduct an ablation study for the readmission task, where we retrain KAMELEON after dropping different components—the retraining and prediction models from M₁ and the demographics used in M₂ (Figure 3 and Table 4). We find that the reasoning component output by M₁ is very significant and affects multiple metrics beyond AUROC. In the full model, KAMELEON achieves balanced performance (accuracy=0.80; macro F₁=0.55; AUROC=0.844; AUPRC=0.147) with both high specificity (0.80) and sensitivity (0.77). When we drop the reasoning component, sensitivity falls by over 80% (falling to 0.06), and AUPRC is nearly halved, revealing strong bias toward the majority class; however, there is a gain in accuracy (rising to 0.92). Removing the reasoning component from KAMELEON drops performance from 0.844 to 0.7, a 17% decline in AUROC, highlighting the critical role of the fine-tuned LLM’s reasoning in risk prediction. Removing reasoning and prediction (output from M₁) causes intermediate degradation, with AUROC falling by ~18%, consistent with the loss of calibrated probability signals and semantic rationale. Eliminating reasoning together with demographics and prediction yields the sharpest overall decline, with macro F₁ dropping by ~13% and AUROC by more than ~22% (to 0.663), confirming the complementary value of these components. These analyses collectively demonstrate that reasoning substantially improves minority-class detection: it boosts sensitivity and AUPRC by more than threefold compared to variants without it, while also preventing misleading accuracy gains driven solely by the dominant negative class. Figure 3 visualizes these contributions, showing that reasoning, M₁’s prediction embedding, and demographics each add critical and nonredundant signals for accurate readmission risk estimation.

Textbox 1 shows a sample output generated by the fine-tuned LLM for the 30-day hospital readmission prediction task. Instead of returning a raw binary value (0 or 1), the model is prompted to generate both a prediction and its reasoning based on the patient’s diagnoses, procedures, medications, and comparisons with similar cases. This structured explanation allows the model to ground its prediction in a clinical context, improving reliability and interpretability over naive classification. By incorporating rationale into the output, the LLM demonstrates better alignment with real-world clinical decision-making.

Figure 4A illustrates the distribution of predicted probabilities for the readmission task. In the full model, the majority of negatives cluster near zero, while positives are shifted upward and concentrated above a low threshold of ~0.16. This optimal threshold is chosen using the Youden J statistic, which maximizes sensitivity and specificity. This separation, although not perfectly distinct, reflects the extreme class imbalance in the data: a very low cutoff is required to recover a reasonable fraction of positive cases. The overlap between classes explains the modest AUPRC values, since many positives still lie in regions dominated by negatives. Nevertheless, compared to ablated variants, the full model achieves tighter grouping of positives in the higher-probability region, resulting in better sensitivity and precision-recall trade-offs.

The t-distributed Stochastic Neighbor Embedding (t-SNE) plot (Figure 4B) further visualizes patients’ features on the held-out test set. In this space, readmitted patients do not form sharply separated clusters but instead appear partially embedded within the larger manifold of nonreadmitted cases. The lack of clear separation reflects the difficulty of the task and the subtlety of signals driving readmission, where positives and negatives overlap substantially. Still, there is evidence of localized groupings of readmitted patients, suggesting that the model captures some latent patterns that distinguish higher-risk subgroups. This partial clustering is consistent with the modest AUPRC values: while the model cannot fully disentangle the classes, it is able to concentrate a portion of true positives in regions of elevated probability. Clinically, this underscores the challenge of predicting readmission but also highlights the value of identifying even partially coherent patient subgroups for targeted follow-up.

Mortality prediction is less imbalanced, with approximately 13% positive cases. Table 3 reports that KAMELEON-XGBoost and KAMELEON-MLP models achieve high accuracy (0.92 and 0.90, respectively) and AUROC (0.92 and 0.89, respectively). KAMELEON-XGBoost demonstrates strong performance for mortality prediction, achieving a high precision of 0.79, meaning that most patients flagged as high risk did not survive. It also attains a specificity of 0.98 and an NPV of 0.92, indicating that nearly all patients predicted as low risk were indeed survivors. Furthermore, for a positive class that, while less imbalanced, is still a minority, the AUPRC is a vital metric. Here, the KAMELEON with 0.650 sets the benchmark, significantly outperforming all other baselines. Unstructured models consistently yield the lowest performance for mortality prediction, AUROC values hover just above random chance (around 0.51-0.53), and AUPRCs remain very low (maximum 0.125), indicating a limited ability to discern between mortality and survival solely from clinical notes. SHAP results (Figure 5B) demonstrate that laboratory results and vital signs strongly drive predictions, with prediction embeddings playing a less critical role compared to readmission. Overall, Table 3 shows that for both tasks, our multimodal model outperforms all individual structured and unstructured baselines across all metrics.

The SHAP values (Figure 5B) show that the prediction embedding has a smaller influence compared to laboratories and vitals, which contrasts with the readmission task. The AUROC curve (Figure 5A) further illustrates that removing reasoning reduces AUROC from ~0.92 in the full model to ~0.88, a relative drop of about 4%. This indicates that mortality prediction is comparatively easier, as a strong AUROC is retained even without reasoning. The lower-class imbalance and the strong signal contained in laboratory values and diagnostic codes allow the model to capture patterns associated with terminal illness more directly. Nevertheless, reasoning still contributes by improving discrimination at the margin, capturing subtler risk factors that are less apparent in structured features alone.

We show inference outputs for 2 patients on the mortality prediction task, generated by M₁, our fine-tuned LLM with reasoning, in Tables S10 and S11 in Multimedia Appendix 1.

Figure 6A shows predicted probabilities for in-hospital mortality, separated by true class: survivors (blue) and deceased (red). The dashed line marks the optimal threshold (0.276) balancing sensitivity and specificity.

Most survivors cluster near zero probability, reflecting strong model confidence, while deceased cases spread across a wider range, showing prediction uncertainty. Overlap between classes causes some misclassifications, highlighting the challenge of predicting this rare event. Despite this, the clear separation and tight clustering of survivors demonstrate the model’s strong ability to distinguish between classes, supporting the usefulness of the selected threshold.

Figure 6B shows a 2-dimensional t-SNE embedding of the test data, where the samples are colored by their true class labels (Class 0 in blue and Class 1 in red). The visualization reveals that the majority of the data forms a consistent, structured manifold dominated by Class 0 points, with only a small fraction of Class 1 points distributed across the embedding. Notably, a compact cluster of Class 1 samples appears on the right-hand side, indicating localized patterns that can be exploited by advanced models. This structure suggests that although Class 1 is relatively sparse, it exhibits distinct feature signatures in specific regions, which the proposed KAMELEON-X model is designed to capture effectively, contributing to improved predictive performance.