Electronic health record (EHR) notes serve as valuable sources of information that can significantly benefit patients. Programs like OpenNotes [1] and the Blue Button [2] initiative empower patients by providing access to their EHR notes [3-9]. Nevertheless, the benefits of accessing EHR notes can diminish if patients do not comprehend their content [10-15]. EHR notes are lengthy and filled with medical jargon [16-19], which can be difficult to comprehend for the average US adult, whose reading ability is around the seventh- to eighth-grade level [20-24]. Therefore, supportive technologies are needed to assist patients in understanding EHR content [25,26], focusing on linking medical terms to lay-friendly terms [27-29], consumer-oriented definitions [18], and educational materials [30]. Early studies have demonstrated that such interventions significantly enhance patient understanding [18,27]. However, initial methods primarily relied on frequency- and context-based approaches to identify unfamiliar terms and propose simpler synonyms [27-29]. Identifying and extracting complex medical jargon from EHR notes is a crucial step toward improving patients’ comprehension, ultimately enhancing patient engagement and reducing anxiety about their health [18,31-33].

Notably, not all medical jargon extracted from EHR notes holds equal clinical importance [34,35]. Existing tools, such as MetaMap [36], ScispaCy [37], medspaCy [38], and QuickUMLS [39], are effective at extracting medical terms, typically predefined terms from the Unified Medical Language System (UMLS) [40], but often fail to prioritize these terms based on their relevance to individual patients, treating all terms as equally important [27-29]. In previous work, we asked physicians to identify medical jargon terms from EHR notes that are important to patients [34]. Our results showed that physicians were able to consistently identify 5 to 10 medical jargon terms from each EHR note and rank each term based on its importance to patients [34]. Furthermore, we developed feature-rich traditional machine learning models (eg, support vector machines) to identify such terms [34]. However, our previous work did not focus on ranking jargon terms based on their importance to individual patients within an EHR note.

In this study, we propose large language model (LLM)-based natural language processing approaches to identify and rank jargon terms from EHR notes based on their importance to patients. LLMs have demonstrated tremendous promise in biomedical natural language processing applications [41-52] due to their exceptional generalizability and performance. However, applications in medical term extraction have primarily focused on tasks such as biomedical named entity recognition (BioNER) [53-55], rather than on prioritizing terms most relevant to patients, which is crucial for enhancing communication between patients and health care providers.

The key contributions of this study are as follows:

Identifying jargon terms important to patients is part of the BioNER task, which involves identifying predefined entities in a text and labeling each token with the corresponding entity. Medical entities encompass categories such as diseases, medications, treatments, laboratory tests, and more [56,57]. Studies such as [58-61] have introduced language models for BioNER tasks, while more recent studies [53-55,62,63] have explored the application of LLMs in BioNER. However, BioNER tasks primarily focus on extracting entities without considering their importance and relevance to the personal needs of patients, which distinguishes them from our objective.

MedJex [32] fine-tunes pretrained language models, such as Bidirectional Encoder Representations from Transformers (BERT [64]), Robustly Optimized BERT Pretraining approach (RoBERTa [59]), BioClinicalBERT [65], and BioBERT [58], on a domain-specific corpus. It leverages Wikipedia hyperlink spans during pretraining and transfers the learned weights to a target model fine-tuned on MedJ, an expert-annotated medical dataset. More recent studies [66] have investigated whether LLMs, such as ChatGPT [67], can outperform baseline pretrained language models (eg, MedJEx [32] and SciSpacy [37]) in extracting personalized medical jargon. Similarly, GAMedX [68], a medical data extractor using LLMs (Mistral 7B and Gemma 7B), uses chained prompts to navigate the complexities of specialized medical jargon. Other works [69-71] have demonstrated how LLMs can enhance the readability of EHR notes by extracting medical jargon.

This work also shares similarities with topic modeling, a task that extracts topics from input text. Using unsupervised learning algorithms, topic modeling can identify both explicit and implicit themes within a text corpus [72-74]. Through topic modeling, a text can be represented by multiple keywords or topics, which can then be incorporated into supervised models. However, topic modeling heavily relies on term frequencies and may easily overlook important terms that are clinically relevant to individual patients.

Among the most relevant works, such as FOCUS [34], ADS [75], and FIT [35], FOCUS [34] uses MetaMap [36] to extract medical jargon from EHR notes and uses feature-rich learning-to-rank techniques to determine whether the terms are important. However, none of the previous works have identified and ranked medical jargon terms in a note-specific manner. This is an important task, as ranking terms based on their relevance to a specific note may help the patient comprehend the note by linking important jargon terms to their lay definitions [76] or help generate patient-friendly after-visit summaries [77].

We evaluate the performance of the LLMs in three distinctive settings: (1) we assess the performance of closed- and open-source models by varying prompts and extraction tasks using the 10-fold annotated medical note (gold-standard dataset); (2) next, we benchmark the open-source models fine-tuned on portions of the gold-standard dataset; and (3) finally, we apply data augmentation, generating synthetic data from GPT-4o to fine-tune the open-source LLMs and evaluate them under the same varying settings. Our study finds that using data augmentation, the models can reach comparable or even superior performance in personalized medical jargon extraction tasks.

We evaluated both closed-source and open-source LLMs for their efficacy in extracting key information from annotated medical notes, aiming to assess performance across different strategies. Figure 1 provides an overview of our experiments, which leverage physician-annotated medical notes, closed- and open-source LLMs, and in-context learning (ICL). We examined the effects of prompting styles, fine-tuning, and data augmentation to enhance model performance.

Our gold-standard dataset consists of 90 medical notes, each annotated by two physicians [34,35]. For each note, agreement from both physicians was used as the final annotation. The annotation agreement (micro average) on these notes was 0.51 Cohen Kappa [35]. This EHR note dataset comprises text reports across six medical categories: cancer, chronic obstructive pulmonary disease, diabetes, heart failure, hypertension, and liver failure (Table 1). Each medical note includes detailed patient information and is accompanied by physician annotations highlighting the most critical terms or phrases relevant to the patient’s health status. Figure 2 presents a snippet of a sample EHR note from the gold-standard dataset.

We used both publicly available LLMs (open-source LLMs) and proprietary models that are not publicly available (closed-source LLMs). The open-source LLMs included Mistral 7B v0.1 [78] ( Mistral 7B), BioMistral 7B [79], Llama3.1-8B [80] (Llama 3.1 8B), DeepSeek-R1-Llama-Distill-8B [81] (DeepSeek 8B). For proprietary models, we used 2 closed-source LLMs, which were from OpenAI [67]: GPT-5.2 and GPT-5-mini.

We used both zero-shot and few-shot prompts, also known as ICL, to compare the performance of the LLMs. For zero-shot prompting, we provided the model with general instructions, whereas for few-shot prompting, we included two examples randomly selected from the gold-standard dataset.

To evaluate the model’s performance, we implemented two variations of prompts: general prompts and structured prompts, each designed to assess how effectively the model could extract relevant clinical information from medical notes. Multimedia Appendix 1 presents an example of a structured prompt. In this approach, the prompts are explicitly designed to closely align with the original task defined in the gold-standard dataset. The structured prompts instruct the LLMs to extract key medical conditions or diagnoses, followed by the relevant medications associated with them. In contrast, general prompts provide a more flexible and broader context. These prompts instruct the model to extract key medical terms without explicitly differentiating between conditions and medications, assigning the same base rank to both. This approach allows the LLM to interpret the extraction task more broadly, offering insights into how well the model generalizes its understanding of medical terminology when provided with less specific guidance.

To improve the performance of open-source LLMs (Mistral 7B, BioMistral 7B, Llama 3.1 8B, and DeepSeek 8B), we conducted low-rank adaptation [82] based on parameter-efficient fine-tuning. Low-rank adaptation is an efficient fine-tuning technique that allows models to be adapted to specific tasks without the need to update all of the model’s parameters. Instead, it applies low-rank updates to specific layers, reducing the computational cost and memory usage typically associated with traditional fine-tuning methods. The training was done with a batch size of 1 per device and gradient accumulation over 128 steps, and the low-rank dimension was set to 64. The learning rate was configured at 3e − 4, and the models were trained over 5 epochs to allow the models to converge effectively on the task-specific patterns present in the dataset.

Annotation by domain experts is expensive, and data augmentation using AI-generated data can help alleviate this challenge [83]. Few-shot prompting integrates task examples directly into input prompts, allowing models to observe patterns and generalize from limited examples to effectively handle new, unseen data [84]. We created an augmented dataset using the ICL technique, which was then used to refine the models (Multimedia Appendix 1). This augmented dataset was derived from discharge notes in the MIMIC-IV clinical database [85]. A subset of MIMIC-IV discharge notes was randomly selected, and GPT-4o from OpenAI [67] was used to process and rank key terms based on their importance for patient understanding. A dataset generated via GPT-4o was used because its outputs aligned more closely with the original annotations, exhibiting higher reliability during manual validation. The extraction process was guided by the examples, where two annotated notes from our gold-standard dataset were provided as examples to instruct the model on identifying and prioritizing terms.

After the raw generation, we filtered out terms that do not appear in the medical text using string matching. The resulting augmented notes were 9995 notes. Using only a few-shot technique and no other filtering mechanism, we anticipated a lot of noise being injected into the augmented dataset. To this end, we further conducted an analysis on the augmented dataset, collecting 100 random cases and comparing the synthetic annotations with an expert annotation. We asked a clinician to annotate the MIMIC-IV notes with the same instructions given to the GPT-4o and compared the agreement between them.

To explore the effects of augmented dataset size, we progressively increased the dataset across four scales: 10, 100, 1000, and 9995 notes. By testing the original gold-standard annotations along with AI-generated augmented data, we conducted a comprehensive evaluation of model performance across varying data sizes. Our objective was to enhance the robustness of LLMs in extracting critical medical information for patients.

We evaluated the performance of the open-source LLMs against several prominent baselines in the field of named entity recognition tasks: MedJEx [32] and BioClinical-ModernBERT [86]. For MedJEx, we did not fine-tune the model since it is already a trained model to extract medical jargon. Also, MedJEx does not output any confidence score; therefore, we could only report precision, recall, and F₁-score. For BioClinical-ModernBERT, we fine-tuned the model on our dataset and measured the performance. BioClinical-ModernBERT also outputs confidence scores. We used it as a proxy for ranking, assigning higher rankings for higher confidence scores. For the candidate entity extraction, we used MedspaCy [38] and QuickUMLS [39] to form inputs for the baseline models.

We used precision (Equation 1), recall (Equation 2), F₁-score (Equation 3), and mean reciprocal rank (MRR; Equation 4) as our metrics. Performance metrics, including precision, recall, and F₁-score, were first calculated at the individual note level (). To aggregate these into a single representative score for each of the 10 folds, we calculated the macroaverage across all notes within that fold. The final results reported are the mean and SD of these fold-level averages. Precision and recall gave insights into the model’s ability to cover the annotated terms. The macro-F₁-score allowed us to evaluate the model’s balanced performance. MRR metric (Equation 4), in particular, evaluated how well the model ranked the terms according to their relevance, reflecting the model’s alignment with the annotated order of terms. Precision and recall are provided in the Multimedia Appendix 2. For MRR calculation, integer-valued ranks were used for both outputs from using the general prompt and the structured prompt. Because the structured prompt outputs used integer values in the MRR calculation, some results could receive tied ranks. To avoid inflating MRR, we used average ranks for tied results (Adjusted Rank_i). Specifically, when multiple terms shared the same rank, each term was assigned the average of the positions they occupied. This provides a more conservative evaluation by penalizing ambiguous predictions and rewarding models that rank the correct term more specifically. We used two different string matching approaches: (1) relaxed string matching, which checks the matching as true positives when it either perfectly matches or contains a gold-standard input [87,88], and (2) the Jaccard Index method that uses set-based similarity to identify matches, using Jaccard Distance to measure the gap between two strings. Here, we set the similarity threshold to >0.5. Any extracted term achieving a Jaccard similarity score greater than 0.5 against the gold-standard term was counted as a true positive for the subsequent precision and recall calculations. While the Jaccard Index method penalizes verbosity, relaxed string matching does not. This enables us to have a clear picture of the extraction performance and quality.

To further understand the nature of the model’s behavior, we conducted an error analysis of the model’s outputs. First, we classified the model’s outputs that were erroneous. We used three types of errors: (1) spurious error (SE), where the model included terms that are not in the gold-standard dataset but are included in the medical note; (2) granularity error (GE), which reflects overly specific or overly broad extractions; and (3) misalignment error (ME), where the model simply did not understand the instruction and misbehaved. Using this taxonomy, we identified the erroneous behavior of the LLMs.

All experiments were performed with two Nvidia A100 graphics processing units, each with 40 GB of memory, an Intel Xeon Gold 6230 CPU, and 192 GB of RAM. We used Python 3.9 and the Hugging Face transformers library [89] for our experiment. For the closed-source models, we used OpenAI’s ChatGPT API [67]. To reduce the randomness of the experiment, we set the generation temperature to 0.1 for every model.

The requirement for ethical approval and informed consent was waived by the institutional review board at the VA Bedford Health Care System. The experiments were performed in accordance with the Declaration of Helsinki. The clinical data used for data augmentation were obtained from the MIMIC-IV database, a publicly available, deidentified repository of EHRs from patients admitted to the emergency department or an intensive care unit at the Beth Israel Deaconess Medical Center in Boston, Massachusetts. Access to the database was granted following the completion of the required PhysioNet Credentialed Health Data Use Agreement 1.5.0.

Our findings suggest that there are minimal differences between zero-shot and few-shot prompting. Although there are some cases where a few-shot prompt outperformed a zero-shot prompt, these results are not always consistent. Results differ by model, and it is hard to tell which prompting, zero-shot or few-shot, performs better. This is consistent with the findings reported by Chamieh et al [90], which found that few-shot prompting is not always more advantageous than zero-shot prompting (Figure 4).

The effectiveness of prompts can vary across models, with certain prompt styles enhancing performance for specific models (Table 3). While differences between models were generally minimal, some models performed better with particular prompt styles. For example, in the Llama 3.1 8B model, structured prompts outperformed general prompts with few-shot prompting, whereas in Mistral 7B and in other models, general prompts showed improved performance over structured prompts regardless of string matching metrics. For both Llama 3.1 8B and Mistral 7B, structured prompts induced higher recall than precision. Since a structured format gives more specific instructions, this may cause the model to generate concise and precise results (Tables S1 and S2 in Multimedia Appendix 2). A plausible explanation is that structured prompts impose stronger output constraints, narrowing the model’s generation space. Largely, our results align with prior research suggesting that tailored prompts can optimize a model’s performance in specialized tasks [91]. Interestingly, in most cases, we found that F₁-score trends are aligned with MRR scores, suggesting that certain prompts helped the model reach its full potential. This shows that testing diverse prompt styles is essential in maximizing model performance, which is highly aligned with the current research results [92-95].

Fine-tuning LLMs using domain-specific data proved effective in enhancing model performance, even surpassing the performance of closed-source models (GPT-5) under the Jaccard Index (Table 5). The size of the datasets used for fine-tuning was small but led to performance gains. Increasing the size of the fine-tuning dataset is likely to further improve the model’s performance [96]. Figure 5 illustrates the impact of fine-tuning: while vanilla models show limitations in extracting key points using prompts alone, fine-tuning enables the models to learn relevant patterns, leading to improved performance. Although BioMistral 7B could extract some information with instructions, MEs were present in its output. After conducting fine-tuning, the model dramatically improved performance, reducing its errors as shown in Figure 5. This trend is further evidenced by the DeepSeek 8B results in Table 6, where the SE exhibits a substantial reduction. Furthermore, the verbosity of model outputs was significantly reduced after fine-tuning, as reflected in improved Jaccard Index scores and overall performance gains. Previous research has similarly shown that fine-tuning on domain-specific data can significantly enhance performance by adjusting model weights to reflect the unique characteristics of the target domain, such as better handling of abbreviations, acronyms, and clinically relevant contexts [58]. Moreover, instruction fine-tuning has been shown to improve the model’s zero-shot performance [97].

Our results reveal that different string matching metrics yield divergent performance narratives. Closed-source models set the benchmark under relaxed string matching; however, fine-tuned models achieved superior F₁-scores and MRR when evaluated using the Jaccard Index. We contend that using a combination of string matching metrics enables a more nuanced assessment of model performance. Specifically, relaxed string matching prioritizes semantic preservation and the recognition of domain-specific terminology, while the Jaccard Index measures exact token-level overlap, thereby penalizing verbosity and rewarding concise outputs.

The data augmentation process relied solely on few-shot prompting, with string matching–based filtering as the only quality control measure. Consequently, the augmented dataset may be of lower quality than the gold-standard dataset. To evaluate this, we randomly sampled 100 instances from the augmented dataset and had a clinical expert manually annotate them. Agreement between the expert’s annotations and the synthetic labels was measured using the Jaccard Index [98], and to measure the ranking agreement, MRR. The scores were 0.255 and 0.585, respectively. The terms extracted showed low Jaccard Distance, which likely contributed to lower performance. However, for MRR, the ranking showed comparable performance.

We explored the impact of data augmentation using various sizes of MIMIC-IV [85] discharge notes to enhance the performance of LLMs. Data augmentation simulated diverse scenarios within medical notes, enabling the LLM to generalize better across a broader range of note types. In accordance with fine-tuning, the performance gains from data augmentation were meaningful (Figure 3). In some models, we found that using augmented data was helpful in improving the performance (Llama 3.1 8B, BioMistral 7B, DeepSeek 8B) under relaxed string matching in the top10 extraction task. Furthermore, all models trained on the maximum dataset size (n=9995) demonstrated improved F₁-scores that exceeded those of closed-source models under the Jaccard ˍindex–based string matching, but not under relaxed string matching. These findings align with prior studies [97,99], which argue that using LLM-generated data can improve model performance on downstream tasks. Additionally, studies have shown that while data augmentation provides benefits, substantial improvements often require a high degree of variation in the augmented data [100,101].

However, fine-tuning with an augmented dataset did not outperform fine-tuning with the gold-standard dataset for other models (Mistral 7B, BioMistral 7B, and DeepSeek 8B) when evaluated using Jaccard Index–based string matching (Table 5). Since we only used two examples for ICL, this may have affected the overall quality of the augmented dataset [102]. Nonetheless, this still leaves a question of why the LLMs’ performance increased in the first place, considering the quality of the augmented dataset. Although data augmentation did not generate the most “accurate” synthetic dataset, it did not generate critical errors that severely harmed the model’s performance. As a result, the augmented data provides weak but structured supervision that remains correlated with expert judgment. Consistent with prior findings in weakly supervised learning, such signals can still guide models toward meaningful patterns, particularly when aggregated across large-scale data. The observed performance improvements across models suggest that the augmented dataset captures clinically relevant structure despite instance-level disagreement, supporting the validity of our approach.

The effect of augmented dataset size on model performance varied depending on the string matching metric. Under relaxed string matching, only BioMistral 7B demonstrated a consistent performance improvement with increasing dataset size (Figure 3A). Conversely, string matching using the Jaccard Index showed that all models achieved higher F₁-scores as the dataset size grew (Figure 3B), indicating metric-dependent performance trends. Our results partially align with findings from Yuan et al [103] and Kim et al [104], where increasing the size of synthetic datasets significantly enhanced model performance compared to models trained on smaller real-world datasets. For other models such as DeepSeek 8B, the model’s performance oscillated over dataset size in top 5 and top 10 extraction under relaxed string matching, although it ultimately showed the best performance at n=9995 regardless of the metrics.

Based on our findings, we categorize the risks of deploying LLMs into two distinct domains: operational risk and clinical risk. (1) Operational risk encompasses failures in prompt adherence or structural inconsistencies where the model’s output deviates from the required format, potentially causing system-level failures. These risks are largely mitigable through rule-based validation and explicit format-checking mechanisms. (2) Clinical risk refers to the generation of factually incorrect information (hallucinations) that could directly compromise patient safety. Our results demonstrate that despite performance improvements across models, clinical risk remains a critical concern. Consequently, we argue that all LLM-generated outputs must undergo human-in-the-loop verification and be subjected to rigorous fact-checking protocols before clinical implementation.

This study has several limitations. First, we tested only a limited selection of available closed- and open-source LLMs. Specifically, we only tested models with fewer than 10B parameters, which are relatively small open-source LLMs. Second, our gold-standard dataset, being based solely on physician annotations, captures only the clinical perspective, not considering the actual needs of patients. Third, despite the improvements achieved through fine-tuning and data augmentation, the performance of the models still falls short of human annotation. Fourth, the effectiveness of these methods should be validated in real-world settings, such as Hyper-DREAM, which tests the efficacy of the blood pressure management system [105].

Our study provides a comprehensive evaluation of closed- and open-source LLMs for identifying and prioritizing medical jargon within expert-annotated EHR notes. Strategic interventions, prompting, fine-tuning, and data augmentation notably enhanced the use of open-source models for domain-specific clinical tasks. Fine-tuning on a gold-standard dataset emerged as the most effective overall strategy, yielding the highest performance gains and a substantial reduction in error counts under Jaccard Index–based string matching. While the data augmentation strategy did not consistently outperform fine-tuning, it demonstrated measurable improvements in model-dependent scenarios. Nevertheless, our error analysis underscores that all evaluated models, regardless of architecture or optimization status, retain inherent degrees of both operational and clinical risk.