The dataset consisted of the EHR data of patients from the Massachusetts General Hospital (MGH) memory clinic (collected between 2015 and 2022), who were over 50 years old at their first visit and had at least two MGH memory clinic encounters. The dataset was further filtered to exclude patients without an office or telemedicine visit or those who did not have a progress note with at least 512 characters. The final dataset was filtered to only include patients with 1 of 6 ADRD diagnoses during their latest encounter: AD, DLB, FTD, PD, PPA, or VCI, and without mixed dementia in their EHR history. See Multimedia Appendix 1 for the full list of diagnosis names by ADRD category.

This study was approved by the Mass General Brigham Institutional Review Board (protocol 2015P001915), with a waiver of informed consent granted for secondary analysis of electronic health records. No participant compensation was provided. Data were extracted from Epic and securely stored on servers within the Mass General Brigham firewall, with access limited to authorized study personnel in accordance with institutional privacy and data security policies.

To process the notes, we applied medspaCy, a specialized text analysis tool for clinical notes [19]. We extracted key sections of the notes that held important information regarding the patient’s symptoms such as medical history, examination, and impression. The extraction tool was customized for each physician’s template. Subsequently, we sampled notes based on ADRD diagnoses and split notes into sentences or phrases for symptom annotation.

An expert (AB) conducted thorough review of the medical literature and identified symptoms from seven domains typically present in patients living with ADRD: (1) memory, (2) executive function, (3) motor, (4) language, (5) visuospatial, (6) neuropsychiatric (which also incorporates symptoms related to behavior and mood), and (7) sleep (Multimedia Appendix 2). A behavioral neurologist (JD) provided critical input throughout both processes. Subsequently, another expert (MM) annotated sentences or phrases as symptom (patient shows intact or impaired symptoms) or no symptom (no information on patient symptoms). Further, MM annotated sentences or phrases as intact, impaired, or no information for each of the 7 symptom domains, using a web-based JavaScript annotation tool developed by AS. Using these annotations, we created 2 gold standard datasets: gold standard dataset I (composed of sentences or phrases labeled as symptom or no symptom) and gold standard dataset II (composed of sentences or phrases labeled as intact, impaired, or no information across the 7 symptom domains). The process for creating the gold standard dataset is illustrated in Figure 1A.

We developed a 2-tier hierarchical model for symptom extraction. The stage I binary symptom classification model classified each input sentence as symptom or no symptom. The stage II multi-label symptom classification model is composed of 7 distinct models, with each trained to classify sentences or phrases from 1 of the 7 symptom domains, namely memory, executive function, motor, language, visuospatial, neuropsychiatric, and sleep. Each stage II multi-label symptom classification model classifies sentences or phrases into 3 categories: impaired, intact, and no information. The impaired category encapsulates symptoms indicative of impairment within the specific domain, highlighting manifestations of dysfunction. Conversely, the intact category encompasses symptoms that reflect normal functioning of the respective symptom domain. The no information category encompasses all remaining symptoms from other categories (eg, a sentence that only mentions motor symptom is categorized as no information in the memory model), supplemented by nonsymptomatic sentences.

Both the stage I binary symptom classification model and stage II multi-label symptom classification model were developed using BioBERT [20], an LLM pretrained on a large corpus of biomedical text (eg, PubMed abstracts and PubMed Central full-text articles) and implemented using the HuggingFace’s Python transformers package (version 4.8.2) [21]. The stage I binary symptom classification model was initialized with its pretrained parameters of BioBERT and then fine-tuned on the gold standard dataset I (80% training set, 20% hold-out set). The stage II multi-label symptom classification model was again initialized with pretrained parameters and later fine-tuned on gold standard dataset II (80% training set, 20% hold-out set). Optuna hyperparameter tuning was used to tune the hyperparameters for both models, including training epochs, batch size, and learning rate, with a 20-trial study to maximize the area under the precision-recall curve. An early stopping criterion was implemented to cease training if the loss did not change substantially in 4 epochs, preventing overfitting.

Figure 1B shows how we used BioBERT for the stage I and stage II models. We used the pretrained BioBERT model as a starting point and fine-tuned it for our task. As shown in Figure 1B, the extracted sentences are first processed through the BioBERT tokenizer, which splits the raw text into tokens. For example, the sentence “Patient has difficulty walking” is tokenized. Then, each token is converted into a pretrained embedding, capturing the semantic meaning of the word in the context of the sentence, along with a position embedding that encodes the token’s location within the sequence to help the model understand word order and structure. A [CLS] token is added at the beginning of each sentence. Its embedding is used to represent the aggregated meaning of the entire sentence. A [SEP] token is placed at the end to signify the boundary between input tokens. E (embedding) from 1 to n represents the token embeddings, with the total count of n including [CLS] and [SEP]. These embeddings are passed through BioBERT’s transformer layers, which use self-attention and feed-forward neural networks to generate context-aware embeddings. As the sentence passes through the layers, the embedding of the [CLS] token becomes enriched with contextualized information derived from the full sentence, which represents the overall meaning of the input. Finally, the embedding of the [CLS] token is used as the input for the linear layer, which calculates the logits for each class. Sigmoid (for binary classification) or SoftMax (for multi-class classification) as a decision function is applied to these logits to obtain class probabilities, and the class with the highest probability is selected as the model’s predicted label. We fine-tuned BioBERT separately for stage I (binary classification) using gold standard dataset I and for stage II (multi-label classification) using gold standard dataset II. The fine-tuning process primarily involves adjusting the parameters of the BioBERT transformer layers and the linear layer to optimize performance for each stage’s specific classification task.

We also experimented with other pretrained models as part of our preliminary experiments, including ClinicalBERT, RoBERTa, and LLaMA 2, with the latter being a generative transformer model. Despite fine-tuning (for ClinicalBERT and RoBERTa) or prompt engineering (for LLaMA 2), the models did not achieve the same level of performance as BioBERT in symptom classification based on the area under the receiver operating characteristic curve (AUROC) and F₁-score. All text processing and LLM development procedures were conducted in Python (version 3.8.15).

We created a list of regex patterns for ADRD symptoms to compare the efficacy of our advanced LLM approach with the traditional rule-based regex technique. First, 100 patient visit notes across the 6 ADRD diagnoses (AD, DLB, FTD, PD, PPA, and VCI) were randomly sampled. These notes were analyzed to identify examples from each of the 7 symptom domains (memory, executive function, motor, language, visuospatial, neuropsychiatric, and sleep) and develop a comprehensive set of regex patterns for each symptom domain. An expert behavioral neurologist (JD) provided critical guidance throughout this process. Next, these regex patterns were used to flag sentences or phrases corresponding to each symptom domain in the entire set of visit notes. The symptom counts for each note were then aggregated to calculate the total number of matches for each domain. For the full list of regex patterns, please see Multimedia Appendix 3.

We compiled symptom counts across 7 domains (memory, executive function, motor, language, visuospatial, neuropsychiatric, and sleep) based on predictions of our 2-tier hierarchical model on the entire set of visit notes. These symptom counts served as input features for a multinomial L1-regularized logistic regression model to classify 6 ADRD diagnoses (AD, DLB, FTD, PD, PPA, and VCI). To optimize the model, we employed 5-fold cross-validation and grid search cross-validation to determine the optimal value of alpha for L1 regularization using the Python scikit-learn (version 0.24.2) package. Additionally, we incorporated the aggregated symptom counts, derived from applying the ADRD symptom regex patterns (Multimedia Appendix 3) on the same dataset, as features in the machine learning model. We hypothesized that symptoms identified with our 2-tier hierarchical model would have superior performance than those derived from regex patterns in predicting ADRD diagnoses. All ADRD differential diagnosis analyses were conducted in Python (version 3.8.15).

To evaluate symptom predictions using MRI, we selected memory clinic notes with an MRI scan performed within 1 year of the visit. We ensured that none of these notes overlapped with the gold standard datasets. Each clinical note was matched with a unique MRI scan from the Mass General Brigham patient database, with the imaging date being within 1 year of the visit date. The SynthSeg+ pipeline [22] was used for brain segmentation and volume estimation. Only those images whose subcortical regions collectively surpassed a threshold of 0.65 in the average automated quality control score were selected for further analysis. For patients with multiple eligible clinical images, the final brain volume was determined by averaging the volumes across all qualifying images. Furthermore, to account for individual differences, the volume of each brain region was normalized by the intracranial volume.

In our brain volume analysis, we first selected a priori brain regions associated with 2 of the most commonly disrupted functions in patients with ADRD: memory and motor. For memory symptoms, we investigated the bilateral hippocampus and entorhinal cortex, both associated with the memory of recent events, as well as the prefrontal cortex, which is related to immediate memory [2,23-25]. For motor symptoms, our evaluation encompassed the bilateral primary motor cortex, the secondary motor cortex, the basal ganglia (including the caudate, putamen, pallidum, and nucleus accumbens) along with the thalamus (a structure with strong connections to the basal ganglia), and the cerebellar gray and white matter [26-29].

Logistic regression was used to evaluate the volumes of brain regions associated with symptoms, with a contrast of cases having impaired symptoms and those having either intact symptoms or no information. The analysis was conducted for both memory and motor symptoms, with adjustments made for age and sex, using the function glm in the R stats (version 4.3.2) package. The reported results were adjusted for multiple comparisons using the Benjamini-Hochberg method [30]. All MRI brain volume analyses were conducted in R (version 4.2.1; R Core Team). For a detailed workflow of validation using MRI, see Figure S1 in Multimedia Appendix 4.

The study data consisted of visit notes from the latest encounters of 1712 patients (Figure 2). The visit notes were from 866 (50.6%) male and 846 (49.4%) female patients, with an average age at visit of 77.5 (SD 8.3) years. All patients had 1 of the following ADRD diagnoses: AD, DLB, FTD, PD, PPA, and VCI. The patient demographics are described in Table 1.

From these 1712 visit notes, we compiled 2 gold standard datasets. Gold standard dataset I included 10,089 sentences or phrases labeled as symptom (n=5468, 54.2%) or no symptom (n=4621, 45.8%). Gold standard dataset II included 6784 sentences or phrases labeled as intact, impaired, or no information across the 7 symptom domains. The ADRD diagnoses in dataset II predominantly included AD (2862/6784, 42.2%) and DLB (1866/6784, 27.5%), followed by FTD (879/6784, 13.0%), PD (628/6784, 9.3%), VCI (479/6784, 7.1%), and PPA (70/6784, 1.0%). Specifically, AD had the highest counts for memory and visuospatial symptoms; DLB led in executive function symptoms; PD was predominant in motor symptoms; PPA led in language symptoms; and FTD was notable for neuropsychiatric and sleep symptoms, with high counts also noted in visuospatial and sleep symptoms for VCI and DLB, respectively (refer to Table 2 for detailed distributions). A standardized mean difference (SMD) threshold of 0.1 was employed to assess the equilibrium of each metric, with measurements exceeding 0.1 indicating a comparative lack of balance. The MRI validation dataset included 582 visit notes from 528 unique patients and had clinical MRI performed within 1 year (Figure S2 in Multimedia Appendix 4). For demographic distribution related to these visit notes, refer to the last column of Table 1.

We trained, validated, and tested a transformer-based LLM to identify symptoms related to ADRD diagnoses. The symptom extraction process was executed through a 2-stage framework. The stage I binary symptom classification model categorized sentences as either symptom or no symptom. The model attained a micro-averaged AUROC of 1.00 (95% CI 0.99-1.00), along with a micro-averaged F₁-score of 0.98 (95% CI 0.97-0.98), micro-averaged precision of 0.98 (95% CI 0.97-0.98), and micro-averaged recall of 0.98 (95% CI 0.97-0.98), highlighting its ability to accurately detect symptom presence. The 95% CIs for each metric reflect the reliability of these estimates, confirming the model’s overall efficacy in symptom classification across diverse clinical features.

This initial classification is followed by the use of the stage II multi-label symptom classification models, which further classify each detected symptom into impaired, intact, and no information. The 7 stage II models are tailored to each specific domain (memory, executive function, motor, language, visuospatial, neuropsychiatric, and sleep). All symptom domains showed robust model performance, with micro-averaged AUROC values of 0.97-0.99, micro-averaged F₁-score values of 0.89-0.96, micro-averaged precision values of 0.87-0.96, and micro-averaged recall values of 0.91-0.96 across all symptoms. Among these, we observed slightly lower metrics in the visuospatial domain (micro-averaged AUROC: 0.97, 95% CI 0.95-0.99; micro-averaged F₁-score: 0.89, 95% CI 0.85-0.93; micro-averaged precision: 0.87, 95% CI 0.83-0.91; micro-averaged recall: 0.91, 95% CI 0.87-0.94). Table 3 provides a comprehensive evaluation of the performance metrics for both models.

To validate the accuracy of our 2-tier hierarchical symptom classification model, we used a machine learning model to classify ADRD diagnoses with the counts of identified symptoms as model features. We compared 2 L1-regularized logistic regression models: one based on regex-derived symptom counts and another using counts derived from the 2-tier hierarchical LLM. This method allowed us to assess the efficacy of traditional regex techniques against more advanced LLM approaches in the context of ADRD diagnostic accuracy.

First, we predicted ADRD diagnoses using L1 logistic regression based on regex-derived symptom counts. Using regex patterns, we extracted symptom counts from the latest visit notes of 1712 patients diagnosed with ADRD, spanning 7 domains: memory, executive function, motor, language, visuospatial, neuropsychiatric, and sleep. These counts were used to build an L1-regularized multinomial logistic regression model, which predicted the type of ADRD diagnosis using symptom counts as features. The model’s average AUROC was 0.59 (95% CI 0.51-0.66). Detailed AUROC values for each ADRD diagnosis relative to the rest are displayed in Figure 3A.

Second, we predicted ADRD diagnoses using L1 logistic regression based on LLM symptom counts. The second model, leveraging symptom counts extracted from patient visit notes via the 2-tier hierarchical LLM, aimed to predict specific ADRD diagnoses using L1-regularized logistic regression. This model demonstrated a substantial enhancement in diagnostic accuracy, achieving an AUROC of 0.83 (95% CI 0.77-0.89) compared to the AUROC of 0.59 (95% CI 0.51-0.66) obtained with the regex-based model. This marked improvement highlights the model’s efficacy in accurately classifying ADRD categories, underscoring the potential of transformer-based BioBERT models in capturing the context of clinical symptoms from notes. The detailed AUROC for each diagnosis compared to the rest is displayed in Figure 3B.

Further, analysis using feature importance derived from the LLM-based logistic regression model showed that executive function had the greatest predictive power on average, followed by language, motor, memory, neuropsychiatric, visuospatial, and sleep. This ranking, illustrated in Figure 3C, emphasizes the critical roles of executive function, language, memory, and motor symptoms in predicting ADRD diagnoses. Feature importance rankings for each ADRD diagnosis are illustrated in Figure S3 in Multimedia Appendix 4.

We used MRI brain volume data to assess our model’s ability to identify symptoms from clinical notes. We hypothesized that the volumes of selected brain regions associated with each domain would be smaller in patients with impaired symptoms predicted from the notes compared to those without. The model analyzed 582 sentences or phrases, identifying memory impairment in 90.7% (528/582) and motor impairment in 80.6% (469/582) of cases. In particular, we observed that memory-impaired individuals showed smaller hippocampal and prefrontal cortex volumes (SMDs >0.1), while motor-impaired individuals had reduced volumes in subcortical regions, including the thalamus, putamen, pallidum, and accumbens area (SMDs >0.1). For brain volume summary statistics from memory and motor BioBERT model predictions, see Figure 4A.

The memory model predicted that visit notes of patients with AD had the highest proportion (93.7%) of memory symptoms relative to the other ADRD diagnoses, which is consistent with our understanding that memory impairment is the initial and primary symptom for most patients with AD [2] (Figure 4B). The MRI analysis of memory symptoms revealed that a smaller hippocampal volume was associated with an increased likelihood of memory impairment (odds ratio [OR] 0.62, 95% CI 0.46-0.84; P=.006) (Figure 4C). Power analysis for the logistic regression, using 1000 simulations, yielded an 89.7% chance of detecting a significant impact of hippocampal volume on memory symptoms, thereby confirming the reliability of these findings. Nonetheless, the volumes of the entorhinal cortex and prefrontal cortex did not show a significant relationship with memory symptoms (P>.05), but the prefrontal cortex had high SMDs (Figure 4A).

In terms of motor symptoms, the motor model predicted that visit notes with DLB (95.5%) and PD (100%) diagnoses had the highest proportion of motor symptoms across visit notes of ADRD diagnoses, which is consistent with our understanding that motor impairment is the primary symptom for patients with DLB and PD [3,5] (Figure 4D). The MRI analysis of motor symptoms revealed that a smaller pallidum size was significantly associated with the presence of motor impairments (OR 0.73, 95% CI 0.58-0.9; P=.04) (Figure 4E). Power analysis for the logistic regression, conducted with 1000 simulations, revealed an 84.7% probability of accurately detecting a significant influence of pallidum volume on motor symptoms, which substantiates the robustness of our results. Other regions related to motor function did not exhibit significant volumetric differences (P>.05). Age and sex were accounted for in all analyses. All results were corrected for multiple comparisons [30]. Thus, the MRI findings corroborated both memory and motor symptom predictions made by our 2-tier hierarchical LLM.

We performed an error analysis to gain insights into the misclassifications made by the 2-tier hierarchical LLM, particularly in its ability to classify symptoms as intact or impaired across the 7 domains. We included both the held-out test set and the MRI validation dataset in our analysis to ensure thoroughness. It is worth mentioning that since the MRI validation dataset does not include true labels, we relied on chart reviews to validate predictions (Figure S1 in Multimedia Appendix 4).

Our error analysis began by examining instances where the models’ predictions of symptoms across ADRD types did not align with known disease profiles. For example, in AD cases, where memory impairment is a prominent symptom [2], the model did not predict memory symptoms in 6.1% (23/378) of cases. These instances were notable for their focus on broader cognitive decline or general test scores rather than explicit mentions of memory symptoms. In FTD, 93% (62/67) of visit notes referenced memory symptoms, which is intriguing since memory impairment is not typical in FTD, particularly in its behavioral variant [31]. Manual review confirmed that these symptoms were indeed documented. In VCI, 87% (47/54) of visit notes mentioned memory symptoms, with a consistent recognition of memory issues as a feature of VCI [32]. The model detected memory symptoms in 84% (37/44) of DLB visit notes and 79% (19/24) of PPA cases, which often concerned semantic memory challenges. Another example involves motor symptoms. The model showed a small margin of error in DLB cases, failing to detect motor symptoms in just 2 cases (2/44, 5%). In AD visit notes, motor symptoms were predicted accurately in 76.2% (288/378) of notes. FTD cases showed an 88% (59/67) occurrence of motor symptoms, and VCI notes included motor symptom references in 92.6% (50/54) of cases, often related to lower body motor challenges. PPA patients were identified with motor symptoms in 63% (15/24) of notes, with manual verification confirming the presence of true motor symptoms in majority (11/15, 73%) of these cases.

The second part of the error analysis investigated visit notes by random sampling, with a focus on notes with high symptom counts (more than 10 symptom predictions). This examination uncovered several types of errors affecting prediction accuracy across all symptoms, including six types of false positives: (1) generalizing cognitive function as a symptom, (2) confusing one symptom with another symptom, (3) identifying evaluation or test statements as impairment, (4) misrecognizing intact as impaired, (5) misleading by ambiguous or complex sentences, and (6) confusing medical history as present symptoms. Four types of false negatives were also identified, including (1) overlooking particular expressions, (2) overlooking particular test scores, (3) misrecognizing impaired as intact, and (4) overlooking sentences or phrases that require contextual information. Table 4 provides a detailed breakdown of these error types and examples from visit notes. Additionally, to understand the distribution of false positives and false negatives across the model’s predictions at the sentence level, we calculated confusion matrices based on the held-out test set for each symptom, and the data are presented in Figure S4 in Multimedia Appendix 4.