To minimize editing after generation, the model was designed to produce draft SOAP notes in the same style as clinicians at KidsAbility. Ideal training data for this purpose would include historical SOAP notes linked to corresponding point-form scratch notes. However, because clinicians do not typically save their scratch notes, such paired data were not available.

KidsAbility had access to tens of thousands of historical SOAP notes stored in their previous EMR, GoldCare, which were migrated to their new system, AlayaCare, in 2023. The data were deidentified and made accessible to the research team via Amazon Web Services. Additionally, historical progress notes from AlayaCare were later provided to the research team within the same secure infrastructure.

During data review, the research team found systematic differences between notes authored before and after the migration. Notes from AlayaCare, collected between January 2023 and March 2024, followed a more consistent SOAP structure, due to the use of standardized templates in the new EMR. In contrast, the older notes often lacked clear section headings and were highly variable in style. For clarity, these datasets are referred to as the GoldCare dataset (larger body of historical data from prior to the EMR migration) and the AlayaCare dataset (the smaller but better formatted dataset of notes from after the migration).

The GoldCare dataset consisted of 411,812 progress notes taken from 2012 to 2023. The data were filtered to include only OT progress notes, excluding speech-language therapy and physiotherapy notes. After this filtering, the dataset contained 37,437 OT notes.

The GoldCare dataset was also filtered by token length, using the Llama 2 7B tokenizer, to include notes with only 50-1000 tokens, accounting for the majority of the notes within the distribution. Inspection of examples showed that notes with less than 50 tokens were less likely to be proper SOAP notes, and a 1000-token maximum enabled faster, more cost-effective pretraining with a sequence length of 1024.

Furthermore, the dataset was divided into separate datasets including a domain-adaptive pretraining (DAPT) dataset (90%) and a fine-tuning dataset (10%). The division of the full dataset considered client IDs such that individual clients appeared only in one dataset or the other. Finally, the fine-tuning dataset was filtered once more to exclude any notes under 150 tokens since further inspection revealed that notes in the range of 50-150 tokens were less likely to conform to SOAP format. The pretraining dataset was not filtered further. This left 2298 notes in the fine-tuning dataset and 25,164 notes in the pretraining dataset.

The AlayaCare dataset originally consisted of 1127 SOAP notes from OT appointments, which were used only for fine-tuning as the dataset was not large enough to support DAPT. This dataset was filtered to include notes with 100-1000 tokens, using the Llama 3 8B tokenizer, which is similar to the Llama 2 7B tokenizer. A smaller minimum length was allowed than in the GoldCare fine-tuning dataset because these notes were more reliably structured despite shorter token lengths. After filtering, the AlayaCare fine-tuning dataset comprised 973 notes in total.

The models chosen for training were the Llama 2 7B Chat model and then the Llama 3 8B Instruct model after its release in 2024. The smallest Llama models were chosen to minimize training and inference compute costs. Compared with other open-source LLMs, the Llama models are well documented, perform well on benchmarks, and performed best in our initial, exploratory testing. The Chat and Instruct versions of these models are those that have undergone fine-tuning and posttraining that refine the models’ ability to respond to chat prompts and instructions, which was desired for the given task.

For both DAPT and fine-tuning on the SOAP note task, the models were trained through causal language modeling (CLM), which is a self-supervised training process that tasks the model with autoregressively predicting the next token in a sequence [54]. The model was trained for a single epoch to reduce the risk of memorizing the training data and compromising privacy [48,55,56]. The hyperparameters were based on those used in the Llama 2 fine-tuning process [57]. Further implementation details, including the hyperparameters, can be found in Multimedia Appendix 1.

LLMs are pretrained on massive text corpora. DAPT is a form of transfer learning that involves continued pretraining using CLM on text from a specific domain. DAPT is used to train a general pretrained model to understand domain-specific vocabulary and nuances and to produce text more suitable to the target domain [58]. The GoldCare pretraining dataset, composed of 25,164 historical progress notes, was used for DAPT. Training was performed for 1 epoch on the Llama 2 7B Chat model.

In contrast with DAPT, fine-tuning is used to adapt a model to a particular task, rather than just a particular kind of text. Parameter-efficient fine-tuning was explored, specifically low-rank adaptation (LoRA), a prevalent method that trains large models efficiently in terms of both computation and the size of the fine-tuning dataset [59,60]. This method uses low-rank adapters, n × n weight matrices that consist of an outer product of 2 low-dimensional weight matrices (eg, n × 2 and 2 × n), forcing these weights to have low rank and few parameters. A low-rank weight matrix is summed with an existing n × n weight matrix. Only the low-rank adapters are optimized during LoRA fine-tuning while the original parallel weights are frozen.

The GoldCare and AlayaCare fine-tuning datasets were used for different fine-tuning training runs. Each of these datasets was further split into train and validation datasets, such that 80% of the data were in the train dataset and 20% were used for evaluation. Supervised fine-tuning was performed for a single epoch. The performance of these variations was evaluated as described in the “Qualitative Model Evaluation” section. The best-performing process, with a fine-tuned Llama 3 8B Instruct model, leading to a version that was piloted with clinicians, is shown in Figure 1.

The best-performing model, as determined by the qualitative evaluation, was incorporated into a web-based user interface and piloted with a group of 10 occupational therapists at KidsAbility. The goal of the pilot was to compare timing and quality measures between the traditional manual process, the custom model, and Microsoft’s Copilot application, which is based on GPT-4. Due to constraints such as the timing of the study and the availability of clinicians, the pilot was sequential, noncrossover, and nonrandomized. The occupational therapists first wrote all their SOAP notes using Copilot for3 weeks. For each SOAP note, they submitted an online form that indicated the time and date of the note and the self-reported time spent writing it. After a week break from the pilot during which clinicians could either continue using Copilot or revert to a manual process, the clinicians used the custom model for a period of 3 weeks. In this stage, our web application automatically saved their scratch notes, generated draft SOAP notes, and edited SOAP notes. As before, they submitted a form with self-reported time spent on each note. This self-reported time included time typing scratch notes, interacting with the model, and editing the generated SOAP note. All notes required clinician review and editing before being submitted.

SOAP note quality was evaluated by a separate research group to reduce bias [62]. Briefly, 4 clinicians, each with at least 2 years of experience in providing pediatric rehabilitation treatment, independently reviewed 256 SOAP notes. The SOAP notes included 64 notes from 4 different categories: manually written SOAP notes (Non-AI Notes), notes written by Copilot and edited by the clinicians (Copilot Edited), unedited SOAP notes from the custom model (Custom), and notes from the custom model that had been edited (Custom Edited). Each clinician was given a rubric with 5 criteria (Clear, Complete, Concise, Relevant, and Organized) on which to score the 256 SOAP notes and a spreadsheet that contained the SOAP notes in a randomized order. The clinicians were not informed of the category of each note, although this may sometimes have been apparent from the note contents, as the manual notes varied more widely in style.

Although self-reported timing data were collected during the pilot, the research team sought more precise timing data. Therapists often paused or switched to other tasks while working on SOAP notes, so that automated time stamps would not accurately reflect the time used to write the notes. To collect more accurate timing data, an experimenter observed clinicians and collected timing data in virtual meetings. Four clinicians participated and were asked to write 2-5 SOAP notes per meeting. The clinicians were asked to refrain from beginning their note, including typing any scratch notes, prior to the call. During the virtual meeting, a method for writing each note was chosen at random, with a 50% chance of using the custom model or manually writing the note. A timer was started when the occupational therapist had opened the application they intended to use and were ready to start typing. The timer was stopped once the occupational therapist reported submitting the note in the AlayaCare EMR.

At the start of the KidsAbility Pilot Study, much of the feedback about the custom model included actionable suggestions for improvement. Clinicians requested that the model use a more consistent bullet point marker, which was quickly addressed by adding a find-and-replace function to the code. They also noted that the model produced high-quality plan and analysis sections, especially compared with the Copilot model, and that the text it generated was professional, concise, readable, and familiar in tone. Sample-generated notes can be seen in Multimedia Appendix 2 listings 1 and 3 for Llama 3 and Copilot, respectively. Negative feedback included frustrations with the editing process, incorrect placement of information under headings, occasional omissions of details from scratch notes, and rare instances of irrelevant or incorrect text being added.

Subjective reporting of the time taken to write SOAP notes with the custom model varied from 5 to 26 minutes, with an average of 13.83 (SD 7.10) minutes (n=73). In comparison, the average time for using Copilot was 14.08 (SD 8.99) minutes (n=190). Significant variability in the time required to complete notes with both models was reported, and the difference was not statistically significant between models.

Some therapists appeared to use the custom model more efficiently than others. Upon investigation, the KidsAbility Innovation Team found that the 2 therapists who reported the best time savings were providing much shorter scratch notes, while other therapists were spending more time to write out detailed scratch notes for the model that already closely resembled full SOAP notes. Following this discovery, the innovation team held one-on-one sessions with 6 of the clinicians, coaching the clinicians to provide sparser scratch notes to the model and to spend less time writing these. Examples of a detailed scratch note and a sparse scratch note can be seen in Multimedia Appendix 3.

The average time taken after this coaching, based on self-reporting, was 7.6 (SD 3) minutes (n=6). An example from one of the coaching sessions illustrates the potential impact of coaching. Initially, a clinician spent 10 minutes writing detailed scratch notes, including SOAP headings. After one-on-one coaching, the clinician produced a much sparser version of the scratch notes in 1 minute, and the entire process was completed in approximately 6.5 minutes. While anecdotal, these observations suggest that the model has the potential to reduce documentation time, but the actual time savings depend strongly on how the model is used.

The clinicians at KidsAbility were accountable for reviewing all generated notes and editing them as appropriate. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) scores [63] were calculated to compare each edited model note against the unedited, generated note.

ROUGE scores measure similarity between a piece of text and a reference piece of text. ROUGE-1 scores are F₁-scores for which the recall and precision are calculated using the number of overlapping words. While ROUGE-1 scores provide a measure of the similarity based on the number of individual words that are the same, ROUGE-2 scores are based on 2-word sequences. ROUGE-L and ROUGE-LSUM scores indicate sentence-level similarity and similarity at the level of the entire note, respectively [63,64]. A higher score indicates a higher degree of similarity and, in this case, less editing.

Sixty-four pairs of edited and unedited notes were compared. The means and SDs of 4 different ROUGE metrics calculated for all model output notes compared with the final edited notes are shown in Table 2. All the ROUGE scores were high, indicating light editing at both the word and document levels. Somewhat lower ROUGE-2 and ROUGE-L scores may indicate that most changes are related to word order and sentence structure.

The mean scores for the 5 criteria across the 4 different groups of SOAP notes are shown in Table 3, and the corresponding SDs are shown in Table 4. The Custom Edited notes had the highest or second-highest scores for all 5 criteria. Meanwhile, the Non-AI notes were consistently scored lowest except on the Conciseness criterion. Finally, the custom model notes had the lowest SDs across all criteria, with the Custom Edited notes having the lowest for 4 out of 5. It is possible that the low SD indicates higher consistency in quality.

Two-factor ANOVA tests with replication on each of the 5 criteria were performed, with the 2 factors being the clinician evaluator and the category of note. Both main factors were significant with P<.001 across all criteria (17.37 ≤ F_3,1004 ≤126.17). There were also moderate interaction effects between the 2 factors for all criteria, the greatest of which was F_9,1004=12.16, P<.001 for clarity. The ANOVA test was appropriate as a robust omnibus test for Likert-type ratings given the large, balanced sample and approximately symmetrical residuals.

Pair-wise comparisons between scores of different note categories were performed using Tukey test to correct the significance threshold for multiple comparisons. These tests showed that there was a significant difference (0.25‐0.77, P<.001) in mean scores between the Non-AI notes and the Custom Edited notes for all criteria except for conciseness. Conversely, there was a significant difference (0.31, P<.001) between the mean scores for the custom model notes and Copilot notes only for conciseness. See Multimedia Appendix 4 for further details.

Interrater reliability was measured using the intraclass correlation coefficient (ICC) metric. The metric varied by criterion, with single-rater reliabilities being low (ICC_3,1=0.16-0.41), indicating limited interchangeability of clinician evaluators [65]. In contrast, reliability of the average rating across clinicians was higher (ICC_3,4=0.43-0.74).

Online meetings were used to record more accurate timing data for a small number of notes. A member of the research team timed the note-taking duration of 18 instances, including 10 using the custom model and 8 manually written. The average time spent on writing notes with the custom model was 9.55 (SD 3.5) minutes (n=10). The manually written notes took an average of 7.84 (SD 3.1) minutes (n=8). Of note is the large degree of variability in the times recorded for both categories of notes. Varied complexity of different notes was a factor in this variation. For example, for one of the custom model notes, the clinician commented that it was much more complex than usual and could be considered an outlier. Similarly, the clinician who manually wrote another note commented that it was shorter than usual because it was for a brief virtual call that did not contain any interaction with the client. Excluding these 2 outliers, the average time taken to write notes with the custom model was 8.76 (SD 2.6) minutes, while the average time for the manual process was 8.38 (SD 2.9) minutes. This difference was not statistically significant.

Despite the fact that 3 of the 4 clinicians who participated had previously gone through one-on-one coaching to use the model more effectively, they did not appear to implement the more effective strategy. The scratch notes were longer and more detailed than they had been coached previously to write.

A surprising result was that progress notes that were produced using this model scored better on multiple quality measures than manually written notes. Specifically, compared with SOAP notes written without the use of LLMs, those written with either Copilot or the custom model were given higher scores for all 5 criteria, apart from Conciseness. Non-AI notes received higher scores for Conciseness than the Copilot model notes but not the edited custom model notes. Therefore, the use of LLMs did not degrade the quality of the SOAP notes but instead helped to produce higher quality SOAP notes.

Progress notes produced with the custom model also compared favorably with progress notes produced with Microsoft Copilot. Comparing the notes generated with the custom model with the notes generated with the Copilot model, the custom model received higher scores for conciseness regardless of editing, but otherwise the scores for edited notes of both models were similar. Clinicians informally corroborated this finding, reporting that the Copilot notes were longer and contained more “fluff” than those from the custom model.

The impact of the models on therapists’ time was less clear. During the pilot study, it was found that self-reported time savings varied substantially between therapists. Further investigation suggested that this variation related to different ways in which different therapists used the model. Most clinicians in the pilot wrote long, detailed scratch notes and still spent significant time editing the generated notes. This approach was inefficient, as the model could only rephrase or rearrange points without adding meaningful content. Therapists who used the model more effectively found that writing quick, minimal scratch notes allowed the model to produce a usable draft, which they could then edit. After being coached by the research team, all therapists reported better results and were able to create quality SOAP notes in less time, demonstrating the importance of hands-on training to maximize the model’s effectiveness.

It is unsurprising that writing more detailed scratch notes would reduce the benefit of the system but perhaps more surprising that many therapists were inclined to do so. During the objective timing study, several therapists were observed to have reverted to giving the model long, detailed scratch notes despite previous coaching and positive experience providing short, sparse notes. Further work is needed to understand how best to encourage more efficient use of the system. For example, this could involve user interface changes to show an ideal example, or a limit on the number of words that can be entered in the scratch note interface. Clinicians specifically expressed a wish to provide further prompts or instructions to the model for a given note, which could improve model usability with sparse scratch notes. Alternatively, an analysis of model limitations that may be driving provision of longer scratch notes could be pursued. It is possible that the synthetic training data, which were more well-structured and clearer than typical manual scratch notes, may contribute to model limitations by introducing a bias toward longer and more complete scratch notes. Further fine-tuning of the model on the clinicians’ scratch notes and the progress notes produced by the model and edited by clinicians could improve model performance. If the model with further fine-tuning were to produce progress notes more aligned with their preferences given sparse scratch notes, clinicians may be less inclined to provide overly long scratch notes. Overall, work to promote efficient use of the system appears to be critical for realizing the efficiency gains that would be needed to impact waitlists for treatment. However, even if the model were used optimally, impact on waitlists would require therapists to take on larger caseloads.

Clinicians’ previous experiences with the Copilot model may also have contributed to their initial, less effective use of the custom model. Some had developed biases, believing that detailed input was necessary for generating useful SOAP notes, as they were dissatisfied with Copilot’s Analysis and Plan sections when sparse notes were provided. As a result, some clinicians used only Copilot for the Subjective and Objective sections, preferring to write the other parts themselves. This may have influenced their approach to the new model.

The models that underwent DAPT experienced catastrophic forgetting, where their ability to respond to prompts worsened after continued training. See Multimedia Appendix 5 for an example of degraded performance. Specifically, DAPT caused the models to ignore instructions, similar to findings by Cheng et al [66], who describe a similar phenomenon in their paper, proposing the use of reading comprehension tasks to reduce this issue by training the model on more diverse prompts. However, their methodology was impractical for reframing the SOAP note dataset. Thus, it is hypothesized that a lack of diversity in the DAPT prompts contributed to the degradation in performance, as the training examples did not contain any explicit instructions and merely presented SOAP notes.

Huang et al [67] also propose a mitigation strategy for catastrophic forgetting due to continued training, using self-synthesized data that are meant to reflect the original training data to retain the model’s original proficiencies. Future work on this project could use similar methods to include other types of examples, reflective of original training examples, in the process of DAPT along with the examples of SOAP notes.

Öncel et al [68] explored factors that play a role in whether performance degrades with further training. They found that greater similarity between the original training corpus and the new domain’s corpus resulted in more degradation. Given the breadth of the Llama training corpus for Llama 2, it is possible that the original data the model was trained on contained examples very similar to our DAPT data. Öncel et al also found that increasing the model size could mitigate performance degradation. Therefore, it is hypothesized that the small size of the model (relative to other language models) played a role in the observed catastrophic forgetting. An interesting avenue for future work could include training the model on smaller, more curated datasets and determining when catastrophic forgetting begins to emerge.

Additionally, the models that underwent DAPT were more prone to hallucinating irrelevant text in the SOAP notes, perhaps due to low variety in the DAPT examples. The model often generated repetitive phrases such as COVID-19 protocol text instead of responding dynamically to the scratch notes. Öncel et al [68] make a relevant observation in their study, observing that the models tend to improve at predicting domain-specific tokens and to become worse at generic tokens, especially structural tokens. The COVID-19 protocol appears frequently in training data and so may be seen by the model as a set of domain-specific tokens that should be predicted often.

LoRA fine-tuning was faster than full fine-tuning (9.7 minutes vs 28.9 minutes on the GoldCare fine-tuning dataset). Merging adapter weights back into the model added 12 minutes on average. However, despite a minor speed advantage, LoRA fine-tuned models had higher training and evaluation losses than fully fine-tuned models.

Consistent with these higher losses, the LoRA-trained models generated SOAP notes that were less concise, often overly long, and prone to hallucinating irrelevant or inappropriate text. Many notes failed to follow the correct SOAP format and sometimes included repetitive or nonsensical sections, with placeholder tokens such as “[GOAL]” or “[DATE]” appearing in the note. These issues made the LoRA fine-tuned models less suitable for this task than fully fine-tuned models.

Comparing the generated notes of the Llama 2 and Llama 3 fine-tuned models, the Llama 3 model generally produced more readable notes. The Llama 3 model was more likely to change the wording of a particular point in a scratch note to improve clarity, while still conveying the original meaning. Furthermore, a large difference between the 2 models was that the fine-tuned Llama 3 model responded more appropriately to instructions after training. The fine-tuned Llama 2 model, like the DAPT models, demonstrated reduced ability to respond to instructions [66], most likely due to training on a narrower task. However, the Llama 3 model did not exhibit decreased performance in responding to instructions, instead responding appropriately to different prompts even after fine-tuning. In this way, the fine-tuned Llama 3 model could be prompted to, for instance, use specific headers only or be of a more specific length.

Clinician availability was a limiting factor in this work due to their existing workload. Clinicians were not available for extensive evaluation of the model outputs, nor was there time for clinicians to produce scratch notes for use in training the model. Furthermore, the sequential, nonrandomized nature of the pilot study potentially introduced order effects such that the use of the Copilot model prior to use of the custom model may have influenced how the clinicians used the custom model. Comparative performance and time-saving metrics between the 2 models should be interpreted with this in mind.

The impact of the system on therapists’ time was unclear. Timing data from the pilot study were self-reported. The team made a preliminary attempt to collect more accurate timing data through online meetings. A statistical power analysis with G*Power indicated that a sample size of 176 for each category (total 352 data points) would be needed to find a moderate difference. Thus, a much larger sample would be required, and the described findings cannot be considered definitive. However, the limited data collected, and the fact that therapists had reverted to an inefficient process, argue against substantial time savings. In future work, it would be best to revise the process so that therapists provide sparse scratch notes before collecting a larger timing dataset. Additionally, future work could include clinician satisfaction ratings or other usability metrics and feedback to report more thoroughly on the clinician experience outside of time taken to write notes.

Regarding model performance, the choice to train for only a single epoch may have resulted in poorer performance than could have been achieved with further training (eg, with 3 epochs as commonly suggested for such fine-tuning) [69,70]. There exists an inherent trade-off that in limiting the number of epochs to mitigate memorization and privacy risks, the model’s performance and generalizability will likely be poorer. Also, the use of synthetic scratch notes may have resulted in bias and degraded performance as well.

Finally, this study was focused only on OT SOAP notes for a single institution, and it is not clear whether the same type of model would be generalizable to other settings or disciplines such as physiotherapy. The model would likely not be as useful for other organizations, given that it is trained to mimic the style and culture of the therapists at KidsAbility. A model for a different program at a different organization or for a new discipline would require adaptations but could likely be developed following the same general procedure.