The study evaluated the feasibility of extracting QoL information from forum posts using LLMs through four phases: (1) model comparison: comparing 11 open-source LLMs of varying sizes (8B to 70B) in post-only and post+context settings; (2) optimization evaluation: assessing CoT prompting, instruction optimization, few-shot prompting, simultaneous all-questions prompting, and parameter-efficient fine-tuning with LoRA for the chosen midsized model; (3) evidence generation: prompting the previously chosen model to provide textual evidence supporting every correct yes and no prediction and comparing this evidence with human-annotated spans; (4) error analysis: calculating error counts per question and per error type. The goal of this framework is to provide a comprehensive assessment of LLMs’ capabilities in extracting QoL information and to identify potential limitations.

One of the foremost ethical concerns in AI health care is the security and confidentiality of sensitive health data [26]. While commercial LLMs such as GPT (OpenAI) or Gemini (Google) offer powerful linguistic capabilities to support various medical tasks, their cloud-based nature raises significant privacy challenges [27,28]. Using locally hosted, open-source models can provide a privacy-preserving alternative, adhering to health care regulations and ensuring that sensitive medical information remains within the local environment [29]. In the first part of the study, 11 open-source base (ie, noninstruct) LLMs representing diverse model families were selected: GPT-OSS (20B) [30], Qwen3 (14B, 30B) [31], Gemma3 (12B, 27B) [32], DeepSeek-R1 (14B, 32B, 70B) [33], Llama 3.1 (8B, 70B) [34], and Phi-4 (14B) [35]. The model sizes ranged from 8B to 70B parameters. Models were run locally via the Ollama [36] framework, avoiding transmission of data to cloud-based APIs. Experimental pipelines were implemented using the Declarative Self-improving Python (DSPy) [37] declarative framework, ensuring consistent input and output behavior (Textbox 1).

Due to the complexity of medical data, producing a comprehensive and accurate set of text spans corresponding to specific medical entities often requires involvement of multiple experts with specialized knowledge. The process is therefore costly and resource-intensive, which raises the question of whether LLMs could be used to streamline the process of annotating medical data [47,48].

The following experiment assessed whether LLMs are capable of generating textual evidence supporting their predictions, a task that could be later extended to producing annotations for the QoL information extraction dataset. For all correctly classified “yes” and “no” predictions in the best optimization setting, the chosen model was prompted to provide the exact sentence from the post that justified the answer. The evidence was compared against human-annotated spans contained in the original dataset and categorized into:

Table 2 illustrates the three match categories using examples of model-generated evidence and the corresponding human-annotated spans.

Token overlap was computed as the proportion of tokens shared between the model-generated evidence and the human-annotated span relative to the longer of the 2 spans.

The proportion of exact and partial evidence matches, relative to the total number of evaluated evidence, was calculated to assess how closely model-generated evidence corresponded to human-identified spans of relevant information.

The final experiment examined error distribution across QoL questions and the types of errors occurring:

The predictions for QoL questions with the highest misclassification rates were manually analyzed to identify systematic weaknesses, such as ambiguity or semantic overlap (eg, fatigue vs need to rest).

The study has been approved by the Ethics Committee of Bielefeld University under application number 2023‐216-W1. Informed consent was obtained from all participants to analyze their posts and comments for this study. The first 100 respondents received a US $20 gift card. The collected data is stored in encrypted form and available only to the authors of the study. All models used in the study are open-source and run in a local environment; therefore, no data have been transferred to the cloud servers.

Table 3 summarizes the macro F₁-scores across all 11 evaluated models. Qwen3-14B achieved the highest overall performance in both experimental configurations, reaching a macro F₁-score of 0.59 in the post-only setting and 0.56 when context was included. Removing context consistently improved performance of nearly all models, with ΔF₁ (denoting the increase in macro F₁-score after removing context) varying between 0.03 and 0.11. The results do not point to a single model family as universally superior for this task; however, Qwen3 and GPT-OSS families performed slightly better than the alternatives.

A granular analysis of the per-class metrics reveals the underlying difficulty of the task and the reason for the overall poor performance. While all models excelled at predicting the majority “not in the text” label with nearly all scores exceeding 0.96, performance collapsed on the minority classes. The highest recorded F₁-score for the “yes” class was only 0.50, while performance on the “no” class was even lower, at just 0.33.

Figure 2 illustrates the relationship between model size and the macro F₁-score for 2 different input configurations: post+context (square) and post-only (circle). There is not a simple linear relationship between model size and performance, with several midsized models (14B-30B) outperforming 70B models. Performance generally improves from smaller to midsized models and drops again for larger models.

For all subsequent optimization and fine-tuning experiments, GPT-OSS-20B in the post-only input setting was selected as a representative midsized architecture. It demonstrated one of the strongest zero-shot baseline performances among all evaluated models, serving as the baseline from which we would measure the impact of applying optimization strategies.

Table 4 summarizes the performance of GPT-OSS-20B across the different optimization strategies. Contrary to expectations, CoT prompting resulted in lower performance than the plain prediction baseline, decreasing the macro F₁-score from 0.58 to 0.52. Similarly, both MIPROv2 optimization strategies failed to outperform the baseline, achieving macro F₁-scores of 0.56 in the zero-shot setting and 0.54 in the few-shot setting. Bootstrap few-shot optimization with random search demonstrated the strongest performance among the prompting-based approaches, reaching a macro F₁-score of 0.60, representing only a small improvement over the baseline. Prompting all 53 QoL questions simultaneously resulted in the poorest performance (macro F₁-score=0.46), indicating that increasing the number of questions within a single prompt negatively affected extraction quality. In contrast, parameter-efficient fine-tuning with LoRA substantially improved predictive performance, achieving the highest overall macro F₁-score of 0.71.

Table 5 summarizes the alignment of model-generated textual evidence with human annotations. Out of all 4452 predictions, 47 were correctly classified with a “yes” or “no” label. In 42 of 47 (89%) predictions, the model-generated evidence either exactly matched or partially matched the human-annotated span. Exact matches were observed in 20 cases (n=47, 42%) and partial matches in 22 cases (n=47, 47%). Only 5 cases (n=47, 11%) were classified as no match for 4 different questions: “Did you feel ill or unwell?” (2 cases) and “Have you had pain?,” “Have you had any pain in the area of your affected breast?,” and “Has your physical condition or medical treatment interfered with your social activities?” (1 case each).

Analysis of model errors in the baseline setting revealed that a primary difficulty occurred when deciding whether relevant information was present in the text. Approximately 76% (85/112) of misclassifications constituted the model predicting a positive or negative answer for questions with the correct label “not in the text.” The remaining 24% (27/112) of errors occurred when the model predicted “not in the text” despite the presence of information supporting a “yes” or “no” answer. No cases were observed in which the model generated a reversed answer (ie, predicting “yes” instead of “no” or vice versa).

A less prominent pattern was observed for the LoRA fine-tuned model. 40% (20/50) of LoRA errors corresponded to predicting “not in the text” despite evidence supporting a “yes” or “no” answer, while 58% (29/50) corresponded to predicting a positive or negative answer when the correct label was “not in the text.” Only 2% (1/50) of errors involved inversion of “yes” and “no” labels.

Errors were unevenly distributed across QoL questions, as shown in Table 6. The fallback question “Did you feel ill or unwell?” was the most error-prone question in the dataset, with 22 errors (26% error rate). The second and third most misclassified questions were “Did you worry?” with 10 errors (12% error rate) and “Have you had pain?” with 9 errors (11% error rate).

This study investigated whether open-source LLMs can be used as a supplementary tool for extracting QoL information from online forum posts, using a breast cancer forum as an example, in a way that aligns with standardized questionnaire responses. It further explored which model and optimization technique may be best-suited for this task, assessed whether LLMs can potentially reproduce human annotations, and investigated possible challenges based on the types of errors occurring for specific QoL items.

Open-source LLMs demonstrated poor ability to identify QoL-relevant information from patient-generated text, achieving macro-averaged F₁-scores between 0.32‐0.56 in a zero-shot setting with added context and 0.36‐0.59 without added context. This suggests that QoL information extraction remains a challenging task for current open-source LLMs without task-specific training. Overall performance depends on the model family and size; careful choice of model may therefore be a crucial starting point in the methodology design. Models from the GPT-OSS and Qwen families demonstrated higher potential for successfully extracting QoL information than other models. Notably, increasing the model size did not result in the highest scores, suggesting that increased parameter count alone does not guarantee better performance for this task. From a practical perspective, the stronger performance of midsized models compared to larger 70B models may be especially advantageous in clinical practice, as these models typically require fewer computational resources and less specialized hardware for deployment. However, inference time and resource consumption were not evaluated in this study and remain important directions for future work.

The analysis of per-class F₁-scores further reveals a strong imbalance in performance across classes, with models achieving substantially higher performance on the majority class “not in the text” compared to the minority classes “yes” and “no,” which highlights a systematic weakness when using zero-shot LLMs for QoL information extraction from health care forum posts. From a clinical safety perspective, the models’ strong performance on the “not in the text” label is beneficial. In health care environments, minimizing false positives prevents the system from hallucinating nonexistent symptoms, which could lead to inappropriate clinical conclusions. However, the primary objective of deploying LLMs in this context is the successful extraction of explicit “yes” and “no” signals. In a real-world clinical monitoring workflow, failing to detect instances where a patient reports severe treatment side effects or acute emotional distress introduces critical false negatives. Missing these signals directly compromises the ability to track a patient’s QoL, making a zero-shot extraction framework insufficient for practical clinical deployment.

Providing additional context (the main post to which the investigated post refers) consistently led to decreased performance, suggesting that additional conversational context may introduce noise rather than clarifying information for QoL information extraction. Confusion matrices for the post-only and post+context settings (Figure 3) show that including context increased the number of misclassifications, particularly by incorrectly assigning “yes” or “no” labels to posts containing no information about the symptoms. This observation was supported by a manual analysis of prediction errors, which showed that approximately 37% of errors in the post+context setting resulted from confusion between the target post and the main post. For example, in 1 instance the main post contained the statement “I've been on CO2 for 2 years & never felt depressed. Suddenly I'm having terrible anxiety attacks. I'm barely eating or sleeping...,” while the corresponding target post consisted of another user’s recommendation to seek therapy for anxiety. For the question “Have you had trouble sleeping?,” the correct label was “not in the text,” as the question referred to the author of the target post rather than the person speaking in the main post. However, when conversational context was provided, the model predicted “yes,” suggesting that it relied on information from the context instead. Adding context should therefore be avoided for this task or the methodology should be carefully designed to prevent confusion, particularly if future studies suggest that removing context results in loss of relevant information.

The evaluation of prompting-based optimization strategies applied to GPT-OSS-20B showed that prompt engineering methods provide only limited improvements over the baseline. CoT prompting resulted in a performance decrease compared to the plain zero-shot baseline, which aligns with findings by Liu et al [49] that CoT can reduce performance on tasks where thinking makes humans worse-here reflected as “overthinking” and assigning QoL questions to indirect symptoms. The qualitative examples in (Table 7) provide evidence for this effect, showing cases where the model shifts from direct symptom interpretation to more speculative reasoning paths, which leads to incorrect final predictions. Instruction optimization using MIPROv2 resulted in a slight performance decrease compared to the baseline in both zero-shot and few-shot settings. Bootstrap few-shot prompting with random search achieved the best results among prompt-based methods; however, the improvement over the baseline remained marginal. A possible explanation for the limited effectiveness of prompt-based optimization techniques is that the task requires fine-grained matching between patient-generated texts and specific QoL questions, while prompt optimization methods mainly improve general instruction-following or reasoning behavior. In particular, methods such as MIPROv2 optimize a single global prompt over the entire dataset, which may not be well suited for this setting due to the high variability across the 53 QoL questions, which differ in the symptoms they target and in how explicitly they are expressed in patient language. As a result, a single optimized prompt may not improve performance consistently across all questions and can even introduce more general or abstract reasoning patterns that are not helpful for identifying symptom mentions. While carefully selected few-shot demonstrations can provide some task-specific guidance, they are still not sufficient to cover the diversity of both patient-generated posts and QoL question formulations.

In contrast, parameter-efficient fine-tuning with LoRA resulted in a substantial improvement in performance compared to all prompt-based methods. This suggests that adapting model parameters is more effective for this task than modifying prompts, likely because the model is able to learn task-specific patterns directly from the training data instead of relying on general instruction templates. In particular, fine-tuning allows the model to better capture subtle differences between similar QoL questions and to adjust to the specific language used in patient-generated text. The improvement is especially visible in the minority class “yes” which indicates that fine-tuning helps the model better identify explicit symptom mentions that were often missed in the zero-shot setting. Despite the overall improvement, the results also suggest several possible directions for improving the fine-tuning setup. In this study, class imbalance was addressed using simple random oversampling, which effectively duplicates the limited number of “yes” and “no” examples. While this helps to balance class frequencies, it does not increase the diversity of training signals and may therefore limit the model’s ability to generalize to new expressions of symptoms. More advanced data augmentation strategies, such as synthetic generation of additional training examples, could potentially provide more diverse and informative training data and further improve performance. In addition, in this study, a single LoRA adapter was trained across all 53 QoL questions. While this approach is computationally efficient, it may limit the model’s ability to specialize for different types of questions, which vary in symptom type and linguistic formulation. Training separate adapters for different subsets of questions, or even per-question adapters, could potentially improve performance by allowing more targeted adaptation. However, such approaches would significantly increase computational cost and were beyond the scope of this study.

The analysis of textual evidence demonstrated that the tested model could successfully justify its predictions with supporting text. In nearly 90% of correctly classified yes and no cases, model-generated evidence overlapped with human-annotated spans, either exactly or partially. This finding is particularly relevant for clinical interpretability but also highlights the potential of LLMs to generate synthetic annotations that align with human annotations, which could reduce manual effort and increase the availability of labeled data for optimization methods that benefit from larger training samples.

Error analysis revealed systematic weaknesses related to mapping free-text symptoms description to discrete questionnaire items. Most misclassifications occurred when the model inferred an answer despite the absence of explicit information in the text, especially for a fallback question “Did you feel ill or unwell?” which exhibited a notably higher error rate (26%) compared to other questions. This is likely related to the semantic role of this label in the annotation scheme, as it functions as a broad catchall category for cases that do not match more specific symptom questions, as well as for general statements of wellness or illness [13]. As such, it often captures semantically underspecified or heterogeneous expressions, which may be inherently more difficult to map to a single EORTC QoL construct. Symptoms such as fever or low blood pressure did not fit into other predefined categories and were therefore annotated with this fallback label. However, for a model performing question-wise prediction, it is not clear whether a more specific matching category exists. Emotional and symptom-related questions with overlapping semantics (eg, pain vs discomfort, fatigue vs need for rest) were particularly prone to error. These findings highlight the challenge of defining clear decision rules for assigning free-text expressions to specific QoL questionnaire items. Even for human annotators, it may be difficult to determine whether indirect expressions (eg, “I constantly needed to rest”) should be interpreted as evidence for the question (“Were you tired?”) or whether only explicit mentions (eg, “I felt very tired”) should be considered correct. Overlapping questions, such as general worry and worry about health in the future, further complicate the task, as answering both questions positively may introduce redundancy, while restricting the answer to only one requires nuanced rules that are not defined in the questionnaire structure. These findings are consistent with Schmidt et al [13], who reported similar difficulties during the annotation process of the dataset used in this study.

A foundational work directly related to this research is that of Schmidt et al [13], which introduced a feasibility study for extracting QoL information from cancer forum posts from Inspire.com and comparing it to patient-reported outcomes on the EORTC QLQ-C30 and EORTC QLQ-BR23 questionnaires. This study demonstrated that QoL information is present in online forum posts and that manual coding can reliably link free-text content with structured questionnaire responses. However, it did not explore the potential of using AI to automate the process.

A systematic review by Lázaro et al [50] synthesized the scientific evidence on the application of NLP techniques as a QoL analysis tool in patients with chronic conditions, including cancer. However, this review covered studies published between 2011 and 2021 and therefore did not include recent advances in NLP which are part of this work. Individual studies mentioned in the review have addressed related but narrower tasks, such as detecting depressive symptoms in online forums (Karmen et al [51]), extracting patient-reported breast cancer symptoms from free-text electronic health record notes (Forsyth et al [21]), and detecting and quantifying pain in radiation oncology office notes of patients with cancer with bone metastases (Naseri et al [52]). While these studies demonstrate the feasibility of extracting QoL-related information from unstructured text, they focused on a specific subset of symptoms and did not attempt to map them to standardized QoL questionnaires.

More recently, researchers have begun to evaluate using LLMs for structured information extraction in clinical context, although with different task scopes and data sources than this study. For example, Garcia-Carmona et al [22] assessed the performance of 6 LLMs (both proprietary and open-source) in a zero-shot setting in extracting patient demographics, diagnostic details, and pharmacological data from unstructured medical reports. Lee et al [23] compared the performance of human reviewers and a locally deployed LLM for extracting key thyroid cancer histologic and staging information from surgical pathology reports. While these studies explore the potential of using LLMs for clinical NLP tasks, they focus on texts authored by medical professionals rather than patient-generated text and do not address QoL information.

A closely related study by Nair et al [24] evaluated multiple LLMs and prompting strategies for summarizing patient-generated content from web-based forums and health communities related to breast cancer. The authors compared zero-shot, few-shot, and CoT prompting results with manual reference summaries. They found that few-shot prompting outperformed other approaches, which aligns with observations from this study. However, summarization tasks differ fundamentally from extracting information to answer standardized questionnaire items.

In contrast to existing work, this study provides a systematic evaluation of open-source LLMs and optimization strategies for extracting questionnaire-aligned QoL information from patient-generated forum posts, using a breast cancer forum as an example. Unlike prior research focusing on different tasks, clinical text, or narrow symptom extraction, this study evaluates prediction accuracy for all symptoms included in EORTC QLQ-C30 and EORTC QLQ-BR23 questionnaires. Additionally, it investigates the potential of using LLMs for recreating human annotations by extracting text spans supporting the prediction. These contributions support the potential use of open-source LLMs in a real-world clinical setting for low-burden, interpretable QoL information extraction from patient-generated text.

This study has several limitations that readers should be aware of when interpreting the results. First, the dataset was derived from a single online breast cancer forum and included a limited number of patients and posts. Additionally, all participants were US residents, as Inspire.com members are primarily US-based. As a result, the findings may not be generalizable to other diseases or overall patient populations, where language use and symptom reporting may differ. Different patient communities may vary in how explicitly or implicitly symptoms are described and in how medical terminology is used, which may affect the mapping between forum posts and EORTC QoL measures. Future studies should assess whether similar performance patterns hold across other diseases, languages, and online platforms. This is especially important for non-English or culturally distinct forums, where symptom expression and linguistic framing may differ substantially and may not align with the assumptions embedded in EORTC-based labeling schemes.

Second, ground-truth labels were derived from manual annotations that required mapping patient-generated forum content to structured questionnaire items. As discussed earlier, some texts are overlapping or ambiguous, which poses difficulties even for human annotators. The mean Fleiss’ kappa for the data used in this study was 0.5, indicating moderate to high inter-annotator agreement. This remaining uncertainty may have affected the quality of the dataset and, consequently, both model performance estimates and error analyses.

Third, the dataset was restricted to posts where the author describes their own condition. Posts referring to other individuals (eg, relatives or friends) were excluded to maintain a clear definition of the target variable (QoL of the post author). While this improves label consistency, it artificially simplifies the task. In real-world applications, systems would need to first identify whether a post refers to the author or to another person before applying QoL classification.

Furthermore, this study focused exclusively on locally hosted, open-source LLMs. While this reflects realistic constraints in clinical settings due to privacy concerns, it limits direct comparison with proprietary models that may achieve different performance levels. In addition, only a subset of prompting and fine-tuning strategies was explored, and hyperparameters were not exhaustively tuned. All optimization experiments were conducted using a single model (GPT-OSS-20B) for controlled comparison, but the findings may not generalize to other families and sizes.

Finally, parameter-efficient fine-tuning was conducted using a single LoRA adapter shared across all 53 QoL questions. This design choice, made for computational efficiency, may limit the ability of the model to specialize across question types.

In conclusion, this study demonstrates that extracting QoL information from patient-generated health care forum posts using open-source LLMs is a challenging task with limited and highly variable performance. Without task-specific training, baseline zero-shot models perform poorly. They consistently struggle to detect explicit symptom information (“yes” and “no” responses), which is the primary objective of this extraction task, and instead largely default to the majority “not in the text” class.

We found that prompt-based optimization strategies provide little to no benefit over zero-shot baselines. Parameter-efficient fine-tuning with LoRA proved significantly more effective, achieving the highest overall accuracy and better adapting to the patient-generated text. However, performance on the minority classes remains limited even after fine-tuning. Because missing an explicit symptom report in a real-world setting means failing to detect potential treatment side effects or emotional distress, automated QoL extraction with open-source LLMs is not feasible for clinical deployment. While fine-tuning offers a promising path forward, future work must focus on ensuring the reliable detection of minority-class signals.