This study uses MedWeb, a multilingual pseudo–social media text dataset with multiple symptom labels. The dataset was originally created in Japanese and then human-translated into other languages. The translation-based dataset provides a controlled setting for consistent cross-linguistic comparison of symptom expression. It covers 12 languages, predominantly languages from Southeast Asia (Indonesian, Filipino, Malay, Thai, Khmer, Burmese, and Lao), together with Japanese, English, German, French, and Arabic. Each language contains 640 texts, which are labeled as positive or negative for each symptom or disease (hereafter referred to simply as symptom), allowing multiple positive labels per text [20].

In this study, the languages were categorized by region and resource classification, as shown in Table 1 [6]. The language resource classification was included because LLM performance in NLP tasks is partly influenced by the availability of training data. Therefore, we also investigated whether performance on this symptom detection task differs across languages with varying resource levels.

The study focuses on 8 common symptoms, which are considered indicators associated with other diseases: fever, headache, runny nose, cough, diarrhea, hay fever, influenza, and cold. The label distribution in this dataset is imbalanced between positive and negative labels, with the proportion of positive labels varying across symptoms, as presented in Table 2.

We used the GPT-5 (OpenAI) model as a tool for this study. A previous work showed that models from the OpenAI family consistently outperformed others across both large-parameter and small-parameter settings [13]. In addition, GPT-5 is the strongest OpenAI model for health-related questions, achieving higher scores on HealthBench than previous models evaluated in that benchmark [21]. For the prompting strategy, we implemented a rule-based, zero-shot setting in English to reflect a general and realistic deployment scenario. The rules incorporated into the prompt were aligned with the annotation guidelines used during dataset labeling, as shown in Textbox 1.

We also conducted an ablation study to support the selection of the LLM and the prompting strategy used in the main experiments. For model selection, we evaluated 1 proprietary LLM (GPT-5) and 3 groups of open-weight LLMs: (1) general-purpose models, including Gemma 3 4B (Google DeepMind), Qwen3-VL 8B (Qwen team, Alibaba Cloud), and Llama 3.1 8B (Meta); (2) models fine-tuned for Southeast Asian (SEA) languages, including Gemma-SEA-LION-v4-4B-VL (AI Singapore), Qwen-SEA-LION-v4-8B-VL (AI Singapore), and Llama-SEA-LION-v3-8B (AI Singapore); and (3) models fine-tuned for the medical context, including MedGemma 4B, Bio-Medical-Llama-3-8B, and HuatuoGPT-o1-7B. These models were selected to examine multilingual performance patterns across different model architectures and families and to identify the model with the most suitable performance for further analysis in this study.

In addition to open-weight LLMs, we included supervised models trained on the study dataset and a multilingual zero-shot architecture that was not fine-tuned on the dataset as baselines. These models represented both general-purpose multilingual and medical-domain multilingual approaches across different model paradigms. We further conducted a prompting-strategy ablation to examine the effect of alternative prompt designs. The evaluated strategies included five prompts: (1) rule-based zero-shot setting in English, (2) rule-based zero-shot prompting in each language, (3) rule-based few-shot prompting in English, (4) basic zero-shot prompting in English, and (5) basic zero-shot prompting in each language. This analysis enabled us to assess the contribution of prompt language and structure to model performance.

We further conducted a prompting-strategy ablation to examine the effect of alternative prompt designs. The evaluated strategies included five prompts: (1) rule-based zero-shot setting in English, (2) rule-based zero-shot prompting in each language, (3) rule-based few-shot prompting in English, (4) basic zero-shot prompting in English, and (5) basic zero-shot prompting in each language. This analysis enabled us to assess the contribution of prompt language and structure to model performance.

The complete experimental configurations, including the corresponding results for the evaluated models and prompting strategies, are provided in Multimedia Appendix 1.

In this study, we analyzed the results from 2 perspectives: a language-based and a symptom-based analysis. Both perspectives used macro precision, recall, and F₁-score. In the symptom-based analysis, the macro F₁-score assigns equal importance to each symptom label, regardless of whether the label is frequent or rare, by calculating the F₁-score for each label separately and then averaging them [22]. This metric provides a clearer picture of the model’s consistency and enables analysis of performance variation across symptoms. Meanwhile, recall measures how many true positive symptom labels the model successfully captures on average for each symptom. In the context of surveillance, recall is important because low recall indicates that the system is missing many true symptom signals. Precision, on the other hand, measures the proportion of predicted positive symptom labels that are correct. In a surveillance system, this is useful for identifying which symptoms generate more false positives, potentially leading to false alarms. This study also conducted an error analysis to identify the mechanisms underlying incorrect symptom predictions across languages. An error tweet was defined as a tweet for which (1) at least one language version contained an incorrect prediction and (2) the prediction patterns were not identical across the language versions. Based on the identified error tweets, we first performed a manual qualitative review to identify recurring error patterns and grouped them into several categories. We then defined each error type and prepared representative examples. Using these definitions and examples, an LLM (GPT-5-mini) was asked to classify all identified error tweets into the corresponding error categories and assign a primary error category to each tweet. Approximately 5% of the LLM-generated classifications were subsequently manually reviewed by a human for quality control. The counterfactual impact analysis was then done by correcting the incorrect predictions in each error tweet based on its primary error category and recalculating the F₁-score. The macro F₁-score was recalculated on the complete multilingual dataset (7680 language-specific texts), with F₁-score calculated for each symptom across all languages and then macroaveraged across the 8 symptom labels. The residual impact of each error type (ΔF₁) was then measured as the difference between the corrected and original F₁-scores (baseline). A smaller ΔF₁ means that the error category is causing less remaining performance loss under that prompt.

This study involved no physical or mental interventions and did not include any experiments requiring human participation. In accordance with the Ethical Guidelines for Medical and Biological Research Involving Human Subjects established by the Japanese government, this study did not require institutional review board approval, as no personally identifiable information was used [23]. The dataset used in this study is publicly available as the extended MedWeb corpus. The original study about the dataset also states that it does not contain personally identifiable information and is exempt from institutional review board approval [20]. Therefore, this study raises no ethical concerns with respect to user privacy or informed consent.

Before presenting the main findings, we conducted ablation studies to select the model and prompting strategy. As shown in Figure S1 in Multimedia Appendix 1, among the evaluated LLMs, GPT-5 achieved the highest overall performance with relatively low disparities across languages. In Figure S2 in Multimedia Appendix 1, among prompting strategies, rule-based zero-shot prompting in English showed the lowest performance variability across languages, although its average performance was lower than that of some alternative prompting strategies. Based on these findings, we selected GPT-5 with rule-based zero-shot prompting in English for the main analysis to ensure high and consistent performance across languages.

This section presents the evaluation results of LLM-based multilabel symptom detection across 12 languages, with the aim of examining performance disparities among the languages. The results were analyzed from 2 perspectives: language-level performance and symptom-level performance.

The results of the language-based analysis, as presented in Table 3, showed a noticeable performance gap between SEA languages and the other languages. Overall, SEA languages tend to achieve lower scores across macro F₁-score, precision, and recall than non-SEA languages, with Thai being the only exception. Thai achieved the strongest performance among SEA languages, with scores approaching those of other high-resource languages such as Japanese, English, and French. Among SEA languages, Lao had the lowest F₁-score (0.716) and the widest gap between precision (0.820) and recall (0.658). This indicates that although the model’s predictions are relatively precise, it fails to identify many actual positive cases in the Lao dataset.

This finding is consistent with the ablation study in Multimedia Appendix 1. Figure S1 in Multimedia Appendix 1 generally shows lower performance for SEA languages than for non-SEA languages across LLMs, except for the supervised models. Moreover, this performance gap persists even in models specifically fine-tuned for SEA languages.

To further investigate the disparities, we analyze the performance differences across language resource classification groups, as shown in Figure 2. High-resource languages achieved the highest and the most consistent performance, with Japanese achieving the highest score and appearing as an outlier. The midresource group exhibited greater variability, with Thai standing out as a high-performing outlier, as its performance is closest to that of the high-resource group, while the remaining languages within the group show relatively similar performance levels. Low-resource languages showed the lowest performance among all groups, with Lao obtaining the lowest F₁-score among languages. These results indicate performance disparities across the resource language classification.

These results align with findings from our model ablation study in Figure S3 in Multimedia Appendix 1. High-resource languages generally achieve better performance than low-resource languages across the evaluated LLMs, although the ranking of individual languages varies by model category. Japanese achieves the highest performance on GPT-5, while English consistently performs best among high-resource languages across the 3 open-weight LLM groups and the zero-shot pretrained classifier.

Furthermore, we analyzed symptom-level data across languages. Figure 3 shows that detection performance varies substantially across symptoms, with average macro F₁-scores ranging from 0.560 (SD 0.046) for runny nose to 0.896 (SD 0.017) for diarrhea. This suggests that the model’s ability to detect symptoms differs across symptom types. Beyond overall symptom-level differences, we further examine how performances vary across languages within each symptom.

Specifically, there is a notable variation in performance across languages for several symptoms. Diarrhea, headache, cough, and fever showed relatively stable performance across languages. However, cold, runny nose, influenza, and hay fever exhibit large performance variations. Most notably, hay fever detection reveals 2 distinct performance clusters. The first cluster shows high scores, with an average macro F₁-score of 0.865 (SD 0.024). The second cluster exhibits lower and more varied performance, with an average macro F₁-score of 0.607 (SD 0.105). All languages in the low-performance cluster are SEA languages, and these include midresource and low-resource languages. In contrast, the high-performance cluster consists predominantly of high-resource languages, with Thai being the only SEA language in this group. Despite this variation, a consistent pattern emerges that SEA languages classified as midresource to low-resource tend to be placed lower. This indicates a performance gap between languages for some symptoms.

However, an interesting pattern emerged from the experiments with different prompting strategies, as shown in Figure S4 in Multimedia Appendix 1. Few-shot prompting appears to improve the model’s understanding of certain symptoms, particularly hay fever, fever, and runny nose, while no significant differences are observed for the other symptoms. Compared with the basic prompt, however, some symptoms show decreased performance, including diarrhea, hay fever, cough, fever, influenza, and runny nose.

Furthermore, we analyzed precision and recall at the symptom level to understand how the balance between these 2 metrics shapes implications for global disease surveillance systems, particularly for potential outbreak detection. Table 4 showed that several symptoms exhibit an imbalance between their recall and precision scores, with most symptoms having lower recall compared to their precision. Runny nose demonstrated the greatest disparity between the 2, with recall considerably low at 0.427 and precision relatively high at 0.858. Meanwhile, influenza was the only symptom where precision scored lower than recall, indicating that the model tends to overpredict influenza.

Since the primary focus of this study was to analyze disparities in symptom detection performance, we further examined hay fever as a representative symptom, as its detection performance appeared to vary substantially across languages. Hay fever formed 2 distinct clusters of high-performing and low-performing languages, with most SEA languages grouped in the lower-performing cluster. This pattern suggests that some languages support more accurate detection of hay fever, while others may be less capable of capturing hay fever–related terminology or symptom expressions.

This finding emerges alongside regional differences in hay fever–related research and reported cases. Hay fever or allergic rhinitis has been more frequently reported and discussed in regions such as Europe, North America, the Mediterranean, and Japan, while related studies and reported cases remain limited in Southeast Asia [32-36]. As a result, hay fever may be more clearly defined and more frequently referenced in the languages of those regions compared with those used in Southeast Asia. Meanwhile, in LLM-based systems, LLM training data likely reflect the availability of published literature and web content. Therefore, symptoms that are underrepresented in certain language text corpora will be less recognized by the model, causing those symptoms to be more difficult to predict consistently.

This interpretation is also consistent with the ablation study, where providing the model with examples of hay fever cases through few-shot prompting substantially improved its detection performance. This suggests that increasing the model’s familiarity with symptom-related expressions can improve its ability to recognize hay fever across languages. Therefore, the lower performance observed in some languages could be associated with limited exposure to hay fever–related terminology and expressions.

This study has important implications for the use of LLMs in public health surveillance. Symptom detection is a crucial step in digital surveillance systems, as it enables the identification of health-related signals. However, in global applications involving multiple languages, language-related performance disparities may prevent the model from capturing signals consistently across populations. For example, our findings showed that the model achieved low recall for Lao, indicating that many actual symptom labels in the Lao dataset were missed. As a result, the symptoms from that population could be underdetected. This could lead to certain populations being underrepresented in monitoring and analysis, particularly among populations that are vulnerable to emerging infectious diseases.

In addition to language disparities, performance variation was also observed across symptom types. Large performance differences across symptoms may lead to false alarms or missed signals in surveillance systems. For example, based on our findings, runny nose showed the largest gap between precision and recall, with relatively low recall, suggesting that the system may fail to capture many true runny nose signals. In contrast, influenza was the only symptom for which precision was lower than recall, indicating a greater tendency toward false positive predictions. This may result in influenza-related signals being overdetected in regions without actual outbreaks, potentially leading to unnecessary public concern and inefficient allocation of public health resources.

Furthermore, our findings suggest that the model tends to rely on lexical cues in symptom-related text, while still requiring contextual understanding of diseases and the ability to handle diverse symptom expressions. Therefore, LLM-based symptom-detection systems used in public health surveillance should account for these limitations to ensure consistent and reliable performance. This is particularly important when deploying such systems across diverse linguistic and epidemiological contexts.

This study has several limitations. First, although many symptoms commonly experienced by individuals may serve as signals of other diseases, this study included only 8 symptoms and diseases as representatives. Second, this study uses a translation-based multilingual dataset, which may not fully reflect natural language use in each target language, even though the translations were performed by native speakers. Third, this study uses a limited set of languages and includes Southeast Asia as a representative region that is vulnerable to emerging infectious diseases. However, we acknowledge that other regions may also serve as potential origin points for future outbreaks. Expanding the analysis to a broader range of geographical regions would provide a more comprehensive understanding of global disparities. Fourth, the error analysis is based on qualitative methods in the initial observations, which may not capture all possible sources of model failure. Further research should address these limitations to provide a more comprehensive understanding of potential disparities in symptom detection performance for public health surveillance systems.

This study demonstrates the presence of multilingual performance disparities in LLM-based symptom detection and offers insights into developing a reliable global disease surveillance system that operates across diverse languages. We identified 3 main points from the study’s findings. First, detection performance tends to vary with language resource level. The observed performance gap between high-resource and low-resource languages, which could be associated with differences in language resource availability, could contribute to underdetection of symptoms in linguistically underrepresented populations, particularly those who are vulnerable to emerging infectious diseases. Second, not all symptoms are equally detectable across languages. While some symptoms are reliably detected across languages, others remain inconsistently identified due to linguistic variability, symptom overlap, and training data familiarity with the symptoms. Third, the error patterns suggest limitations in the model’s ability to detect symptoms from social media text. The current model’s tendency to rely on lexical cues, along with limited contextual and disease-level understanding, represents a challenge for the LLM-based symptom detection systems to identify consistently across diverse languages and symptom expressions. Few-shot prompting may improve the recognition of implicit symptom expressions; however, prompting alone does not fully address broader symptom understanding or contextual interpretation. Therefore, to achieve more reliable and equitable LLM-based symptom detection from social media text for global disease surveillance, it would benefit from broader representation of training data for low-resource languages, improved cultural-linguistic sensitivity, and stronger contextual understanding of symptom-related expressions.