Patient-reported outcome measures (PROMs) are pivotal in clinical research because they capture health status, symptoms, functioning, and quality of life directly from patients. The increasing global reach of clinical trials necessitates that PROMs be accurately translated and culturally adapted for diverse linguistic and cultural contexts. Traditionally, this process has involved a rigorous linguistic validation methodology, typically including forward and backward-translation, cognitive debriefing, and expert review, ensuring conceptual, item, and measurement equivalence across languages [1]. This comprehensive approach, while robust, is resource-intensive and time-consuming, posing challenges for the efficient deployment of PROMs in multinational studies [2,3].

The traditional linguistic validation process, while crucial for ensuring high-quality PROM translations, faces several inherent challenges. These include the significant time and financial resources required to execute rigorous forward-backward translation, cognitive debriefing, and expert reconciliation processes [4]. The complexity of managing these multistep workflows across numerous languages can lead to delays in clinical trial timelines, impacting the overall efficiency and cost-effectiveness of global research [5]. Furthermore, the availability of qualified human translators with expertise in both medical terminology and specific cultural nuances can be limited, particularly for less commonly spoken languages or highly specialized therapeutic areas [6].

The rapid advancements in AI, particularly in natural language processing and, more recently, large language models (LLMs), have introduced new possibilities for automated and semiautomated translation. AI-powered translation services offer the potential to significantly streamline the translation process, reduce costs, and accelerate the availability of PROMs in multiple languages [7,8]. These technologies leverage vast datasets and sophisticated algorithms to generate translations with remarkable fluency and, in many cases, accuracy comparable to that of human translation for general texts [9,10]. The integration of AI tools could potentially alleviate some of the burdens associated with traditional translation workflows, making multilingual electronic clinical outcome assessment deployment more feasible and efficient [11,12]. Recent PROM-specific evidence also suggests that LLMs can produce clinically relevant translation outputs under certain conditions, although performance remains dependent on the instrument, language pair, and evaluation framework [13]. However, recent health care research has emphasized that LLMs should be evaluated and deployed with careful attention to oversight, transparency, and task-specific risk, particularly in clinically consequential settings [14].

Current AI translation tools vary in architecture and controllability. Google Translate, Amazon Translate, and DeepL are widely used neural machine translation (NMT) services, whereas GPT-4.1 represents a newer LLM-based approach that can be guided through explicit prompting [9,10,15-17]. Understanding how these systems compare with industry-standard clinical translations (linguistic validation) is important before they can be used responsibly in sensitive settings such as PROM translation. Recent research suggests that machine translation in the era of LLMs has improved substantially, especially in high-resource languages, while still exhibiting limitations that may remain relevant for specialized applications such as clinical questionnaires [18].

The EuroQol 5-dimension 5-level (EQ-5D-5L) is a widely used generic PROM for assessing health-related quality of life across various populations and diseases [19]. Its standardized, simple, and direct language structure, consisting of 5 dimensions (mobility, self-care, usual activities, pain or discomfort, and anxiety or depression), each with 5 severity levels, makes it seem straightforward to translate. However, ensuring the linguistic integrity and measurement equivalence of its translations is critical for the validity of data collected in global trials [20,21]. Ensuring conceptual equivalence is, therefore, essential to preserve the instrument’s validity in multinational research.

This study aimed to benchmark (via algorithms) the quality and comparability of AI-generated EQ-5D-5L translations against linguistically validated human translations. It was designed as a practical benchmarking study against an established clinical translation standard rather than as a test of full linguistic validation or zero-shot performance on unseen proprietary text.

This study used a comparative benchmarking design to assess the quality and comparability of AI-powered translation outputs of the EQ-5D-5L against official, linguistically validated human translations. The methodology involved translating the standardized English version of the EQ-5D-5L into 5 target high-resource target languages: Danish (DA), Dutch (NL), French (FR), German (DE), and Spanish (ES), using 4 distinct AI translation services. Figure 1 illustrates the comprehensive workflow used in this study, from the initial document translation to the final quality evaluation and analysis.

Four leading AI translation services were selected for this study:

These services represent a diverse set of current AI translation technologies. No equivalent domain-specific prompt, medical glossary, or custom terminology constraint was applied to Google Translate, Amazon Translate, or DeepL. These services were evaluated using their standard API translation behavior, as commercially available to end users at the time of analysis. Because the evaluated systems differ in interface and controllability, input standardization was necessarily imperfect. GPT-4.1 permitted explicit instruction through a system prompt, whereas the NMT services did not provide an equivalent prompt-based mechanism within the workflow used here. Therefore, this study reflects a realistic applied comparison of available translation services rather than a strictly parameter-matched head-to-head experiment.

To address potential model drift and ensure reproducibility, all AI translations were generated within a fixed time window in August 2025. The specific service versions and end points used were as follows:

The access dates reported for API documentation references indicate when the documentation sources were consulted and do not necessarily correspond to the dates on which translation outputs were generated.

The source text consisted of the 43 unique text segments from the standard UK English version of the EQ-5D-5L, comprising the instrument’s 5 dimensions and the EQ visual analog scale instructions. For interpretive transparency, Multimedia Appendix 2 presents annotated screenshots of the corresponding sample UK English digital questionnaire structure from the EuroQol user guide. These text segments were systematically translated into the 5 target languages using each of the 4 AI translation services. Official, linguistically validated human translations for each target language were obtained from the EuroQol Research Foundation and served as the gold standard reference for benchmarking. These translations were developed through a multistep process, including forward-backward translation and cognitive debriefing, and represent the current industry standard for clinical use.

Sentence-level translation was used because the workflow compared each source segment with its aligned gold standard counterpart, and many EQ-5D-5L elements function as discrete response or instruction units rather than cohesive prose. However, this design removed the broader questionnaire context that some systems might otherwise use to improve lexical consistency, disambiguation, and register, and it provided only an approximation of dependence among related instrument segments.

Evaluating the quality of machine-translated PROMs requires robust and reliable metrics. This study uses 4 widely recognized automated machine translation metrics. All scores were computed at the sentence level.

BLEU and METEOR are more sensitive to lexical and word-order variation, whereas COMET and BLEURT are more informative for semantic adequacy. Because COMET and BLEURT were used in a reference-based form against the gold standard translations, they may reward closer alignment with the validated reference wording and penalize clinically acceptable alternative phrasings. Recent work has also aimed to improve the interpretability of neural translation metrics through fine-grained error detection, reinforcing the importance of semantic evaluation approaches that move beyond simple lexical overlap [26].

To aid interpretation, we also summarized the proportion of sentence-level scores falling within descriptive quality bands. Because no universally accepted pass or fail thresholds exist for applying these automated metrics to PROM translation benchmarking, the bands were used only as pragmatic interpretive aids for summarizing score distributions. For BLEU and METEOR, which are more sensitive to lexical and word-order overlap, scores ≥0.50 were categorized as high, scores from 0.30 to <0.50 as mid, and scores <0.30 as low. For COMET and BLEURT, which were used as semantic similarity metrics, scores ≥0.80 were categorized as high, scores from 0.60 to <0.80 as mid, and scores <0.60 as low. These bands were not treated as formally validated pass or fail criteria for PROM translation quality. These metrics provide objective, quantitative measures of translation quality that are crucial for benchmarking different AI services.

This study was a methodological benchmarking analysis and did not involve human participants, patient-level data, or identifiable personal information. The materials analyzed consisted of EQ-5D-5L source-text segments and official, linguistically validated translations used as benchmark references for translation-quality evaluation. Because no human-participant research procedures were conducted and no identifiable human data were analyzed, institutional review board review was not required. Informed consent was not applicable. No compensation was provided. No identifying images of participants or users were included in the manuscript or supplementary materials.

The results are presented in 2 complementary parts. First, inferential analyses were used to test whether the 4 AI translation services differed from one another within each metric-language combination. Second, descriptive summaries and visual analyses were used to assess the distribution of translation quality scores relative to the validated human translations and to identify localized sentence-level hotspots.

Translation quality scores from the 4 AI services (Google, GPT-4.1, Amazon, and DeepL) were compared across all 20 metric-language combinations. The Friedman tests revealed statistically significant differences in 11 of the 20 (55%) combinations, indicating that the performance distributions of the services were not identical.

To determine which specific services differed, paired Wilcoxon signed-rank post hoc tests with a Holm-Bonferroni correction were performed on the 11 significant Friedman test results. This analysis confirmed 11 statistically significant pairwise differences, with adjusted P values ranging from <.001 to .049. This is the central finding: the AI services are not functionally interchangeable, and statistically significant performance differences depended on both the target language and the evaluation metric.

The significant differences clustered around specific metrics (Table 1). Of the 11 significant pairwise differences, 10 (91%) were identified by the surface-overlap metrics (BLEU and METEOR), which measure stylistic and lexical similarity. In contrast, the semantic metrics (COMET and BLEURT), which measure meaning, together only accounted for 1 of the 11 differences. Taken together, these findings suggest that the services differ more in stylistic realization than in meaning preservation. To visualize this semantic comparability directly, Figure 2 presents COMET score distributions, as COMET is a reference-based semantic metric with a strong reported correlation with human judgments. Individual score distributions for each target language are detailed in Multimedia Appendix 3.

The post hoc summary shows Holm-Bonferroni–adjusted P values and pairwise effect size r from paired Wilcoxon signed-rank tests for significant pairwise comparisons only. Friedman P values and post hoc adjusted P values address different levels of comparison and are, therefore, not directly comparable: the Friedman test evaluates an omnibus difference across all 4 services, whereas each Wilcoxon signed-rank test evaluates a specific service pair.

This interpretation was supported by the effect size analysis. Omnibus effect sizes were generally small, with a median Kendall W of 0.08 (IQR 0.07-0.10) across the 11 significant tests, indicating limited overall separation among all 4 AI translation services within a metric-language combination. In contrast, some post hoc pairwise effect sizes were moderate to large, showing that specific service pairs could differ meaningfully even when the overall omnibus separation remained modest.

For example, in the BLEU score for Spanish, Google (median 0.325, IQR 0.199-0.632) performed significantly worse than Amazon (median 0.425, IQR 0.230-1.000; adjusted P=.001; r=0.75). Conversely, for the semantic metric COMET in Dutch, the Friedman test indicated an overall difference among services, but no pairwise comparison remained significant after Holm-Bonferroni correction. This finding reinforces the interpretation that semantic metric differences were limited and that most significant pairwise differences were concentrated in surface-overlap metrics.

Threshold-based descriptive summaries further supported this interpretation. Across languages and services, most COMET and BLEURT scores fell within ranges consistent with high semantic similarity to the validated reference translations, whereas BLEU and METEOR showed a greater proportion of lower scores because of their sensitivity to lexical and word-order variation. Detailed threshold-based summaries by metric, language, and AI translation service are provided in Table S1 in Multimedia Appendix 3.

To supplement the aggregate analyses, we conducted a qualitative outlier analysis by benchmarking AI outputs directly against the official, linguistically validated human translations (gold standard). Outliers were defined as sentence-level scores at or below the 5th percentile for each metric. This analysis was intended to localize failure modes rather than determine whether translation services were globally acceptable or unacceptable.

As a descriptive count-based sensitivity analysis, BLEU was the most discriminative metric for flagging potential low-scoring segments (158 instances), followed by METEOR (73), BLEURT (60), and COMET (60). Outlier counts were then examined across services. The distribution was relatively even, ranging from 85 flagged instances for DeepL to 92 for Amazon Translate, suggesting that all services showed a similar frequency of localized deviations from the gold standard.

Outliers were not randomly distributed but clustered within specific hotspots (Figure 3). Language-specific heatmaps are provided in Multimedia Appendix 3, whereas Multimedia Appendix 2 provides a sentence-number mapping guide for the 43 source segments. Because the EQ-5D-5L source instrument is proprietary, this mapping is presented as annotated screenshots from the sample UK English digital questionnaire in the EuroQol user guide rather than as a reprinted text table.

Some of these hotspots corresponded to short digital interface elements rather than clinical questionnaire constructs. In particular, the final source segments included navigation and interface strings such as “Previous,” “Next,” and an error message. These elements are context-dependent and may be difficult to translate optimally when evaluated as isolated sentence-level strings outside their software environment. Therefore, low scores for these segments should be interpreted separately from deviations involving clinical concepts, domain headers, or response options.

BLEU identified the largest hotspot clusters in Spanish and Dutch, indicating where review effort may need to be concentrated for specific language-service combinations. However, because hotspot counts alone do not establish overall adequacy, overall translation quality was interpreted based on the full score distributions and semantic metric patterns, and heatmaps were used specifically to localize low-scoring deviations.

To illustrate the clinical implications of these findings, we conducted a qualitative review of both high-scoring and low-scoring sentences (Table 2). For simple declarative sentences, the AI models frequently achieved perfect alignment with the human reference translations, producing identical, verbatim matches across all 4 services (eg, the German translation for pain severity). This confirms that, for standard grammatical structures and unambiguous phrasing, AI performance was highly reliable and consistent.

However, for section headers, abstract health concepts, and short context-dependent interface strings, some differences from the gold standard emerged. As detailed in Table 2, AI services sometimes defaulted to literal or highly medicalized terminology rather than to the intended clinical concept. This pattern was observed across languages; for instance, in Danish, models frequently selected terms that were technically correct but contextually distinct (eg, translating “Mobility” as the sociological concept Mobilitet rather than the physical capacity Bevægelighed used in the validated version). These examples illustrate a possible limitation of current AI models in distinguishing literal translation from conceptual equivalence. The practical relevance of isolated header differences could not be assessed in the present study. Further equivalence or cognitive-debriefing research will be needed to determine which discrepancies are acceptable and which require the involvement of expert human linguists. Therefore, Table 2 presents representative examples of both high-scoring exact matches and low-scoring conceptual deviations, while the heatmap and sentence-number mapping identify additional hotspots, including interface-related strings.

This study reveals that although AI translation services are not statistically identical, most between-service differences were stylistic rather than semantic. We identified 11 statistically significant pairwise differences (adjusted P<.05, ranging from <.001 to .049), of which 91% (10 of 11) were detected by surface-overlap metrics (BLEU/METEOR), whereas only 1 was detected by a semantic metric (BLEURT). Supported by the generally small omnibus effect sizes (median Kendall W=0.08, IQR 0.07-0.10), this pattern suggests that although services differed in lexical choice, they produced translations with highly comparable semantic meaning relative to the validated human translations. Taken together, the semantic score distributions and the localized nature of the low-scoring hotspots suggest that current AI services can generate translations for this instrument in the evaluated languages. However, this interpretation depends on the combined descriptive, semantic, and hotspot analyses rather than on significance testing alone. Some differences occurred in parts of the questionnaire outside the scored response options; however, their practical relevance for participant interpretation could not be assessed in the present study. Further research will be needed to determine which PROM elements (ie, instructions and answer options) should be focused on when evaluating translation quality.

The finding of statistical comparability of average scores must be interpreted with caution, as aggregate metrics can mask critical, low-incidence discrepancies. Our qualitative outlier analysis revealed that deviations from the gold standard were evenly distributed across all services (85‐92 outliers per service). The lowest-scoring deviations were primarily observed in domain headers, and the 4 representative low-scoring examples in Table 2 came from different AI services. These findings suggest that human oversight may be required across services, although further research is needed to determine the level of review required to ensure acceptable translation quality.

The potential clinical significance of these deviations was evaluated through a manual review of systemic inconsistencies against the official, linguistically validated translations. As detailed in Table 2, semantic inconsistencies occurred primarily in abstract concepts and domain headers rather than in the actual response options themselves. For instance, the domain header concept “Self-care” was frequently translated literally (eg, Soins auto-administrés or “self-administered medical care”) rather than conceptually (referring to personal autonomy in washing or dressing). Although the subsequent item text (“I have no problems washing or dressing”) was often translated correctly, the practical relevance of isolated header differences could not be assessed in the present study. This interpretation is consistent with electronic clinical outcome assessment migration literature, which distinguishes core respondent-facing content, such as instructions, item wording, and response options, from other contextual or presentation elements when considering the level of equivalence evidence required [2]. A follow-up study is planned to quantify the reduction in effort that AI can offer compared with fully manual translation.

We acknowledge several limitations. First, cultural adaptation was assessed only indirectly, as no patient cognitive debriefing was performed. Second, the study used sentence-level translation and evaluation to support aligned benchmarking against the gold standard, which may have constrained systems that benefit from broader document context and may not have fully captured dependence among related instrument segments. Third, the findings are limited to high-resource European languages and should not be generalized to lower-resource or structurally distinct languages without further validation. Fourth, the primary semantic metrics were reference-based and may therefore have penalized acceptable alternative phrasings that differed from the validated reference wording. Fifth, the service configuration was not fully symmetrical because GPT-4.1 was used with a structured medical translation prompt, whereas Google Translate, Amazon Translate, and DeepL were evaluated in their standard API mode. Sixth, because the EQ-5D-5L and its validated translations are widely used, some systems may have been exposed to similar wording during training or optimization. Finally, the qualitative error categories were interpretive and descriptive rather than based on a formally validated annotation framework, and no blinded, independent, multirater review or formal interrater reliability testing was performed.

Future work will extend this benchmarking approach in several directions. Automated metric results should first be compared directly with expert human judgments to determine how well these scores reflect professional linguistic review of PROM translations. Studies should also quantify the operational value of human-in-the-loop workflows by comparing fully manual translation with AI-assisted translation followed by expert editing. Broader evaluation designs should examine whether performance changes when related questionnaire elements are analyzed at the domain level rather than strictly at the sentence level, and whether these findings generalize to structurally different or lower-resource languages. Finally, future studies should explore hybrid and reference-free evaluation strategies, including quality estimation methods, and assess how such approaches could be integrated into clinical trial translation workflows more effectively.

This benchmarking study showed that leading AI translation services were not functionally interchangeable across metric-language combinations. Although statistically significant differences were identified, these were more often detected by surface-overlap metrics than by semantic metrics. These findings support the use of AI to generate PROM translations for this instrument and the evaluated high-resource European languages. Expert human linguist review may need to be retained to confirm conceptual equivalence and fitness for clinical use.