The ethics committee of the Medical Faculty of Leipzig University (231/24-ek) approved the procedure on July 29, 2024. Due to the retrospective collection of patient data, no informed consent for publication was obtained; therefore, written informed consent was not required. Anonymized prompts were used to ensure patient privacy.

In 2024, we used GeriDoc from the Geriatrics in Bavaria database to collect retrospective patient data from a geriatric hospital.

We included data from geriatric patients in the rehabilitation ward who were hospitalized in 2023.

One hundred patients who meet the inclusion criteria must be included. An additional 10 patients were used for the pretesting of the AQUOS, and these patients were excluded from the main analysis.

Eleven randomly selected patients per month from January 2023 to October 2023 were chosen for this study. To be included, renal function (glomerular filtration rate [GFR]) and polymedication (at least 5 drugs) at discharge had to be documented. In this case, polymedication means the patient must take at least 5 drugs; for example, a combined preparation of 2 drugs counts as 2, not 1.

In this study, we compared 4 different AI chatbots and designed structured inputs or prompts. After receiving ethical approval, we generated the AI outputs from October 18 to October 30, 2024, in Leipzig, Germany. The GFR was categorized into the typical 5 stages of renal disease: category 1 as normal renal function, category 2 as slightly reduced, category 3 as moderately reduced, category 4 as severely reduced, and category 5 as renal failure. The inputs were designed in both German and English, and outputs were evaluated at 2 time points: t0 and t1 (8 d later) to test reproducibility (not the learning effect after updates). Thus, t1 serves as the control for t0. We used an AQUOS (Multimedia Appendix 1) for evaluation. Additionally, discharge medications were categorized by complexity: low (5‐9 drugs), medium (10-14 drugs), and high (≥15 drugs). In this study, medication complexity is defined exclusively as complexity based on the number of medications and is divided into 3 categories (low, medium, and high). This measure reflects structural complexity based on the number of prescribed drugs and does not account for pharmacological risk, therapeutic drug classification, or potential drug-drug interactions.

We created a prompt based on prompt engineering, assigning roles to both the AI chatbot and the requester. We specified the renal function as GFR and listed the patient’s discharge medications and dosages. Following this, we instructed the chatbot to provide a precise answer if a dosage adjustment was necessary and to cite sources as a concrete task using both German and English prompts. The following prompting structure was developed based on literature [12,13] by the 2 authors, CB and TB, who are both pharmacists. We adjusted the prompt based on the quality of the AI chatbot’s responses. The aim was to frame the query as a standardized zero-shot prompt without any follow-up prompts. Neither patients nor members of the public were involved in the development process.

An example prompt in English:

The prompt was deliberately restricted to GFR, drug name, dose, and frequency. This standardized, minimal input format was chosen to ensure comparability across all 100 patient cases and all 4 AI chatbots and to reflect a realistic scenario of brief, point-of-care queries as they might occur in routine clinical practice. We acknowledge that clinically valid renal dosage adjustment may additionally depend on variables such as the indication for each drug, route of administration, treatment duration, dialysis status, body weight, or the differentiation of acute and chronic renal impairment. While more complex prompt formats incorporating additional clinical variables could yield more nuanced AI outputs, such designs would compromise standardization and cross-case comparability, which was an essential aspect of this study. The omission of these variables limits the clinical interpretability of the findings and should be considered when applying results to real-world settings.

If multiple GFR values were included in the discharge letter, the median GFR was used. The drug dosing was provided as in the example, along with the drug name and dosage frequency. We extracted chatbot outputs from the platform and saved them locally, starting a new conversation for each prompt while ensuring previous conversations were cleared. We configured the software to prevent it from “remembering” previous conversations. If a network error occurred, the output was regenerated. After generating the outputs, we verified that the companies had made no relevant updates to the AI chatbots.

Considering that the prompts were entered in German and English at 2 different times (t0 and t1) in 4 different AI chatbots each, 16 outputs were generated per patient.

When collecting the AI chatbots, we focused on large, well-known AI chatbots. Moreover, appropriate settings had to be available to prevent entered data from being used for future AI model updates.

In terms of terminology, “AI chatbot” refers to the conversational interface accessed by users, “LLM” refers to the underlying language model architecture, and “model” is used as a general term for naming the different versions.

We used OpenAI GPT-4 (gpt-4o-2024-11-20, knowledge cutoff at October 01, 2023), Microsoft Copilot Business, Google Gemini 1.5 Flash (gemini-1.5-flash-002), and Research Solutions Scite, all of which are closed sources [15,16]. The exact models of the AI chatbots were not consistently accessible at the time of data collection, but we verified post hoc that no relevant model updates were released by the respective providers during the data collection window. We have accessed the AI chatbots using the web interface of each provider. Furthermore, the standard settings (eg, temperature) were applied to all AI chatbots, which were used as base models as provided by their respective companies. For each patient, a new conversation was initiated by entering the prompt and then saving the AI chatbot’s response locally, and the conversation was subsequently deleted before the next case was processed. With this procedure, no carry-over effects between patient cases could occur.

Based on previous research [10,17,18], we implemented an AQUOS (Multimedia Appendix 1) to evaluate the outputs of the AI chatbots. The score consists of 9 items. The first 5 items, rated from 0 to 4, focus on completeness, referencing, reference suitability, correctness of dosage recommendations, and dosing accuracy. The sixth item, rated 0 or 1 point, checks for disclaimers and references to health care professionals, patient monitoring, and the individual patient case. The last 3 items allow point deductions for unnecessary additional information, incorrect use of medical terms, and inappropriate language or phrasing, with greater issues leading to greater point reductions. Thus, the highest achievable score was 21, which equals 100%.

As a reference standard for evaluating the correctness of the AI chatbot outputs, the German database “Dosing” [19], combined with the corresponding summary of product characteristics, was used. In cases of discrepancies, the more recent source was given preference. If discrepancies persisted, clinical guidelines, original publications, or information from the European Medicines Agency were consulted. In Multimedia Appendix 2, some higher quality and poorer examples of AI chatbot outputs are provided.

In addition to AQUOS, the possible harm the chatbot could have caused with its response was assessed. The outputs were evaluated to determine whether the specific output could cause harm if provided to a physician as pharmaceutical advice. The possible harm was ranked using the Conceptual Framework for the International Classification for Patient Safety by the WHO Patient Safety [20]. The ranking of the harm ranges from “none” to “death,” according to the WHO categories: none, mild, moderate, severe, and death (0-5). Since the scaling differs from AQUOS, possible harm was considered separately.

The validation of the AQUOS scoring system was conducted in 2 sequential phases, following the principles of internal and external validation as commonly applied in laboratory and clinical assay development.

This validation design used in this study broadly follows the principles used in the development of clinical scoring systems, where initial reliability testing is followed by a comparison with expert consensus, including an interrater reliability and intraclass correlation coefficient [21-25]. The detailed validation procedure is provided in Multimedia Appendix 3.

We analyzed the points achieved in AQUOS to evaluate the quality of the AI chatbot outputs. Medication count–based complexity (categorized into 3 groups depending on the number of drugs, not the type of drugs or, eg, possible drug-drug interactions) and GFR (categorized according to the typical 5 stages, ranging from normal renal function to renal failure [26]) were analyzed in correlation with the score. Additionally, we investigated whether the AQUOS differs between English and German and whether the AI chatbot answers are reproducible over time (t0 and t1).

To analyze the data, we conducted descriptive analyses (mean, median, relative difference, and correlations) and a paired, 2-tailed t test to assess statistical significance, with P<.05 considered statistically significant. To evaluate the first validation phase, we performed Cohen κ to examine whether the score was objective and to continue with just 1 rater in the study. For the second validation phase of AQUOS, we conducted an intraclass correlation coefficient analysis among the expert opinions, and AQUOS was then correlated with the median of the expert ratings using the Spearman’s correlation.

The statistical analysis was performed using IBM SPSS Statistics version 29 and Excel version 2408 from Microsoft 365.

The main analysis included 100 geriatric patients, while the pilot study involved 10 geriatric patients. The mean (SD) number of discharge medications was 11.4 (4.2) for the primary group and 11.1 (2.8) for the pilot group. Patients’ characteristics and the medication count from the main analysis are summarized in Table 1.

In total, we generated 1600 outputs using the AI chatbots. More precisely, this means that 16 outputs were generated per patient using 4 chatbots (ChatGPT [OpenAI], Copilot [Microsoft], Gemini [Google], and Scite [Research Solutions]), each at 2 different times for the investigation of reproducibility (t0 and t1) and in 2 different languages (German and English). Multimedia Appendix 2 provides examples of how the AI chatbots’ outputs were evaluated. A study flow diagram illustrating patient selection, the pilot sample, the derivation of the final 100 included patient cases, and the generation of 1600 AI chatbot outputs is provided in Figure 1.

As a result of the first validation phase, the score was considered objective with a Cohen κ of 0.971, so there was 1 rater, a pharmacist, responsible for scoring. No patients or members of the public were included in the scoring process. For the second score validation phase, the intraclass correlation coefficient of 0.906 (95% CI 0.795‐0.974; P<.001) shows excellent agreement between the raters of the expert panel. In addition, the Spearman correlation, with a Spearman ρ of 0.650 (95% CI 0.012‐0.912; P=.04) between AQUOS and the median of the expert panel, validates AQUOS in an external validation conducted by experts.

There were 5 patients in the normal renal function category, 62 in category 2 (slightly reduced), 25 in category 3 (moderately reduced), 8 in category 4 (severely reduced), and no patients in category 5 (renal failure) based on GFR classification.

The trends of overall AQUOS in German and English, as shown in Figure 2 and Multimedia Appendix 4, decline with worsening renal function (ChatGPT: –0.215, P=.03; Copilot: –0.258, P=.01; Scite: –0.357, P<.01; Multimedia Appendix 5). Regarding the English outputs, Copilot and Gemini had higher overall mean (SD) scores from category 3 (69.3%, 16.0% and 41.9%, 60.0%) to category 4 (71.4%, 9.4% and 45.8%, 31.4%). The maximum overall mean AQUOS was reached by ChatGPT (81.0%, SD 18.6%). Gemini had the lowest overall mean AQUOS of 41.9% (SD 60.0%). ChatGPT reached the highest single overall AQUOS with 95.2%, while Gemini achieved the lowest overall score in English (−19.0%).

Gemini caused possible mild harm in category 2 (mean 0.6, SD 0.9). In category 3, mild harm was noted in Copilot (mean 0.5, SD 0.7), Gemini (mean 0.7, SD 0.7), and Scite (mean 0.5, SD 0.7). Category 4 indicated overall possible mild harm across all chatbots. Significant correlations between possible harm and GFR categories were found: ChatGPT (0.396; P<.001), Copilot (0.476; P<.001), and Scite (0.443; P<.001).

In the low-complexity category, 33 patients were included; in the medium-complexity category, 46 patients were included; and in the high-complexity category, 21 patients were included.

The trends of overall AQUOS in German and English are shown in Figure 2, with specific values provided in Multimedia Appendix 6. The overall AQUOS declines as the complexity increases (Scite: −0.239, P=.02; Multimedia Appendix 5). Regarding the English outputs, the only exception is Copilot, where AQUOS increases from medium to high complexity, but not more than the score in low complexity. The maximum overall mean AQUOS was achieved by ChatGPT (79.1%, SD 9.0%). The lowest overall mean AQUOS was observed Gemini (43.3%, SD 25.5%). In low complexity, ChatGPT reached the highest single overall AQUOS (95.2%), while Gemini reached the lowest single overall score (−19.0%).

Gemini caused possible harm in medium complexity, with a mean of 0.9 (SD 1.0), corresponding to mild harm. In high complexity, possible harm was mild and was caused by Gemini (mean 0.6, SD 1.0). There was a significant correlation coefficient between the possible harm and the complexity only in German: ChatGPT (0.219; P=.03), Copilot (0.226; P=.02), and Scite (0.237; P=.02).

For every chatbot, the overall AQUOS were higher when the prompt was in English than in German (Figure 3). The only exception was Copilot, whose overall AQUOS scores were the same (median 71.4% [IQR 13.3%] in German and 71.4% [IQR 6.7%] in English) in both languages. ChatGPT achieved the highest scores in English (median 76.2% [IQR 12.5%]), while Gemini reached the lowest scores in German (median 42.9% [IQR 33.3%]). There was no possible harm in the median; however, regarding the IQR of Gemini in both languages, there was mild harm.

There was no significant difference between t0 and t1 regarding the overall AQUOS (Table 2). However, some single-score criteria showed significant differences.

In low complexity, there was a significant difference at Gemini in English regarding AQUOS criterion 6 (referral to medical or pharmaceutical professionals or follow-up checks), with a relative difference of 15.2% (P=.02). In medium complexity, there was a significant difference at ChatGPT in German for criterion 3 (suitability of the references given), with a relative difference of 5.4% (P=.01). Other significant differences were observed at Gemini, all in English, for criterion 4 (correct dose recommendation) and possible harm, with corresponding relative differences of 16.9% (P=.004) and 12.5% (P=.001).

In this study, we tested our AQUOS for the first time, and to our knowledge, this is the first score that rates AI outputs in drug-related queries with different aspects summed up. The following aspects have been investigated individually: completeness [4], references [4], correctness [5,27], accuracy [10,27], and possible harm to patients [4,5,10]. We designed AQUOS to be applied to other drug-related queries, not just renal dosage adjustment. As a few criteria have been similarly used in the studies mentioned above, there is a basis for external validity. Nevertheless, it should be tested in other areas, studies, and languages.

Although AQUOS has some weaknesses, such as the AI chatbots regularly losing points for not giving references at all—despite the prompt instructing them to do so—and for not providing good or accessible references, as long as the content and the recommendations are correct and accurate, there would typically be no harm to the patient. Therefore, it may be necessary to adjust AQUOS depending on the research question.

AI chatbots generate fluent and professionally worded responses that may appear authoritative regardless of their actual accuracy or completeness. Users may, therefore, treat chatbot outputs as expert recommendations rather than as an informational starting point that requires critical evaluation. A related issue is automation bias, in which health care professionals may have overconfidence in the AI-generated outputs due to their confident tone, which can lead to dosage decisions being made without adequate verification.

The recurring question of accountability for such clinical decisions remains largely unresolved. Regulatory frameworks addressing AI in high-risk settings are increasingly emerging internationally, with the European Artificial Intelligence Act [30] being a well-known example that explicitly classifies certain AI applications in health care as high risk and imposes requirements for transparency and human oversight. However, implementation remains variable across countries and health care systems, and the specific allocation of accountability in cases where AI-generated recommendations result in patient harm has not yet been adequately addressed in practice.

These considerations support the conclusion that AI chatbots should be positioned as informational assistance, with the understanding that final clinical responsibility remains with the responsible health care professional, particularly in complex scenarios such as renal dosage adjustment in geriatric patients.

The findings of this study should be interpreted within the context of a clearly defined scope of application. The AI chatbots evaluated here serve as informational aids and should not be mistaken for clinical decision support systems in the regulatory sense or as defined by medical device regulations. Appropriate use scenarios include support for educational purposes, preliminary orientation for clinicians, or self-checking in low-risk, nonurgent cases. AI chatbots are not suitable for directly guiding prescribing decisions, replacing pharmacist consultations, or operating without subsequent verification by a qualified health care professional. This distinction is particularly critical in high-risk settings, such as renal dose adjustment in geriatric patients, where errors can directly harm patients. Any integration of AI chatbot outputs into clinical workflows must, therefore, be accompanied by explicit precautions that ensure the final clinical decision remains with the responsible health care professional.

This study has some limitations. First, we used a uniform input structure, varying the GFR values and drug dosages. This approach ensured consistency across the cases, though it does not capture the potential variability that may arise from alternative prompt formulations or interaction types. While we did not incorporate few-shot examples, follow-up dialogs, adjustment settings such as temperature or system instructions, or the usage of the web interface instead of an application programming interface—which reduces reproducibility—our method was intended to reflect a common practical usage scenario. Since the rater was not blinded during scoring, the possibility of bias cannot be ruled out.

The patient data were from a single hospital, and we focused on the discharge medication. Our analysis concentrated specifically on the GFR and related dosage adjustments, which are clinically relevant, though other aspects of the medication could also be considered in future research.

These methodological choices, while deliberate, mean that the findings should be interpreted within the context of the study design. Future studies could expand the range of clinical variables, use diverse data sources, and explore different interaction modes and types to further evaluate chatbot performance.

Besides pharmaceutical or drug-related queries, there is an increasing number of scientific studies evaluating medical chatbots in health care.

Huo et al [31] conducted a systematic review to examine the aspects investigated in studies evaluating AI chatbots in health care contexts. They found that almost two thirds of the 137 included studies evaluated AI chatbots using subjective parameters only. They also noted a lack of detailed descriptions of prompt engineering, as well as insufficient consideration of patient safety, regulatory issues, and ethical considerations. Our study directly addresses several of the gaps identified by Huo et al [31], including the need for objective assessment and methodological transparency, such as in prompt design and safety or harm assessment. Complementary to this, other research has focused on evaluating conversational AI chatbots with particular attention to their limitations and associated concerns [32]. Wang et al [32] included 65 studies in their systematic review in which AI chatbots like ChatGPT were applied in health care contexts. Nearly half of the reviewed studies examined medical knowledge inquiries and reported a rather high precision in the AI chatbots’ responses. In 85% of all papers analyzed, concerns such as the reliability and bias of the AI chatbots were raised [32]. Our findings align with these results, as our evaluation using AQUOS and, for example, the study design with 2 different prompting times also places a strong emphasis on reliability.

Li et al [33] compared 8 different AI chatbots on 48 clinical questions using a combination of an expert panel and a 6D evaluation framework. They found that all the chatbots showed limitations and weaknesses in complex cases but also highlighted the potential of AI chatbots, always considering the associated risks [33]. Similarly, our study revealed such limitations, especially in more complex cases, such as those involving reduced renal function or multiple concomitant medications. Additionally, another review found that LLMs can support clinicians in various tasks but are not yet reliably applicable across all clinical domains, making deployment challenging [34].

We developed and validated a new score (AQUOS) to examine the quality of AI chatbot answers for dose adjustment in patients according to their renal function. Using real patient data from a geriatric setting, we found that the quality of the AI chatbots’ responses varied depending on the chatbot, with values ranging from −19.0% to 95.2%. First, these responses were language-dependent. This is relevant because, in inpatient and outpatient settings, information is often sought in the patient’s native language. Furthermore, the results were highly reproducible and did not differ significantly at 2 points in time with independent search queries. However, it was found that the quality of the queries and the potential harm were adversely affected when kidney function decreased and the number of medications increased. In general, however, the quality of the responses is not yet convincing, especially in complex situations where advice from a chatbot is particularly sought after. ChatGPT proved to be the best chatbot in terms of quality. Future studies should evaluate AI chatbot performance using broader, more individualized clinical contexts to assess and compare directly whether additional information about the patients improves the accuracy and safety of AI-generated dosing recommendations. Further research should also be encouraged to find ways to influence the results obtained to make them suitable for practical use. In addition, ethical and legal issues surrounding the use of chatbots with real patients in treatment routines still need to be clarified in the future.