Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review

Introduction

The rapid rise of chatbots, also known as conversational agents, has garnered substantial interest in the health care market. Valued at $787.1 million in 2022, the global health care chatbot market is expected to grow at an annual rate of 23.9% from 2023 to 2030 [1]. This expansion is driven by the increasing demand for virtual health assistance, growing collaborations between health care providers and industry players, and the acceleration prompted by the COVID-19 pandemic. For example, over 1000 health care organizations worldwide developed COVID-19-specific chatbots using Microsoft’s Health care Bot service to manage patient inquiries and reduce the burden on medical staff [2]. Entering the age of generative artificial intelligence (AI), health care chatbots have received even more attention since they enable human-level fluent conversations, have reached physician-level performance on board residency examinations [3] and comparable performance on other medical examinations and questions [4-6], and offer easy ways to train and adapt.

However, despite their popularity and potential, evaluating health care chatbots poses many challenges [7-9]. A lack of standardized evaluation approaches has led to diverse and inconsistent methods, making comparing chatbot performance difficult. Rapid technological advancements, particularly in generative AI, outpace existing regulatory frameworks [10], complicating the establishment of evaluation standards. These new chatbots using generative AI are not constrained by decision trees and are often built on top of larger models, meaning both the output and foundation are not stable. With such a moving target for evaluation, there is no widely accepted guideline or framework for evaluating health care chatbots. Developers lack a guide for assessment [11], and users often rely on company advertisements or marketing claims.

Several evaluation frameworks [12-22] have emerged in response to these challenges over the last few years, particularly following the popularity of generative AI. These frameworks vary—some review existing works and regroup metrics into a new structure, others adapt non–health care evaluation frameworks for this field, and some focus on narrow subdirections such as specific specialties or chatbot types. Given the need for a general guiding evaluation framework, a novel approach is necessary. Inspired by a framework for evaluating health apps [23], which has now been adopted by the American Psychiatric Association, we crafted a general evaluation framework integrating a literature review and broad stakeholder analyses. This approach involves the perspectives of developers, clinicians, patients, and policymakers to create a comprehensive evaluation structure.

Methods

As health care chatbots face a variety of users, there is no single way to evaluate a chatbot. Factors such as safety and privacy, user preferences, technology literacy, accessibility, and treatment goals are crucial in determining the most suitable evaluation method. In addressing these issues, organizations like the Coalition for Health AI (CHAI) have been working on designing guidelines for trustworthy AI. In April 2023, a group of experts representing diverse stakeholders crafted a blueprint for trustworthy AI implementation guidance [24]. This blueprint includes 7 aspects of trustworthy AI in health care: usefulness, safety, accountability and transparency, explainability and interpretability, fairness, security and resilience, and enhanced privacy. But this framework serves more as a theoretical foundation rather than an empirical evaluation framework, and its similarity or overlap with other frameworks remains unclear.

Building on the construct definitions in this blueprint and existing evaluation frameworks, we (1) identified a total of 11 evaluation frameworks; (2) extracted all individual questions from these frameworks; (3) removed redundant and nonrelevant questions; (4) mapped the remaining questions to CHAI constructs, their subcategories, and constructs not covered by CHAI’s blueprint; (5) improved the evaluation framework structure with stakeholders, including health care providers, patients, technology developers, epidemiologists, and policymakers; and (6) further merged and rephrased questions based on assigned constructs.

To identify and evaluate existing frameworks for health care conversational agents, we followed the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines to conduct a systematic review (Figure 1). The literature search was performed across multiple databases to ensure comprehensive coverage of relevant studies. The databases and corresponding search terms are shown in Table 1.

The initial search results were screened based on titles and abstracts. Two authors (YH and WX) independently reviewed the titles and abstracts for full-text retrieval, with any discrepancies resolved by discussion with a third reviewer (JT). Full-text articles were then retrieved for further assessment against the inclusion criteria. From the initial 266 records, 152 were screened, and 21 reports were sought for retrieval. After detailed assessment, 11 studies were included in the review, providing a comprehensive evaluation of frameworks for health care conversational agents.

We began by summarizing each framework’s intended use to assess specific concepts within a particular domain. The sections detailing the evaluation framework’s questions were then extracted and listed. If the study did not explicitly present evaluation criteria in the form of questions, these criteria were rephrased as questions for clarity. The following steps were taken:

1. Describe the use intention: the purpose and intended application of the framework were articulated, highlighting its relevance and scope.
2. Identify the concepts evaluated: the key concepts and dimensions the framework evaluates were identified and outlined.
3. List the evaluation questions: a thorough list of the questions evaluated by the framework was provided. In cases where the study did not present evaluation criteria as questions, these criteria were rephrased into question format for consistency and clarity.

‎

To improve the evaluation framework structure, we adopted a similar approach to the 2019 study on creating a taxonomy for smartphone app evaluation [23]. This process involved a form of qualitative factor analysis, where all 4 authors, including 1 clinician, independently reviewed and discussed how the evaluation questions aligned with specific factors or categories. Consensus was reached through iterative discussions to ensure accuracy and relevance, allowing us to systematically organize the questions into meaningful constructs.

Results

The reviewed frameworks are listed in Table 2 with details of their intended use, the concepts evaluated, and the terms used for conversational agents.

Initially, 275 questions were extracted from these frameworks (Multimedia Appendix 1). Questions that contained multiple subquestions were broken down for clarity. Broad questions that were too general to be useful were removed (eg, “Can strategies or solutions be developed to address problems of conversational agents?”). This iterative process led to a refined list of 271 questions, mapped across constructs outlined in the Coalition for Health AI blueprint and beyond (Multimedia Appendix 2).

The final structure of the Health Care AI Chatbot Evaluation Framework (HAICEF) represents 3 priority-level constructs, 18 second-level constructs, and 60 third-level constructs (the first 2 levels are shown in Figure 2 and the full framework is shown in Table 3). The 271 questions covered 56 third-level constructs. Among these questions, Design and Operational Effectiveness accounted for 108 (40%) questions. Trustworthiness and Usefulness accounted for a similar weight of 107 questions each (39%). The most fundamental level of Safety, Privacy, and Fairness included 56 questions (21%).

‎

Table 3 presents the full structure of HAICEF, with definitions provided for each concept to enhance clarity. This review-based framework addresses multiple dimensions to ensure the effective, safe, and ethical operation of health care chatbots. For instance, safety considerations include measures such as data provenance, which ensures that the chatbot relies on credible data sources including hospital electronic health records and peer-reviewed studies, and harm control mechanisms to minimize risks, such as providing inaccurate advice that could lead to adverse outcomes. Privacy and security are critical pillars, with requirements for compliance with standards like the Health Insurance Portability and Accountability Act and the General Data Protection Regulation, robust data encryption practices, and clear privacy policies outlining how user data are handled. Fairness and bias management address systemic, computational, and population biases, ensuring that the chatbot provides equitable responses across diverse demographics. The emphasis on accountability and transparency mandates that chatbots clearly communicate their limitations, data usage, and performance metrics while adhering to ethical standards that prioritize patient safety and respect for autonomy. For example, questions such as “Is the chatbot compliant with data privacy regulations?” and “Does the chatbot avoid causing harm through incorrect information?” illustrate the necessity of rigorous evaluation to maintain trustworthiness and ethical integrity in health care applications (Multimedia Appendix 2).

Figure 3 shows the distribution of constructs in the HAICEF framework, which shows a strong emphasis of current framework studies on “design and operational effectiveness,” accounting for 108 level 3 constructs, making it the most densely populated level 1 category. Within this, “accessibility” and “personalized engagement” dominate, with 27 and 20 level 3 constructs, respectively, reflecting the importance of inclusivity and tailored user interactions in chatbot evaluation. Conversely, “trustworthiness and usefulness” exhibits a balanced distribution across its level 2 constructs, such as “reliability” (11 constructs) and “transparency” (7 constructs), emphasizing the need for dependable and comprehensible chatbot systems. The “safety, privacy, and fairness” category, though less populous with 56 level 3 constructs, prioritizes critical areas like “privacy” (11 constructs) and “safety” (9 constructs), underscoring foundational principles necessary for secure and equitable operation.

In Table 4, we compare all reviewed frameworks across key evaluation dimensions, including safety, privacy, fairness, trustworthiness, and operational effectiveness. Each prior framework contributes valuable insights but remains partial, often focusing on isolated aspects rather than providing a comprehensive evaluation structure. HAICEF integrates these fragmented approaches, deduplicating and synthesizing 356 questions into a unified, adaptable framework that addresses the full spectrum of chatbot evaluation needs. By incorporating elements from all existing frameworks, HAICEF establishes the first standardized guidance for a holistic and comprehensive assessment across diverse health care applications.

‎

Discussion

Principal Findings

Chatbots are increasingly widely used in health care, but no comprehensive framework for evaluating their performance has been available. We surveyed the existing frameworks and developed a new framework using PRISMA guidelines, which we hope will facilitate future comparisons. The new proposed framework, HAICEF, is designed to meet the myriad users, use cases, and advances around health AI chatbots by providing a flexible scaffolding to support informed decision making.

HAICEF’s foundation in safety, privacy, and fairness is well aligned with recent research raising concerns about these aspects of chatbots. A 2024 review of AI apps concluded these apps may cause harm associated with bias [25], and the 2023 real-world case of an AI chatbot for eating disorders giving dangerous information to users [26] highlights the importance of step 1 in our framework (see Figure 2). Not all AI chatbots are patient-facing, and the framework is relevant to scaffolding conversations about clinical documentation chatbots, differential diagnosis chatbots, and even scheduling chatbots given the core aspects of the framework are relevant. For example, while efforts are underway to identify and address bias in conversational agents [27], checking for and identifying bias in any chatbot is a productive first step in considering any conversational agent and is a foundational step for avoiding harm.

Likewise, HAICEF’s second step, trustworthiness and usefulness, is grounded in recent research. From concerning trends of conversational agents drawing schizophrenia in a stigmatizing manner [28] to some chatbots providing details on self-harm and how to die by suicide [29], it is critical to assess the trustworthiness and usefulness of conversational agents. Given most conversational agents today are trained on social media, not health data [30], there is justified concern about the utility of the information provided. Additionally, subtle errors can be mixed with correct responses that are difficult for even experts to detect [31]. While there are many approaches to determine trustworthiness and usefulness, and our framework does not dictate which should be used, the structure ensures a focus on this critical issue.

HAICEF also celebrates the success of conversational agents with step 3 considering factors like their often-high degree of accessibility and efforts to personalize content. In placing step 3 after the prior 2, our framework reminds the user to first consider the potential risks and appropriateness of the conversational agent. The majority of frameworks we assessed (Table 4) focused on the questions included here in step 3. Our approach provides a complimentary means to consider these same questions but in the broader context of steps 1 and 2.

The framework offers several advantages by synthesizing insights from previous efforts into a new, synergistic model applicable across diverse health conditions and stakeholder groups. Unlike traditional methods that report isolated metrics, HAICEF re-evaluates existing frameworks to distill and integrate them into a comprehensive general guiding framework. It is not designed to challenge or replace any framework and is flexible enough to incorporate new ones that will likely be developed.

A distinctive feature of the framework is its multilevel tree structure, mapping questions into granular constructs without assigning scores to individual questions. This approach facilitates future development of more detailed, domain-specific evaluation methods, using our framework as a reference or guide. Additionally, we aimed to maintain a consistent level of granularity across all levels of the framework, ensuring that each aspect of evaluation is addressed with equal thoroughness.

The pyramid structure, similar to Maslow’s Hierarchy of Needs [32], serves as a visual reminder that evaluation may begin at the base, and progression is likely unnecessary if any level fails to meet the required standards. Still, the user may opt to approach the constructs and questions in any manner that suits their needs. The process of going through these questions will likely facilitate productive dialogue and reveal tensions that must be addressed by the user to make the optimal selection. Thus, this structure does not itself perform an evaluation but rather serves as a scaffold for evaluation. The same chatbot will be evaluated differently depending on the user and their intent for use, reflecting the flexible nature of this framing. The detailed questions, summarized in Multimedia Appendix 1, are designed to encourage and facilitate dialogue among stakeholders, with responses contextualized within each stakeholder’s unique situation. For instance, some chatbots may collect user conversation histories for training purposes by default. Some patients may find this unacceptable, while others may be comfortable with it. Similarly, developers focused on improving chatbot validity and reliability should not be compelled to conduct user feedback field studies if their research scope explicitly excludes user experience.

The rise of generative AI, such as ChatGPT [33], has expanded interest in health care chatbots, placing a pressing need for robust evaluation guidance. Yet the emergence of so many frameworks may create more uncertainty. By assessing the details of numerous frameworks, we were able to simplify and unify different approaches to help inform decision-making. The current framework is designed to be flexible and serve various decision makers—from a designer seeking to create a new chatbot to a patient selecting one from the marketplace. Depending on the user and use case, a different weighting to each construct will be necessary, in the same manner that ethical principles offer a scaffold to guide diverse decision making. Our analysis (see Figure 3) suggests that while most frameworks emphasize factors like user experience and task efficiency, stakeholder feedback indicates that focusing on safety and usefulness (see Figure 2) may better match user needs and concerns. Furthermore, we hope these findings will help guide policymakers to design effective evaluation regulations for health care chatbots, safeguarding the quality of information and providing a clear roadmap for businesses worldwide to further develop tools that improve care.

Finally, certain constructs, such as accessibility assurance and accountability assurance, remain placeholders due to limited coverage in the literature. These elements underscore the need for further investigation to develop more specific evaluation questions and classifications. As research on these areas grows, we anticipate integrating new findings into an expanded framework that remains both comprehensive and adaptable.

This approach has several limitations. The framework should be validated prospectively in different contexts to ensure that it is comprehensive and captures important dimensions. There may be additional dimensions that need to be added as the underlying technology quickly evolves, uncovering new issues. Another limitation pertains to the selection and appraisal of source articles. We did not perform a formal quality assessment of each included framework beyond ensuring they met the inclusion and exclusion criteria (eg, scope and relevance to health care chatbots). All frameworks that aligned with these criteria were deemed acceptable. We justified this approach in order to cast a broad net and capture the wide diversity of existing evaluation tools. However, future iterations of this review and framework could benefit from a standardized critical appraisal of each source to more rigorously assess the quality of the evidence guiding HAICEF.

Given the absence of a universal standard for evaluating health care chatbots, many parallel review tools have emerged, often failing to capture the full range of important considerations. Our framework addresses this gap, offering a comprehensive, adaptable tool for the evaluation of health care chatbots, which we hope will lead to responsible integration of chatbots into health care settings. Furthermore, we hope that this review will help guide policymakers to design effective evaluation regulations for health care chatbots, both to safeguard the quality of information and provide a clear roadmap for businesses worldwide to further develop tools that improve the quality, efficiency, and effectiveness of care.

HAICEF presents a starting point that will evolve. Next steps include fully exploring the needs of different users of health AI chatbots and their most common intents and goals. Exploring chatbots beyond the classical medical domains (eg, nephrology and radiology) and understanding functions across the health care ecosystems (from scheduling to crisis support) will help ensure the framework is responsive to real-world needs. Further work to expand the granularity of individual questions and their focus on users (eg, developers versus clinicians) will help improve usability. Additionally, we plan to engage a broader range of stakeholders through a Delphi consensus process to ensure the framework’s inclusivity and adoption. These efforts aim to establish HAICEF as a rigorous and widely adopted standard, enabling consistent and comparable evaluations across the health care sector. Through these efforts, we hope to establish a more rigorous, inclusive, and widely adopted evaluation framework for health care chatbots and enable “apples to apples” comparisons between them.

Conclusions

This is the first work to develop a structured and adaptable framework for evaluating health care AI chatbots, addressing the urgent need for standardized assessment criteria. By synthesizing insights from existing frameworks and diverse stakeholders, we developed a structured approach that prioritizes safety, privacy, trustworthiness, and usefulness. This framework is intended to guide the responsible evaluation and implementation of chatbots in health care, helping to ensure their safe and effective use. Future work will focus on validating and refining this framework in different contexts.