Accessibility settings

This paper is in the following e-collection/theme issue:

Applications of AI (951) Reviews in Digital Mental Health (555) Chatbots and Conversational Agents (1173) Artificial Intelligence (4754)

Published on in Vol 4 (2025)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/69006, first published .
Person using ChatGPT on a laptop, showing the AI chatbot interface with examples and capabilities.

Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review

Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review

Authors of this article:

Yining Hua1 Author Orcid Image ;   Winna Xia2 Author Orcid Image ;   David Bates3 Author Orcid Image ;   George Luke Hartstein4 Author Orcid Image ;   Hyungjin Tom Kim5 Author Orcid Image ;   Michael Li1 Author Orcid Image ;   Benjamin W Nelson2 Author Orcid Image ;   Charles Stromeyer IV2 Author Orcid Image ;   Darlene King6 Author Orcid Image ;   Jina Suh7 Author Orcid Image ;   Li Zhou3 Author Orcid Image ;   John Torous2 Author Orcid Image
Article Authors Cited by (20) Tweetations Metrics

Journals

  1. M U P, H R P, K P P, D N P, H N T. AI - Powered Medical Chatbot for Symptom Check. E3S Web of Conferences 2026;692:03010 View
  2. Barnawi G, Lazarowitz R, Nukaly H, Kashkari S, Lipner S. Ethical implications of artificial intelligence language interpretation tools in dermatology. Journal of the American Academy of Dermatology 2026;95(1):e37 View
  3. Wang X, Yin C, He H, Guo J, Fu X, Bai F. Benchmarking public large language model responses to patient-facing inflammatory bowel disease questions: informational quality, transparency proxies, and readability. Frontiers in Public Health 2026;14 View
  4. McGee A, Parsons G, Zhaunova L, Paul A, Kazlou A, Xu Y, Stsefanovich H, Klepchukova A, Meczner A. Safety-focused development and evaluation of an LLM sexual well-being chatbot for women: A methods-focused feasibility study. DIGITAL HEALTH 2026;12 View
  5. Castaño-Villegas N, Villa M, Barrientos K, Llano I, Velásquez L, Zea J. Arkangel AI, OpenEvidence, ChatGPT, Medisearch: Are They Objectively up to Medical Standards? A Real-Life Assessment of LLM Chatbots in Health Care. Mayo Clinic Proceedings: Digital Health 2026;4(2):100356 View
  6. Ourang S, Kahler B, Ha W, Jafari B, Zahedrozegar S, Nosrat A. Artificial Intelligence Chatbots’ Performance on Dental Trauma Case-based Queries: Examining the Effect of Prompt and Content Engineering. Journal of Endodontics 2026;52(9):1444 View
  7. Maw A, Lupi A, Johnson-Koenke R, Coats H, Zhou L, Li B, Mitchell J, Kotz A, Plasek J, Goss F. Iterative Multidisciplinary Development and Evaluation of a Patient-Facing Social Determinants of Health Chatbot Using Synthetic Data Simulation: Mixed Methods Study. JMIR Formative Research 2026;10:e89837 View
  8. Birk M, Kochhar S, Myrick K, Schueller S, Torous J. Digital Mental Health Research Priorities, Revisited for the AI and Large Language Model Era. JMIR Mental Health 2026;13:e104118 View
  9. Abesadze N, Fernandes A, Rubin E. Conversational Artificial Intelligence and Neuropsychiatric Risk: A Narrative Review and Case-Based Synthesis Proposing a Delusional Feedback Loop. Cureus 2026 View
  10. Rehman T, Jarrett P, Lesko J, Augustine J, Shy B, Sangal R, Genes N, Apakama D, Abbott E, Mehrotra A, Taylor R. An Evidence-Based Framework for Patient-Facing Artificial Intelligence Integration in the Emergency Department. JACEP Open 2026;7(5):100466 View
  11. Scherr S. AI conversation audit: How social media influencer prompt framing and free-versus-paid AI model tiers change chatbot response quality. Computers in Human Behavior: Artificial Humans 2026;9:100351 View
  12. King A, Banks A, Hernández L, Thompson S, Stevens L, Potter L, Kaphingst K, Estabrooks P, Del Fiol G, Wetter D, Schlechter C. An Acceptance Criteria Framework for Determining the Implementation Fit of Custom Large Language Models in Public Health Interventions. Journal of Medical Internet Research 2026;28:e92356 View
  13. Scalia J, Laprise J, Thrift J, Farrell C, Sarasua S. Metrics Used for the Evaluation of Chatbots Providing Cancer Genetic Risk Assessment and Education: Systematic Review. JMIR AI 2026;5:e76400 View
  14. Sharpe A, Chan Y, Saparamadu A. Mobile App Rating Scale for Health Care Professionals to Assess the Quality of mHealth Apps: Questionnaire Development and Psychometric Analysis. JMIR mHealth and uHealth 2026;14:e48828 View
  15. Koscelny S, Sadralashrafi S, Neyens D. The danger in easy answers: Conceptualizing subjective processing fluency for healthcare chatbot information presentation in an inductive reflexive thematic analysis of user experience and perceptions. Computers in Human Behavior: Artificial Humans 2026;9:100370 View
  16. Baglivo F, Gorga A, Vella G, Barbati C, D'Atri A, Pecchioli M, De Angelis L, Siliquini R, Mazzucco W, Odone A, Rizzo C. Training the public health workforce for generative AI: validation of the Italian Chatbot Usability Questionnaire. Frontiers in Public Health 2026;14 View
  17. Zhou Y, Honary M, Fayoumi A. Design Principles for a Risk-Aware Conversational Agent in Digital Mental Health Information Access: The Case of Cognitive Theatre. Information Systems Frontiers 2026 View
  18. Diel A, Torous J, Cuijpers P, Kleesiek J, Nensa F, Weber N, Faust F, Lalgi T, Mellis F, Teufel M, Bäuerle A. A scoping review on the mental health harms of LLM-based chatbots. npj Digital Medicine 2026;9(1) View

Books/Policy Documents

  1. Theilmann K, Steffny L, Dahlem N, Podevin D, Greff T, Bleistein T. Human-Computer Interaction. View

Conference Proceedings

  1. Hasan H, Salman S, Dargham A, Mohanna A, Chehab A. 2026 5th International Conference on Computer Networks, Big Data and IoT (ICCBI). Toward Trustworthy Large Language Model Agents in Healthcare View