Since May 2022, outbreaks of mpox (formerly known as monkeypox) have been reported primarily in Europe and the United States, with more than 99,500 cases and more than 200 deaths reported from 122 countries [1]. Japan has reported 247 cases and 1 death [1]. Outbreaks of mpox appear to occur disproportionately among men who have had sexual contact with other men [2], but infection can also occur through close and prolonged physical contact, regardless of sex, gender identity, or sexual orientation [3].

The increasing availability of health education materials online has transformed public health education, particularly regarding topics related to sexual health, where accessibility and anonymity are important [4]. However, many patient education materials created by medical professionals are written at a reading level higher than that recommended for the general adult population, thereby creating a barrier to understanding critical health information [5-10]. The National Workgroup on Cancer and Health [11], the American Medical Association [12], and the Centers for Disease Control and Prevention [13] recommend that patient educational texts be readable at a sixth- to eighth-grade level or lower. However, evaluations of mpox-related content on the internet to date have indicated that much of this material is difficult for the general public to understand [14].

Artificial intelligence (AI) tools such as ChatGPT offer novel approaches to making health education materials more accessible. Recent studies have shown that AI-generated texts can simplify complex medical information and improve readability, outperforming traditional web-based materials [14-16]. By enhancing clarity and accessibility, AI-generated texts have the potential to reduce health disparities and ensure equitable access to vital health information, particularly during public health crises such as the mpox outbreak.

This study aims to evaluate the comprehensibility, actionability, and readability of Japanese online health education materials on mpox in comparison with content generated by ChatGPT-4o. Our findings may guide public health policies by demonstrating how AI tools can improve the clarity of health education materials. Further, this research will support the integration of AI-generated text into public health strategies, potentially reducing the burden on health care professionals and enhancing communication during emergencies.

This study compares the comprehensibility, actionability, and readability of Japanese health education materials on mpox published online with that of AI-generated content from ChatGPT-4o. Our results suggest that AI-generated texts are generally easier to understand and written in simpler Japanese compared to traditional web-based materials. These findings are consistent with those reported in previous research [14,22,23].

AI-generated texts such as those from ChatGPT-4o offer the advantage of providing clearer and more straightforward information, which is critical in the context of public health emergencies. By reducing the cognitive load required to comprehend and act on health information, AI has the potential to improve health literacy and facilitate more effective communication with populations that may face barriers in accessing or understanding complex medical information. However, one key limitation of AI-generated content is the lack of visual aids, charts, and graphs that often accompany web-based texts, which can significantly enhance understanding. Incorporating multimedia elements such as images, diagrams, and graphs into AI-generated content could further improve the comprehensibility and practical application of health information, particularly for complex medical topics.

Regarding actionability, the results showed that web-based texts provided more explicit calls to action than AI-generated content. This difference may be due to the nature of the prompt used to generate AI content, which did not specifically encourage behavior-specific guidance. Research suggests that customized prompts such as asking AI to explain at a sixth-grade level can enhance clarity and actionability [23]. In this study, however, we intentionally used simple keyword-based prompts (eg, Tell me about mpox) to reflect realistic information-seeking behavior by general users. Given that the public understanding of mpox remains limited, particularly in Japan, we assumed that users would initiate queries with minimal phrasing. Assessing the quality of AI-generated materials in response to such basic prompts is important for evaluating the real-world utility of AI in public health education. To further support behavior-oriented communication, developing prompts that elicit more specific and actionable outputs warrants consideration. For instance, prompts such as “Explain mpox to someone with no medical background” or “List 3 actions I should take to avoid mpox” might enhance comprehensibility and actionability. Moreover, instead of expecting complete guidance in a single interaction, users could issue follow-up prompts like “What should I do if I think I might have mpox?” or “How can I talk to my partner about mpox prevention?” These examples are exploratory and suggest that more systematic investigation into prompt design will be essential for maximizing the practical utility of AI-generated health content. For AI-generated health information to be practically applicable and useful in public health contexts, future research should emphasize optimizing prompt use to ensure that the results not only provide information but also encourage preventive actions and healthy behaviors.

The readability analysis suggested that AI-generated texts were easier to read than web-based materials. This finding is important because readability plays a critical role in how effectively public health information can be understood and acted upon, particularly in times of crisis like the mpox outbreak. By maintaining lower levels of text complexity, AI-generated content can serve as a useful tool for delivering health education to broader audiences, including underserved populations.

This study is, to the best of our knowledge, the first to compare health education texts on mpox available on web pages and that available via AI chatbots. This is also the first study to evaluate the understandability, actionability, and writing difficulty of Japanese health education texts on mpox published online. Many searches for health education information related to mpox are likely conducted on the web and via AI chatbots because of ease of access and anonymity. For this reason, we believe evaluating the quality of information related to mpox is important.

Several limitations to this study must be kept in mind. First, AI chatbots sometimes generate inaccurate information; therefore, issues with the reliability of the resulting information need to be investigated. Second, this study used a simple prompt, “Please tell me about the keyword,” and did not consider prompts aimed at reducing sentence difficulty, generating summaries, generating charts, or promoting prevention or health behaviors. Future research should consider prompts that increase comprehension and ease of action while reducing text difficulty. Third, because this was a cross-sectional study, only information available at the time of the survey was assessed. Web and ChatGPT-4o health education texts are constantly changing. For example, as the spread of mpox has increased, information available on the internet has been updated frequently. In addition, new versions of AI chatbots are released every few months, and the quality of the generated text has reportedly improved with each new version [23,24]. In this study, we used ChatGPT-4o (OpenAI) from August 20 to 30, 2024. Therefore, the findings reflect the performance of that specific version at that time. It is important to note that the results of this study merely represent a snapshot of the situation at the time of the evaluation. Additionally, the moderate interrater reliability observed (ICC=0.43) may be explained by differing interpretations between the physician and epidemiologist who served as raters. For instance, in PEMAT item 4 (definition of medical terms), the physician assessed that definitions were present, whereas the epidemiologist judged that the explanations were insufficient due to the complexity of the terminology. In contrast, for item 21 (clear steps for actions), the epidemiologist found the materials actionable, while the physician considered the instructions inadequate. These discrepancies suggest that the raters’ professional backgrounds influenced their assessments, highlighting a potential limitation in applying PEMAT across multidisciplinary evaluators. Finally, it should be noted that current AI models such as ChatGPT-4o cannot generate visual materials (eg, diagrams, illustrations) in response to general prompts such as “Tell me about mpox.” Due to copyright and security limitations, they are also unable to automatically incorporate images from the internet. Therefore, for AI-generated materials to include visual aids, additional steps such as human editing or integration with external tools are required. Alternatively, providing trustworthy visuals from academic articles or official sources to the AI model and requesting explanations based on them may be a practical workaround in future applications.

AI-generated health education materials such as those from ChatGPT-4o appear easier to understand and read compared to traditional web-based content. However, visual aids and clear action-oriented guidance are lacking. These findings suggest that AI can improve how health information is shared with the public, particularly during health emergencies like the mpox outbreak. Nonetheless, more refinement is needed to ensure that the information provided is both actionable and comprehensive. One possible approach to addressing this limitation is prompt engineering, which involves designing questions that elicit more specific, behavior-oriented responses from AI. Additionally, fine-tuning the AI model using public health domain–specific data could help generate more tailored and context-appropriate health guidance.