TY  - JOUR
AU  - Spina, Aidin
AU  - Andalib, Saman
AU  - Flores, Daniel
AU  - Vermani, Rishi
AU  - Halaseh, Faris F
AU  - Nelson, Ariana M
PY  - 2024
DA  - 2024/8/13
TI  - Evaluation of Generative Language Models in Personalizing Medical Information: Instrument Validation Study
JO  - JMIR AI
SP  - e54371
VL  - 3
KW  - generative language model
KW  - GLM
KW  - artificial intelligence
KW  - AI
KW  - low health literacy
KW  - LHL
KW  - readability
KW  - GLMs
KW  - language model
KW  - language models
KW  - health literacy
KW  - understandable
KW  - understandability
KW  - knowledge translation
KW  - comprehension
KW  - generative
KW  - NLP
KW  - natural language processing
KW  - reading level
KW  - reading levels
KW  - education
KW  - medical text
KW  - medical texts
KW  - medical information
KW  - health information
AB  - Background: Although uncertainties exist regarding implementation, artificial intelligence–driven generative language models (GLMs) have enormous potential in medicine. Deployment of GLMs could improve patient comprehension of clinical texts and improve low health literacy. Objective: The goal of this study is to evaluate the potential of ChatGPT-3.5 and GPT-4 to tailor the complexity of medical information to patient-specific input education level, which is crucial if it is to serve as a tool in addressing low health literacy. Methods: Input templates related to 2 prevalent chronic diseases—type II diabetes and hypertension—were designed. Each clinical vignette was adjusted for hypothetical patient education levels to evaluate output personalization. To assess the success of a GLM (GPT-3.5 and GPT-4) in tailoring output writing, the readability of pre- and posttransformation outputs were quantified using the Flesch reading ease score (FKRE) and the Flesch-Kincaid grade level (FKGL). Results: Responses (n=80) were generated using GPT-3.5 and GPT-4 across 2 clinical vignettes. For GPT-3.5, FKRE means were 57.75 (SD 4.75), 51.28 (SD 5.14), 32.28 (SD 4.52), and 28.31 (SD 5.22) for 6th grade, 8th grade, high school, and bachelor’s, respectively; FKGL mean scores were 9.08 (SD 0.90), 10.27 (SD 1.06), 13.4 (SD 0.80), and 13.74 (SD 1.18). GPT-3.5 only aligned with the prespecified education levels at the bachelor’s degree. Conversely, GPT-4’s FKRE mean scores were 74.54 (SD 2.6), 71.25 (SD 4.96), 47.61 (SD 6.13), and 13.71 (SD 5.77), with FKGL mean scores of 6.3 (SD 0.73), 6.7 (SD 1.11), 11.09 (SD 1.26), and 17.03 (SD 1.11) for the same respective education levels. GPT-4 met the target readability for all groups except the 6th-grade FKRE average. Both GLMs produced outputs with statistically significant differences (P<.001; 8th grade P<.001; high school P<.001; bachelors P=.003; FKGL: 6th grade P=.001; 8th grade P<.001; high school P<.001; bachelors P<.001) between mean FKRE and FKGL across input education levels. Conclusions: GLMs can change the structure and readability of medical text outputs according to input-specified education. However, GLMs categorize input education designation into 3 broad tiers of output readability: easy (6th and 8th grade), medium (high school), and difficult (bachelor’s degree). This is the first result to suggest that there are broader boundaries in the success of GLMs in output text simplification. Future research must establish how GLMs can reliably personalize medical texts to prespecified education levels to enable a broader impact on health care literacy. 
SN  - 2817-1705
UR  - https://ai.jmir.org/2024/1/e54371
UR  - https://doi.org/10.2196/54371
DO  - 10.2196/54371
ID  - info:doi/10.2196/54371
ER  -