Informing individuals about preventive care opportunities remains a priority for public health officials and clinicians due to the large proportion of deaths attributable to preventable risk factors [1] and the net socioeconomic benefits of prevention [2]. Historically, digital resources—such as online tools [3], text messages [4], and eHealth applications [5,6]—have advanced this priority. Today, large language models (LLMs)—that is, generative AI systems that produce text in response to user queries—have the potential to augment these existing digital resources. LLMs appear particularly attractive as a means of informing individuals about preventive care activities because they can provide individualized guidance tailored to the unique details of a particular user’s case, and they can do so at scale by automating advice delivery [7].

These capabilities of LLMs would help patients and providers surmount impediments to the provision of preventive care guidance. Survey evidence suggests that individuals show interest in preventive care upon learning about it [8], yet both commentary [9] and research [10] raise concerns about the time required for providers to share preventive care information. Indeed, underscoring these concerns, research finds a negative relationship between the number of preventive care guidelines and rates of preventive care use [11], and it reports survey evidence showing that physicians prioritize which preventive care guidelines to convey in time-constrained environments [12]. Optimistically, LLMs have the potential to address time and resource constraints by providing individualized recommendations (a format known to improve understanding of preventive care opportunities [1]) outside of clinical settings and at low cost, thus facilitating large volumes of personalized guidance.

The realization of these beneficial outcomes, however, depends on whether LLMs provide information consistent with authoritative sources of preventive care. This consistency may vary by LLM (ie, some LLMs might exhibit greater consistency with authoritative guidance than others) and by a user’s approach to prompting LLMs (ie, some prompting methods might generate outputs consistent with authoritative recommendations more regularly than other methods).

Measuring such consistency across LLMs warrants effort because individuals now have access to myriad LLMs, yet they lack the information needed to steer themselves toward LLMs that provide preventive care information that coincides with authoritative guidance. Although health care providers can direct patients to bespoke models customized for narrow forms of preventive care [7,13], many people express an intention to use general-purpose, commercial AI models to access health guidance [14], and those commercial models differ in the quality of the information they supply due, in part, to variation in their training and features. Model developers, for instance, might use distinct datasets to train their models [15], subtly different methods of parameter optimization [16], varying amounts of compute during training [17], alternative reasoning architectures [18], or dissimilar safeguarding techniques [19]. Any of these sources of LLM variation might lead to divergent performance across models, thus creating a rationale for exploring how a wide range of popular LLMs respond to preventive care queries. Existing research on the use of LLMs in preventive care guidance, however, has focused on a small set of LLMs—in particular, the models underlying OpenAI’s ChatGPT [20-24]. As a result, the present research aims to study a more extensive set of LLMs across multiple popular commercial model providers.

Existing research also has focused on a narrow set of preventive care topics, and it has not systematically assessed how alternative prompting approaches might affect responses [20-24]; that is, existing work has not systematically considered how variation in the inputs to LLMs might affect consistency with authoritative preventive care guidance. Users of LLMs remain unconstrained in both the preventive care topics that they can inquire about and the form of the prompts they use to query an LLM. Some users might inquire about cancer screening, while others might prompt LLMs to inquire about statin use to mitigate the risk of cardiovascular disease. Users might deploy straightforward zero-shot prompts [25], few-shot prompts that contain examples [26], chain-of-thought prompts that request stepwise reasoning [27], role-based prompts that assign simulated personas to LLMs [28,29], or iterative prompts that seek clarification on previous LLM outputs [30]. Exploring prompt variation across a wider range of preventive care topics would produce insight into how LLM consistency with authoritative recommendations changes as users vary the inputs they supply to LLMs. In so doing, such research would expand upon the firm foundation of insights from existing studies.

Those existing studies have found that ChatGPT deviates from established guidance in colorectal screening [20,24], while providing sound guidance about lung cancer screening [21] and breast cancer screening and prevention [22]. Such variation in performance suggests that health care providers and users cannot assume that LLMs will repeat authoritative guidance. That outcome provides further reason for ongoing investigations to assess a wider set of LLMs on a comprehensive list of authoritative preventive care recommendations with variation in user prompting approaches.

Accordingly, here we build upon existing research by studying whether 35 popular LLMs respond to queries about preventive care with outputs that accord with the recommendations and grades of the US Preventive Services Task Force (USPSTF)—a globally recognized source of authoritative guidance on preventive care. Our investigation includes the entire universe of published recommendations that appeared on the USPSTF’s official website as of May 2025, thus providing a comprehensive evaluation of how LLM-generated responses to preventive care queries compare with the official guidance of the USPSTF.

We conducted our research in 2 waves. Wave 1 (late spring/early summer of 2025) focused on how a set of 28 popular LLMs responded to a comprehensive range of preventive care queries (ie, queries that spanned the topics of all published USPSTF recommendations) when using a basic zero-shot prompting strategy. Wave 2 (late spring/early summer of 2026) repeated these procedures while also incorporating a wider range of prompting strategies—namely, chain-of-thought, few-shot, role-based, iterative, and an alternative zero-shot prompt. Wave 2 also added new models to ensure that the study’s findings remained relevant to current LLM users. Specifically, due to model deprecations [31] that eliminated access to some of the models studied in wave 1 and model innovations that created the opportunity to include more advanced models in our investigation, we formed a test set of 10 LLMs in wave 2 consisting of the most competitive models across the 5 commercial model providers in our study. We selected models using a deliberate procedure. If a commercial model provider still offered access to its top-performing model in wave 1, then we included that model in our study and paired it with its most recently released, high-performance model. If a commercial model provider no longer provided access to its top-performing models in wave 1, then we added the 2 most recently released high-performing models. In the event that a provider did not release any new models between wave 1 and wave 2, we included in wave 2 that provider’s top 2 performers from wave 1.

Through these procedures, our study examined the potential of LLMs to provide individualized preventive care guidance at scale. Not only did it test whether popular LLMs generate responses to preventive care queries that agree with the USPSTF’s official recommendations, but it also examined the sensitivity of LLM-USPSTF concordance rates to the domains of preventive care (eg, the modality or health category of preventive care) and to variation in user inputs (eg, across health topics or prompting styles). Furthermore, it used content analysis to seek out the reasons for LLM-USPSTF discrepancies and studied whether accuracy would improve by instructing LLMs to grade preventive care activities using the USPSTF grading scale. Together, these avenues of investigation simulated the diverse ways in which LLMs will be used to acquire guidance on preventive care, thus offering insight into the prospects of LLMs serving as sources of individualized preventive care that can be delivered at large scale.

Our investigation studied the responses of widely available LLMs developed by Anthropic, Google, Meta, NVIDIA, and OpenAI to queries concerning preventive care topics. The study occurred in 2 waves. Note that gpt-4.1-2025-04-14, Llama-4-Maverick-17B-128E-Instruct-FP8, and Llama-4-Scout-17B-16E-Instruct-FP8 were included in both waves 1 and 2, with the latter 2 models accessed via Meta’s online API in wave 1 and via the OpenRouter API in wave 2 (as shown in the Results section, all 3 models’ rates of agreement with USPSTF recommendations remained comparable across waves). Across each wave of the investigation, the study examined output generated by LLMs in response to the queries of simulated users, not real people; thus, the investigation did not require institutional review board review. All computer code and data used or generated in this investigation can be found at the OSF project page associated with this study [32].

The study’s focal tests—across waves 1 and 2—sought to assess whether LLMs provided advice consistent with authoritative preventive health recommendations from the USPSTF. We collected all 142 recommendations published on the USPSTF website as of May 13, 2025. The USPSTF classifies these recommendations by topic, population, age group, type, category, and grade. The data contained 90 unique topics. Topics can include multiple recommendations because USPSTF guidance occasionally varies within a given topic by population—that is, by the definable segment of the community for which a recommendation is relevant. The data concerned 116 unique populations. Related to the population variable, the USPSTF website also included a variable labeled age group (“adolescent,” “adult,” “pediatric,” “senior,” or some combination of those categories), and we included this variable in our data. Furthermore, the website listed the type of preventive care, which categorized care activities into those involving “counseling,” “preventive medication,” “screening,” or some combination thereof, and it recorded the clinical topic classification (eg, “infectious diseases” or “perinatal care”) via the variable “category,” which we also included in our data collection. The data also included the grade that the USPSTF had associated with its recommendation. The USPSTF issues 1 of 5 letter grades for a preventive care activity:

With these data, we performed our focal test in which we queried LLMs about whether an individual in a particular population ought to engage in a given preventive care activity and compared LLM-generated responses with official USPSTF guidance.

Queries in this test needed to include both a question about a preventive care activity and details concerning the individual who wrote the query: the latter information proved necessary because USPSTF recommendations only apply conditionally on membership in the relevant population. Thus, the LLM would need to receive information indicating whether the user who queried the system belonged to the population to which the recommendation applied. We did not study how LLMs responded to queries from hypothetical users who affiliate with populations to which USPSTF recommendations do not apply because of our focus on the degree of LLM consistency with USPSTF recommendations; one cannot assume that nonmembership in the relevant population obviates the USPSTF recommendation. Future research might consider an open-ended exploration of how LLMs respond to queries about preventive care topics when users do not belong to the populations to which USPSTF recommendations directly apply.

Given that USPSTF recommendations apply conditionally on the population, our queries included a description of a hypothetical user’s personal attributes. To mitigate the possibility that constructing these hypothetical personas and authoring our own queries might bias results, we designed a prompt that asked an LLM (namely, gpt-4o-2024-08-06) to produce a question on a particular USPSTF topic for a specific population. For instance, we supplied the topic “Abdominal Aortic Aneurysm Screening” and the population “Men aged 65 to 75 years who have ever smoked” to the system, asking it to provide a question such as, “I am a 70-year old man who has smoked. Should I receive screening for abdominal aortic aneurysm?” The LLM performed this task for each of the 142 topic-population combinations available from the USPSTF website at the time of the study (ie, all of the topics and associated relevant populations from the entire universe of published recommendations on the USPSTF website as of May 2025). The exact wording of the prompt that we used to develop these simulated queries and user characteristics, as well as the exact wording of the LLM-generated questions, is available in section 1 in Multimedia Appendix 1. Given that we did not study users’ actual queries, nor queries seeking advice that we personally would act on, we regard our tests as studying LLMs in simulated settings.

After generating queries via gpt-4o-2024-08-06, we embedded the queries in a prompt that provided additional context. The prompt stated, “I have a health question and I only want your non-binding, hypothetical advice. I will make my own decision based on additional information. [LLM-generated question stated here.] Again, I will treat your advice as non-binding and hypothetical; the ultimate decision solely will be my own.” We submitted 142 variants of the above prompt one at a time (ie, 1 for each LLM-generated question) to each of the 28 LLMs tested in wave 1 of our study, thus resulting in 3976 queries total in wave 1, and to each of the 10 LLMs tested in wave 2 of our study, thus yielding 1420 queries total in wave 2.

After receiving LLM responses to those queries, we submitted the responses and the USPSTF recommendations to the OpenAI model gpt-4.1-mini-2025-04-14, prompting it to assess whether LLM responses matched the USPSTF recommendations (exact prompt wording is available in section 2 in Multimedia Appendix 1). Please note that we did not serve as the raters for any of the responses we received from the LLMs in our study. Instead, LLM raters interpreted model responses to queries about preventive care throughout the study as a means of allowing large-scale systematic evaluation of the LLMs in our study (ie, it would not be possible to use human raters for the large number of responses we obtained across multiple LLMs). To assess LLM-rater consistency, we repeated the LLM-rating process twice to assess interrater reliability (ie, whether the model would make the same determination in each instance); the 2 runs showed high interrater reliability (wave 1: Cohen κ=0.8893; wave 2: Cohen κ=0.9366). We studied LLM-USPSTF concordance by model three times: (1) using only results in which both raters agreed (wave 1: n=3756 cases; wave 2: n=1375 cases), (2) using only the first set of ratings (wave 1: n=3976 cases; wave 2: n=1420 cases), and (3) using only the second set of ratings (wave 1: n=3976 cases; wave 2: n=1420 cases). Here, we focus only on cases with complete agreement; we report the latter 2 analyses, which yielded substantively similar findings, in section 3 in Multimedia Appendix 1. These results offer insight into the rate at which each model in our study echoed the recommendations of the USPSTF.

In addition to studying LLM-USPSTF concordance by model, we also explored whether and how the rate of concordance varied by substantively important cross-classifying variables supplied by the USPSTF. That is, recommendations vary in any number of dimensions; thus, we limited our initial investigation of how the LLM-USPSTF concordance rate varies with cross-classifying factors to those that the USPSTF reported on its website’s search results as of May 2025. By presenting these variables, the USPSTF gives reason to consider them important, and it offers an authoritative coding of each variable. Accordingly, we cross-tabulated LLM-USPSTF concordance by the type of preventive care mentioned in the recommendation (ie, counseling, preventive medication, screening, or some combination thereof), the health category (ie, clinical topic classification) to which the recommendation pertained, and the age group to which the recommendation applied.

Furthermore, in wave 1, we repeated our overall experiment at different values of the LLM’s temperature parameter. Temperature adjusts the sampling of candidate word tokens for model output. That is, after receiving an input prompt, an LLM produces a set of candidate tokens and assigns a probability to each candidate token; the LLM samples from those tokens in proportion to their assigned probabilities to generate a response. Adjusting the temperature modifies these probabilities. By increasing the temperature, high probabilities decline, and low probabilities increase, thus leveling their distribution; decreasing the temperature to 0 preserves the original assigned probabilities. In the main text, we report the results of experiments with temperature=1 (the default value of the temperature in the OpenAI Playground at the outset of our study). In section 4 in Multimedia Appendix 1, we present a repetition of our wave 1 experiment with temperature=0, which yields substantively similar results to those presented here. We did not perform such an analysis in wave 2 because AI developers no longer provide users with the ability to alter the temperature parameter for contemporary reasoning models studied in wave 2. Models in wave 2 for which the temperature can be adjusted were studied under multiple temperature parameters in wave 1.

To understand potential discrepancies between LLM recommendations and the official recommendations of the USPSTF, we also used an automated classification method to categorize the substance of LLM responses. The full prompts used in this portion of the study are available in section 5 in Multimedia Appendix 1. The procedure classified each recommendation (LLM and USPSTF, respectively) as either (1) recommending a preventive care activity, (2) recommending against a preventive care activity, (3) equivocating or encouraging the user to seek additional information or consultation, or (4) stating a lack of information or evidence to make a recommendation. We used the OpenAI model gpt-4.1-mini-2025-04-14 to make these determinations and had the model perform the classification twice to assess interrater reliability (wave 1: Cohen κ=0.9628; wave 2: Cohen κ=0.9070). We then studied the frequency of the LLM’s joint classifications of LLM and USPSTF recommendations for each topic-population combination.

In wave 2, we also explored, at the suggestion of reviewers, how variation in prompting approaches altered LLM responses. Our wave 1 prompt was a zero-shot prompt (ie, a prompt that did not offer any examples), and it contained phrasing that instructed the LLM that any response would be treated as nonbinding, hypothetical advice. The study deliberately added to or removed from this prompt to create variants that implemented alternative prompting approaches.

The prompt variant in wave 2 that required the least alteration of our original prompt retained wave 1’s zero-shot prompt format, yet removed wording indicating that the user would regard any responses as nonbinding, hypothetical advice. The resulting prompts consisted solely of the questions from simulated users and contained no other language adorning those queries. Section 1 in Multimedia Appendix 1 provides the exact wording of each of the inputs used in this prompting approach.

Another simple way that we studied variation in prompting approach was to use an iterative prompt that reflected the conversational nature of many LLM interactions. This prompt asked an LLM to elaborate on previously issued advice. Specifically, we fed, respectively, the user prompt and LLM response from our wave 2 baseline condition back into the LLMs via the following prompt: “Previously, I asked you to respond to the following prompt: <begin prompt> [Original Prompt Appeared Here] <end prompt> In response you provided me the following information: <begin response> [Original Response Appeared Here] <end response> Could you please clarify the previous response and/or provide more-definitive information?” To a stateless LLM, supplying previous iterations of a conversation simulates memory [33]; thus, the above prompt creates a scenario that resembles the conditions in which a user instructs an LLM to reduce ambiguity during a multiturn chat conversation.

Finally, we followed the guidance of reviewers to explore 3 additional, well-known prompting approaches: few-shot [26], chain-of-thought [27], and role-based [28,29] prompting. These approaches, respectively, provide examples, require stepwise reasoning, and assign hypothetical personas to the LLM with which a user interacts. Because countless alternative ways of implementing such prompts exist, we sought to avoid biases that might arise from our own efforts to author such prompts and devised a structured approach to prompt creation. In particular, we used Anthropic’s Claude Opus 4.7, Google’s Gemini Pro 3.1, and OpenAI’s ChatGPT-5.5—namely, the premier models deployed in each company’s public-facing online graphical user interfaces at the time of wave 2 (viz, May 2026)—to recast the prompts in our baseline condition into few-shot, chain-of-thought, and role-based prompts. Section 6 in Multimedia Appendix 1 provides the prompts used in the process of implementing the conversion. Section 7 in Multimedia Appendix 1 provides an example of a baseline prompt’s chain-of-thought, few-shot, and role-based prompt variants. The exact wording of every prompt across all portions of our investigation can be found in the replication data available at the OSF project page for this paper [32]. After submitting these new prompts to the models in our study, we repeated the procedures described above that, respectively, assessed whether LLM responses matched USPSTF guidance and gauged the reasons for matching or nonmatching sentiment between LLM responses and USPSTF recommendations.

In addition to studying the concordance of LLM advice and USPSTF recommendations, we also examined whether LLMs tasked with issuing grades using the USPSTF scale could replicate the actual grades issued by the USPSTF for preventive care activities. Our complete prompt for this task is provided in section 8 in Multimedia Appendix 1. The prompt instructed LLMs to use the USPSTF grading scale to assign a grade of A, B, C, D, or I to a particular USPSTF-evaluated preventive care activity as it relates to a particular population. After obtaining results, we extracted the letter grade from the LLM responses using the OpenAI model gpt-4o-2024-08-06 (the full prompt for the automated extraction is provided in section 9 in Multimedia Appendix 1). We then studied the correspondence between USPSTF letter grades and LLM-generated letter grades using cross-tabulation.

In summary, the study’s methods sought to design a series of tests that reflected all of the variation that might prove meaningful when using LLMs in preventive care—from the particular LLM selected for preventive care guidance to the health topic in the preventive care query to the prompting approach that a user might deploy to whether the prompt targets long-form preventive care guidance vs a concise grade. The methods also aimed to provide insight into the reasons for discrepancies between LLM responses and USPSTF guidance. Together, these methods provide insight into the viability of using LLMs to acquire individualized preventive care guidance at scale.

The study’s findings suggest that LLMs vary in the rate at which they generate outputs consistent with USPSTF recommendations. Older models in wave 1 of our study issued responses to preventive care queries that matched USPSTF recommendations in less than 70% of all tests. Mismatches primarily resulted from the LLMs’ resistance to making a definitive statement (ie, equivocation or encouraging more information acquisition) even in cases where the USPSTF provided a clear recommendation. Newer models in wave 2 of our study reached higher levels of concordance with USPSTF recommendations, and this consistency improved further with particular prompting techniques—most notably, chain-of-thought prompting. Varying the prompting approaches in our study also revealed ways in which users could reduce the high rates of equivocation observed in both wave 1 and the baseline condition of wave 2. No prompting approach could drive the rate of equivocation or encouragement for users to seek more information below 50%, but this equivocation or encouragement for further information search might suggest an understandable bias toward safety, the empowerment of user decision-making, and/or the avoidance of direct medical advice provisioning (we thank a reviewer of our manuscript for this important point). When our study sought more detail from LLMs by prompting them to grade preventive care activities using the USPSTF grading scale, it found that some LLMs matched USPSTF grades in a substantial proportion of instances, while others only generated grades consistent with those of the USPSTF in a fraction of instances, and that fraction varied widely across models. Undoubtedly, users prompting an LLM to grade preventive care activities using the USPSTF scale could consult the official USPSTF grades and avoid the performance variation of LLMs; however, our test aimed to see whether an attempt to avoid equivocation through a narrow task would yield results that more closely approximated USPSTF guidance. Together, these findings suggest that the more recent LLMs in wave 2 can serve as promising sources of preventive care guidance, although occasionally deviating from authoritative recommendations, whereas the older models in wave 1 would serve as riskier substitutes for traditional sources of information about preventive care.

In advance of discussing the implications of these findings, several limitations warrant attention. Each limitation not only shapes how one should evaluate the research contained in this paper, but also how one should assess the present work’s contributions to practical measures and future research.

First, our study tests LLMs against an authoritative standard, but it does not compare LLMs against a human benchmark, which might be appropriate given that clinicians often serve as intermediaries between care seekers and USPSTF guidance (ie, a person delivers the USPSTF recommendation). This omission warrants attention because human guidance outside the USPSTF might diverge from official recommendations—a possibility implied by previous research reporting that preventive care recommendations across countries agree at low rates [34]. If such divergence is beneficial, then measuring LLM response consistency with a human benchmark is particularly important, as the human benchmark would presumably capture instances in which deviations from authoritative guidance enhance preventive care. If such divergence from authoritative recommendations is not beneficial, then the present study’s methods can be viewed as a test of LLM responses against a gold standard. Whether the performance of LLMs in preventive care tasks ought to be measured as consistency with USPSTF recommendations, other authoritative sources of guidance, or human clinicians is a discussion that future research and commentary should undertake.

Second, instead of using actual queries from real individuals to prompt the LLMs in our study, we used simulated queries. Although this approach conforms to well-established methods in machine behavior research [35], it fails to fulfill recent calls for patient-produced data in tests of LLMs in health care settings [36]. To the extent that individuals seek information about preventive care on their own using LLMs, our study’s inability to use the wording of real user queries reduces the verisimilitude of our investigation. On the other hand, if individuals rarely seek preventive care information via LLMs on their own, then our simulated queries might lack external validity due to the implausibility of individuals seeking such information on their own. Identifying which of these scenarios holds true will help future research carefully consider how LLMs can be tested in ways that fully reflect their actual use (or lack of use) in the acquisition of preventive care information.

Third, we study LLMs in a capacity—information retrieval—for which they were not designed [37]. Such criticism seems particularly salient given the accuracy of LLMs in other health care contexts, such as diagnostic reasoning [38]. We agree that individuals should use LLMs cautiously for information retrieval absent additional fine-tuning (eg, fine-tuning of the type shown to improve LLMs’ surgical recommendations in clinical contexts [39]). However, we also recognize that a broad user base might not demonstrate such caution. Commercial survey evidence indicates that 42% of surveyed professionals anticipate using AI for information retrieval in lieu of search engines, whereas only 24% anticipate using search engines in lieu of AI [40]. As a result, we deem it important to explore this use case—even if the use case is itself inadvisable—to understand the type of outputs that might result from individuals using LLMs as a source of information about preventive care options.

Fourth, our investigation relies on the use of LLMs not only as test subjects whose outputs we compare with USPSTF recommendations, but also as raters that classify other LLMs’ outputs so that we can assess both rates of LLM-USPSTF concordance and the content of LLM responses (eg, whether responses offer a recommendation or an equivocation). Although we found high rates of interrater reliability among the models used to perform these classification and summary activities, LLMs might differ from human experts in their classifications, or they might err in their adjudications. At the same time, human experts also might err in their classifications or exhibit biases in their categorizations. Coupled with our lack of resources to recruit human experts to evaluate the thousands of LLM responses that our systematic study of all USPSTF topic-population combinations required us to collect, the possibility of human errors and biases lessened our concern that LLMs would not categorize responses as soundly as humans and led us to opt for LLM raters to perform the large volume of work that our systematic investigation required.

Those outputs, in our investigation, suggest the potential for the newer LLMs in wave 2 of our study to be used as sources of preventive care information, but those LLMs—like the older ones in wave 1—still show high rates of equivocation or encouraging the pursuit of additional information, even when USPSTF recommendations are clear. Due to the high rates of such responses, this manner of advice provision deserves further scrutiny and debate. For one, our study put equivocation and encouragement of further information search in the same category in its classification scheme of LLM responses. Ultimately, we feel justified in this decision in the sense that encouraging the user to gather more information implies that the user should not treat the LLM output as definitive. That is, equivocation could come in the form of the LLM presenting multiple conflicting perspectives without offering a definitive recommendation, or it could come in the form of the LLM presenting one perspective and outsourcing the collection of alternative views: in either of those scenarios, the LLM does not offer a definitive recommendation. But, albeit a form of equivocation, encouraging the pursuit of further information does differ from offering multiple conflicting perspectives in that it requires more of the user and intimates that the user should not consider the LLM as the sole source of information on the subject. Future research should investigate whether such responses actually stimulate further information search and how such responses shape individuals’ perceptions of the reliability of information derived from the LLM being used. Second, equivocation warrants further scrutiny in the literature because it suggests a tradeoff between safeguards that aim to inspire a judicious approach to preventive care and the clarity of preventive care recommendations. Ultimately, sources of authoritative guidance such as the USPSTF have developed as a means of creating a clear signal about care activities in a noisy environment that might be difficult for most people to navigate; if LLMs reach sufficient levels of consistency with authoritative recommendations, then safety orientations that compel them to present a multisided perspective on a topic might simply create confusion on a subject where clarity exists and should be communicated. On the other hand, such safeguarding might play an important role in cultivating deliberation skills in patients, empowering patients, and ensuring patient-provider collaboration in care decisions and execution. We hope that our research adds to broader calls to analyze and discuss the tradeoffs resulting from model safeguards.

Our results also point toward how practitioners might further evaluate LLMs before contemplating clinical deployment and how they might need to consider new practices for communicating preventive care guidance in an information environment where patients can access LLMs. First, because interpretation of our findings should recognize that we solely provide a simulation-based benchmark to study LLM-USPSTF concordance under a limited set of prompting conditions, the results of this paper ought not influence care decisions; instead, for practitioners who remain interested in the prospects of using LLMs to deliver preventive care guidance, our findings could identify a smaller set of high-performing models for evaluation before clinical deployment, thus improving the efficiency of practitioners’ efforts to appraise whether LLMs can viably play a role in preventive care guidance in actual clinical settings. Second, the paper provides users with guidance on techniques that can improve LLMs’ outputs related to preventive care. For instance, the evidence in this paper suggests that a user who adopts chain-of-thought prompting and elects to use gpt-5.5-2026-04-23 will gain information consistent with the USPSTF in approximately 95% of all instances (Table 5). Using the information in this paper to guide the selection of prompting approaches will support individuals planning to further assess the viability of using LLMs to gain information about preventive care. Again, such plans should remain cognizant of the limited scope of our investigation and the need for clinical validation before adoption for uses that influences preventive care. Third, the findings of this paper inform evaluators about health categories of preventive care for which the use of LLMs is more apt to be successful—for instance, when seeking preventive care guidance concerning cancer, infectious diseases, and perinatal care—vs categories in which LLM guidance might be less helpful—for instance, with respect to guidance pertaining to development and behavior or to hard-to-classify care activities (Table 4). Directing further evaluation to those promising areas offers another means to enhance the efficiency of future assessments directly aimed at discerning the viability of LLM deployment in clinical settings, which remains beyond the scope of our investigation. Fourth, the findings hint at areas in which model developers and clinicians might collaborate to improve LLMs’ consistency with authoritative sources of care advice. Models showed less success matching USPSTF recommendations in areas that could involve multiple preventive care modalities (ie, some combination of counseling, preventive medication, and screening; Table 3), implying a need for expert guidance on how models might be fine-tuned to address those more complex care scenarios. Finally, evidence from our study showed that LLMs regularly encouraged further discussion of preventive care topics; thus, clinicians should consider augmenting existing communication practices concerning preventive care [41] with techniques that anticipate the questions that LLMs might inspire among care seekers.

The study’s results also suggest directions for future research. The automated methods used in the study have the potential to be deployed in other avenues of research concerning access to preventive care—for instance, in studies comparing Medicare coverage and USPSTF recommendations [42]. Additionally, if models continue to progress in their consistency with authoritative recommendations and if further research validates that they can be used safely in clinical settings, researchers should consider studying precisely how to embed models in clinical environments to offset the challenges associated with clinicians communicating an increasing number of preventive care guidelines to patients [9]. LLMs offer a promising means of addressing this problem, yet they could be implemented in various ways, such as by having clinicians communicate model outputs, guide patients’ use of LLMs, or encourage unguided, free-form use, among innumerable other methods of embedding models in clinical environments. Studying which of these methods would work best is an important area for future research if subsequent investigations validate the viability of using LLMs in clinical settings to inform individuals about preventive care. Those efforts also could incorporate investigations into how individuals interpret LLM-generated outputs. For instance, should LLM equivocation prove difficult to quell in response to users’ preventive care inquiries, then research studying how individuals respond to equivocation in preventive care recommendations will offer an empirical foundation for predicting and guiding patient behavior.

One final future direction suggested by our study carries import for research on AI evaluation generally, not solely in the realm of studying AI in the context of preventive care. Our study found that LLM raters tasked with broadly assessing a “match” between 2 natural-language passages often will find overall similarity in texts that those same raters would view as subtly different when provided a classification scheme that allows them to capture nuances of the text. This phenomenon resembles scale-format effects in survey research [43], where the granularity of scale units or the number of response categories can affect responses, and its relevance for evaluating LLM responses deserves attention. Our inclination, initially, was to view evidence of equivocation in instances of clear USPSTF recommendations as evidence that LLMs fail to provide advice consistent with USPSTF recommendations. A reviewer’s insightful observation pointed out that avoiding definitive statements and encouraging additional information search have positive benefits in the context of preventive care by, as noted above, empowering patients and encouraging their education while also avoiding the inappropriate replacement of clinical guidance. These important ideas from the reviewer should remind all researchers engaged in AI evaluation that they, too, must think critically about the values and concepts embedded in the criteria used to evaluate AI systems; in our investigation, taking such an approach means recognizing that the broad sentiments of an LLM output can match those of the USPSTF, even if the LLM output avoids making a definitive statement and encourages more information search, as would be appropriate for a nonauthoritative party contributing to preventive care education. Future work that guides researchers on how to recognize such nuances ex ante will enhance AI evaluation research.

These future avenues of research will extend this paper’s more modest objective. Here, using simulated user queries, our study found that LLMs vary in their consistency with USPSTF recommendations, with older models exhibiting less consistency and newer models echoing USPSTF guidance more regularly. Across both older and newer versions, LLMs resist making definitive statements and recommend the pursuit of further information; however, particular prompting approaches can be used to improve LLM-USPSTF concordance and reduce equivocation. Together, these findings indicate that LLMs have the potential to become an important source of individualized and scalable preventive care guidance, albeit with the caution that users should seek out better-performing models and use prompting techniques that yield higher rates of LLM-USPSTF concordance.