Biomedical natural language processing (NLP) methods aim to support clinical research and practice, with suggested uses spanning information retrieval, medical documentation generation, and patient data analysis [1,2]. As language has been studied as a marker for conditions such as depression [3], schizophrenia [4,5], and Alzheimer disease [6], mental health–oriented tasks have become popular in NLP. Notably, methods to analyze patient-authored texts or medical records as indicative of a given condition or symptom have been developed, with the underlying hypothesis that this could translate into clinically useful tools for diagnosis or large-scale screening [7-9]. A number of these methods are now based on large language models (LLMs), which recently garnered global attention from the public, industrials, and researchers for their assumed general language capabilities.

However, it is increasingly documented that LLMs can produce unreliable results and potentially harm users by perpetuating and amplifying bias [10-14], which has been defined as “the presence of systematic errors or disparities within decision-making processes that disproportionately affect specific subgroups” [11]. While current medical practice is admittedly not exempt from bias itself [15], automated systems such as LLMs amplify existing stereotypes, which can lead to unjustified differences in simulated diagnoses and treatment plans for vignettes of patients with different ethnicities and genders [13]. Relatedly, it has been reported that male profiles are overgenerated by LLMs (regardless of the true prevalence of the considered condition) both in English and in French [13,14]. Despite substantial research efforts, no reliable mitigation technique has been proved to fully resolve these issues [16,17], which may be intrinsic to LLMs [18].

Moreover, existing evaluation settings for biomedical LLMs are limited [10,19] and real-world impact studies are still lacking in the NLP community [20], leaving many ethical and regulatory challenges unaddressed [21]. As a result, it is still unclear to what extent these LLM-based systems are clinically “[useful] in improving patient outcomes, informing clinical decision-making, and optimizing health care resources” [22] in mental health. In addition, computer scientists and NLP engineers creating such systems may lack the medical expertise needed to make accurate design choices. It seems therefore crucial to methodically audit LLM-based systems both for bias and clinical utility before considering downstream clinical deployments.

Existing reviews have investigated the applications, benefits, and risks of LLMs in health care [23] and mental health [24-26], highlighting general ethical challenges such as data privacy, fairness and bias, clinical integration, and ethical governance. There is, however, a lack of literature that explicitly connects these different issues and systematically assesses them throughout the development steps of the considered systems, taking a distance from indicators of (possibly ill-defined [27]) performance. Notably, although they are crucial to understand downstream risks, existing bias studies in health-related scenarios generally posit bias as a single-dimension construct (eg, gender or racial bias) [17,28], using LLMs as readily available tools whose development is left unquestioned. Yet the decision-making processes from which bias can stem are numerous, which calls for a bias investigation throughout the entire development pipeline of the considered system [29,30]. Similarly, the limitations and barriers to the clinical implementation of NLP technologies [19,31] tend to be listed as post hoc challenges of the systems rather than connected with underlying design choices. In this review, we consider bias and clinical utility not only to be both related and important issues, but to be complementary when it comes to implementing (or envisioning the possible implementation of) any automated system in real-world health care scenarios. Indeed, both these dimensions (How biased will the final system be? How clinically useful?) depend greatly on methodological choices spread throughout the development of the system.

Building on these considerations, we propose to study bias and clinical utility as 2 intricate dimensions rooted in methodological choices, with expected downstream social impact on users. To our knowledge, this is the first such extensive, large-scale joint analysis of bias and clinical utility for predictive mental health applications based on LLMs.

The objectives of this work are 2-fold. First, we wish to broadly describe the methodological decisions adopted in publications on LLM-based mental health prediction. This will lead us to analyze possible bias and barriers to clinical utility stemming from these methodological choices, at each step of the system’s development pipeline.

Second, we intend to document the awareness of researchers on matters of bias and clinical utility, as indicated by relevant reported considerations in these same publications. Indeed, increased global research attention on these issues does not necessarily translate into actionable plans for other researchers [32]. Furthermore, propensity to bias is still absent from reference evaluation benchmarks used to evaluate these systems, which shows limited appropriation of these issues within the community.

Drawing on these insights, we mean to foster discussions as to the extent to which it is currently possible, safe, and desirable to integrate LLMs into clinical routine and mental health care in particular.

This scoping review was preregistered in the Open Science Framework registry [33]. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines [34] (refer to Checklist 1).

We analyzed original research articles published in conferences or journals between January 2019 (to account for the release of BERT [35]) and December 2024. Eligible studies had to (1) assess the presence or severity of a mental health condition; (2) use, for that purpose, an NLP system comprising at least one LLM; and (3) use nonsynthetic textual samples as input data, possibly along with synthetic samples or other modalities. We excluded preprints, reviews, and studies which were not openly available online, as well as papers not written in English.

Importantly, following criterion (1), articles claiming to perform only stress detection or sentiment analysis were not considered eligible. Regarding criterion (2), as it can be underspecified in the literature which models fall under the designation of “LLMs,” we followed the recommendation of Rogers and Luccioni [36] to provide an explicit working definition. In accordance with standard considerations [37], we therefore called LLMs NLP tools that (1) model and can generate text, (2) receive large-scale pretraining, (3) can be used for transfer learning, and (4) rely on the Transformer architecture [38]. Notably, this definition includes both encoder models such as BERT [35] and generative decoder models such as GPT [39], in line with existing reviews about applications of LLMs to mental health [24]. Finally, although criterion (3) allows for the inclusion of papers developing systems for audio or imaging data, our analysis was oriented toward the LLM-relevant part of these systems, that is, the one processing text. Besides, even if synthetic datasets are increasingly praised for clinical research [40], we wanted to focus in this review on systems that fulfill at least partially expectations of pending deployment—generally comprising testing in realistic conditions which are, up to now, only imperfectly proxied by generated corpora [41,42]. In addition, our framework comprises a step which focuses on the included data, considering demographics of included participants, which is not applicable to synthetic datasets.

In order to provide a wide coverage of relevant publications, we searched 5 databases spanning biomedical, NLP, and more general AI-related literature: MEDLINE (PubMed), Web of Science, IEEE Xplore, ACM Digital Library, and the ACL Anthology. All searches were performed on January 20, 2025, with queries following the template (terms related to “MENTAL HEALTH”) AND (terms related to “LARGE LANGUAGE MODELS”) AND (terms related to “PREDICTION”). The queries were crafted following multiple search rounds and designed to minimize false negatives; hence, the large number of allowed terms in each keyword group (request details are presented in Multimedia Appendix 1). When allowed by the database search engine, we used filters for date, publication language, and publication type according to the eligibility criteria listed above.

The initial set of records identified via database search was loaded into the Rayyan software (Mourad Ouzzani) [43]. A sample of 100 records was first screened based on title and abstract by 3 reviewers (CB, AN, and KF) independently. Interannotator agreement was high, with pairwise Cohen κ [44] scores ranging from 0.654 to 0.807 [45], while resulting discussions helped clarify the screening strategy (see “Eligibility Criteria” section). Subsequently, the first author (CB) screened the remainder of records, as well as the full text of all the eligible studies before conducting extraction. Exclusion reasons and record counts at every step are detailed in the flowchart. Remaining doubts were addressed through discussion between authors.

In accordance with our research objectives, we extracted 2 types of information in the reviewed papers: methodological information (what do the authors do?), and authors’ self-reflections and acknowledgments linked with bias and clinical utility (what do the authors say about what they do?). The full-text PDFs of included studies were loaded into Zotero (Corporation for Digital Scholarship) [46] for reading, alongside an auxiliary spreadsheet document for qualitative entity extraction performed by CB.

To our knowledge, there is no existing framework specifically designed to study both bias and clinical utility in the conception pipelines for LLMs applied to mental health. However, 2 previous approaches, which we deem complementary to analyze bias and clinical utility at every step of the development pipeline of an LLM-based system for mental health prediction, were proposed.

On the one hand, Hovy and Prabhumoye [29] identified 5 sources of bias in NLP systems: the data, the annotations made on the data, the input representations fed to NLP models, NLP models themselves, and larger research design choices such as the treated language. On the other hand, Chen et al [30] sketch a 5-step pipeline of ethical stakes for machine learning (ML) in health care, considering (1) the unequal distribution of funding and research attention between health issues with regards to the share of the world population that they affect (problem selection), (2) nonrepresentative data (data collection), (3) imperfect medical proxies for target outcomes (outcome definition), (4) ML models optimization parameters potentially exacerbating bias (algorithmic development), and (5) blindness to issues such as generalizability to various clinical settings, downstream impact assessment or regulatory compliance before real-life system deployment (postdeployment considerations).

Inspired by these 2 previous approaches, we propose a 5-step framework (Figure 1) to study both bias and clinical utility within the selected articles. We identify key arguments of each work, and define qualitative entities to extract from papers for future analysis. For instance, Chen et al [30] discuss the importance of measuring performance metrics across demographic groups to ensure fair treatment among patients, which we operationalize as a binary entity “per-group performance” to be set to “true” when authors report disaggregated results (eg, against sex or gender, race or ethnicity, or other sensitive attributes). This entity relates indeed both to bias (as some groups may be discriminated against if the system performs poorly for them) and to clinical utility (as clinical outcomes are expected to be fair among patients). We redirect the reader to Multimedia Appendix 2 for more details on our analysis framework design and the operationalization of entities.

We also collected sentences from the included articles that refer explicitly to any sort of bias or to elements related to the clinical utility of the developed system, as a proxy for self-reflection of researchers regarding these matters. We extracted these statements independently from the arguments they defend (eg, recognizing the presence of bias in presented results vs stating that bias has been mitigated), within our extraction framework.

We performed the analysis of the extracted data in a Jupyter Notebook (Project Jupyter) environment, using Python libraries (Guido van Rossum). When applicable, usual descriptive statistics (distribution of values, mean value, range, and so on) were computed. In some cases when necessary conditions were met, we modeled the effect of categorical variables (eg, the effect of medical affiliation on the studied conditions) using chi-squared contingency tests (χ²) computed with the SciPy Python package (without correction). The entities for which full sentences or noisy outputs were extracted were further analyzed by clustering the extracted data into meaningful categories, in an iterative and inductive aggregation process inspired by grounded theory [47]. For authors’ self-reflections about bias and clinical utility, we used the same approach to map the highlighted sentences to themes. The final results (extracted entities or categories obtained after coding for each paper) following data analysis are presented as a spreadsheet in Multimedia Appendix 3.

We initially identified 2646 candidate records in databases (search performed on January 20, 2025). After deduplication, 2472 unique papers were screened for eligibility based on title and abstract. The remaining 263 papers were then screened based on full-text read. Finally, 201 (8.1%) studies were included in this review. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram of the process is detailed in Figure 2, and the full list of included papers is available in Multimedia Appendix 4.

Included studies (N=201) are mostly recent: 149 (74.1%) are published in 2023‐2024, 46 (22.9%) in 2021‐2022, and only 6 (3%) in 2019‐2020. Most of the retrieved studies are sourced from IEEE Xplore (124/201, 61.7%) and PubMed (57/201, 28.4%). The ACL Anthology, the Web of Science, and the ACM Digital Library, respectively, raised 10 (5%), 7 (3.5%), and 3 (1.5%) papers.

We found mentions of bias and clinical utility-related themes in 164/201 (81.6%) papers. The pipeline step that triggered the most discussion is that of data collection (107/201, 53.2%), followed by model development (69/201, 34.3%), and problem selection and research design (60/201, 29.8%). Themes associated with outcome definition and postdeployment considerations were only evoked in, respectively, 45/201 (22.4%) and 31/201 (15.4%) papers. In 77/201 (38.3%) papers, the discussion was restricted to a single step of the pipeline. However, 46/201 (22.9%), 26/201 (12.9%), and 10/201 (5%) papers evoked themes associated with 2, 3, and 4 of these steps. A total of 5 (2.5%) papers mentioned considerations from all 5 steps of the pipeline [48-52].

With this study, we introduced a framework to analyze methodological bias and clinical utility throughout the development pipeline of LLM-based mental health prediction systems. Our review notably sketched the prototypical profile of such a prediction system: it focuses on depressive disorders, leverages English data collected from social media, and uses BERT-based classifiers in the absence of a clinically useful outcome for the included participants. In what follows, we argue that this is problematic for multiple reasons.

First, the focus on depressive disorders leaves nearly untouched other condition families, echoing the observations of Wang et al [26]. Although it is true that they account for a large part of the world’s mental health burden, millions of individuals are also affected by anxiety disorders, bipolar disorder, schizophrenia, or eating disorders, to name a few [105]. Besides, it seems that communication disorders (affecting individuals’ language abilities) could also particularly benefit from NLP inputs [106]. In addition, despite a relative diversity among affiliation countries, papers almost exclusively work on English data, which exacerbates the existing bias toward English-based systems in NLP [29]. This bias often goes unnoticed, with nearly half of papers working on English omitting to mention it explicitly, a phenomenon that has been otherwise measured in NLP conferences [107,108]. In order to benefit a wider range of individuals, non-English or multilingual approaches should be considered.

Second, the data used in the studies largely come from social media (which was also observed by Wang et al [26]), even more so when no domain authors are involved. Some authors highlight the ease of access and collection of such data (as opposed to “traditional” clinical data), its alleged relevance to the task of mental health detection, and its wide accessibility (eg, “social media’s ubiquity presents a platform for individuals to express their feelings, instead of traditional, formal clinical settings, with 8 out of 10 people disclosing their suicidal thoughts and plans.” [109]). However, social media users do not make up representative samples of individuals, and it is difficult to ensure the quality of social media posts [30]. This is furthermore problematic as demographic information is generally absent from these datasets, and users are rarely able to provide consent for the collection of their data. In order to avoid harming users, social media research should comply with ethical guidelines as regards consent of participants, data deidentification, and sharing [110].

Third, the proxies used to account for participants’ alleged mental health status lack clinical grounding. It is unlikely that heuristics based on keywords within texts or self-administered questionnaire scores would be considered reasonable indicators by clinicians to validate diagnostic labels. This leads us to question the maturity of the field for clinical integration.

Fourth, authors tend to overlook practical requirements when adopting a model. We note that popular, nonspecialist models (and, increasingly, generative LLMs) are generally preferred, yet these choices are rarely clinically motivated. It is currently debated whether specialist models can actually be competitive with bigger, more recent generalist models [111] on specialized downstream tasks. However, the size and eventual proprietary status of the latter hinder nonnegotiable needs of data privacy and control over computational cost in clinical environments with limited resources, along with issues of environmental impact and models’ propensity to bias [112,113].

Finally, there is an overall lack of awareness of the ethical issues implied by a possible clinical deployment of the considered systems. Importantly, some works seem not to aim for such clinical use cases and deny that their systems should be used as diagnostic tools already. Yet this leads to questioning the clinical utility of tasks confined to computer science laboratories, whose results may be mistakenly interpreted as clinical conclusions. Overall, authors mainly initiate discussion themes that focus on generic ML issues related to model constraints at the model development step (eg, input length constraints, computational cost seen as a practical barrier and not an environmental problem), while revealing a data-centric view of bias around the data collection step (eg, calling for bigger, class-balanced datasets). This technical approach to bias, which calls for technical solutions, has been previously identified as the engineering ethos [114]. However, such complex issues as biases cannot be solved with a sole technique [12], if at all [18]. In what follows, we argue that an ethical approach to LLMs applied to the domain of mental health would benefit from more interdisciplinarity.

In addition to the biases identified above, our observations raise the question of interdisciplinarity and its influence on the clinical utility of proposed solutions. Indeed, while we stressed global shortcomings, it should be acknowledged that studies were more robust in terms of disorder diversity, specificity of disorder definition, and quality of data sources (clinical interviews and electronic health records) when at least one author had a medical affiliation. This aligns with previous findings that constituting interdisciplinary teams (including clinicians and computer scientists) improves results validity in ML problems [115].

For instance, clinical expertise is needed to adequately define what is referred to as “depression” in studies. Some papers ambiguously assimilate the concepts of “depression” and “negative mood/sentiment” (eg, “the task is to discover the mood of the user” [116]), or “depression” and “suicidal risk” (eg, Wang et al [117] guidelines for depression level estimation are actually based on reports of suicidal ideation and plan), or do not seem to make a conceptual difference between self-diagnosed depression and clinical diagnoses produced by health care professionals. The task of disambiguating what is designated by “depression” is all the more complex because according to MeSH definitions, it can cover a sign or symptom (“Depression” [118]) or a disorder with different levels of severity (“Depressive Disorder” [119] or “Major Depressive Disorder” [120]). These differences are rarely described or implemented in data annotation protocols in studies, even though they play a crucial role in diagnostic reasoning and patient care. Consequently, training a system to automatically detect “depression” and claiming it is ready for use in clinical practice is suboptimal at best (what kind of “depression” does the system detect?), and misleading if users are left to make their own assumption of what concept of “depression” is operationalized. The close collaboration between computer scientists and medical doctors is thus required to ensure and to propagate the clinical value of the targeted outcome.

Interdisciplinary collaborations between diverse fields can also improve the robustness of systems’ evaluation. Indeed, while some studies warrant caution before real-world deployment, others present their system as deployment-ready (even in the absence of such validation), while a majority do not address the question at all. Notably, no included study presented a rigorous clinical validation for an LLM-based prediction system to be applied in practice, for example, through a randomized controlled trial [20].

Extending even beyond sole clinical validation, interdisciplinary frameworks for the validation of digital devices in medicine have recently been published. For instance, the certification of digital devices—and particularly those embarking LLMs—by the US Food and Drug Administration or the European Medicines Agency requires them to comply with the criteria of the V3 framework [121]. This framework requires rigorous clinical validation (ie, demonstrating that the metrics computed by the device correlate with meaningful physiological or clinical dimensions), but also rigorous hardware verification (validation of the sensor, if there is one) and analytical validation (ie, demonstrating that the algorithm indeed measures a behavior or a physiological marker).

This validation, however, does not ensure that the device, in its context of use, will be useful. Indeed, for most of the articles included in this review, the motivation for the research often remains vague, although untested potential use cases are described, such as large-scale mental health screening or patient monitoring. For others, very few contextual elements help grasp the medical stakes, which we believe have to be mentioned in such research. Is this to say that mental health prediction could become yet another LLM-equipped task, ensuring reasonable chances of getting a paper published?

Interestingly, an updated version of the V3 framework has been recently introduced: the V3+ framework, completing the 3 previous validations by a criterion of usability validation [122]. Although it differs from clinical utility, the addition of utility validation incorporates considerations regarding the projected use, the target population, and the intended context into the validation of digital health devices. Clarifying these elements, which requires interdisciplinary consultation between computer scientists (regarding technical validity) and clinicians (regarding usage), would thus prevent misunderstandings between the two communities, thereby avoiding empty promises fueled by each community’s overclaims regarding the capabilities of the other—particularly regarding LLMs’ ability in clinical context [123].

While this scoping review focuses on 201 papers published between 2019 and 2024, this period covers both the introduction of masked (eg, BERT) and autoregressive (eg, GPT) language models, which provides an opportunity to observe how they were used for mental health prediction early on. In addition, we believe that the large number of included studies allows us to provide a representative snapshot of the field in this 6-year window, through an original joint analysis of bias and clinical utility. While more recent articles could have been included, we assume that they would only have had a marginal effect on the trends we evidenced. Notably, Reiter [20] recently reported that as of March 2025, only 0.1% of papers from the ACL Anthology provided any form of downstream impact evaluation, echoing our findings in the case of mental health prediction.

Our approach is that of a scoping review, aiming at presenting a representative view of what researchers or clinicians may encounter when looking for LLM-based mental health prediction works. As such, we did not perform quality assessment of the articles at the screening stage [124]. In addition, due to the large number of included studies, we could not ensure an independent extraction by multiple reviewers, which may introduce bias in the presented results: as for other phases of the review, this was mitigated with regular discussions and methodological refinements. Since we considered a large number of qualitative entities, methodological choices were necessary to extract them into quantitative metrics. For instance, authors were labeled as “domain authors” whenever they reported being affiliated with a medical institution, a medical university department or school, or a company providing medical devices or services. While we acknowledge this to be an imperfect proxy for medical expertise, our data are freely available in Multimedia Appendices 1-4 for interested researchers to replicate our results with different methodological choices.

Finally, this study was conducted by a group of French, White, NLP researchers, including some with a background in medical informatics or psychiatry. Although we have aimed at adopting an open and interdisciplinary vision, taking into account issues related to bias and clinical utility, we may have missed some relevant aspects to these questions, starting with the phase of article identification. Because this is the main language used for research dissemination, we reviewed only articles written in English in major scientific databases, which nonetheless allowed us to retrieve studies working on other languages. We intentionally crafted broad queries to minimize false negatives, and considered multiple literature sources relevant to NLP and medical sciences. Yet, different requests and additional sources may retrieve slightly different results. Also, information about eventual clinical deployments of the considered systems outside of what is reported in the papers may have been missed.

This review focused solely on LLM-based mental health prediction. We did not consider interventional applications (eg, chatbots for mental health support), nor other medical domains, and some of our observations may not generalize. Nevertheless, we believe that some of our findings are part of a broader picture. Our study suggests that over the past 6 years, the perspective of downstream usage (and users) has seldom been considered in the development of LLM-based, predictive mental health applications. This echoes trends that have been reported in NLP [20]. The overemphasis on algorithm development [125] and quantitative, benchmark-based competition for performance [126] poses a risk of harm by assuming medical applications are “yet another task” to tackle despite increased ethical risks. Finally, it would be interesting to study more deeply the “promises” and narratives around AI and NLP for (mental) health care (eg, the promise to assist health care workers without replacing them, the promise to be cost-effective, and so on), which is left for future research.

LLMs are increasingly used in mental health prediction tasks. Following a 5-step pipeline as a framework to analyze NLP proposals, we described an emerging field that exhibits some marked tendencies, notably toward the detection of depressive disorders and the use of social media data in English. We showed that researchers are generally aware of some bias and clinical utility–related issues, but that reports in papers are sparse and not systematic. In particular, postdeployment considerations and impact studies are lacking, which leads us to question the idea that LLMs have the potential to revolutionize mental health care. When dealing with NLP-assisted mental health research, we advocate for a combined view of bias and clinical utility, implying that no biased system can be clinically useful, and vice versa. This requires interdisciplinarity and careful efforts all along the research pipeline. We strongly believe that the goal to contribute to mental health research and, ultimately, to benefit patients should remain central in NLP efforts towards mental health.