This analysis was based on a selection of mental health questionnaires that have been previously investigated with respect to their content overlap [6-11]. This included questionnaires for adult depression (n=7) [30-36], childhood depression (n=15) [30,37-50], CHR-P (n=11) [51-61], mania (n=7) [62-68], OCD (n=7) [69-75], and sleep disorder (n=12) [76-86]. The questionnaires were included based on their frequency in literature, inclusion in reviews, and citation count. The full details about the selection process can be found in the previous publications. A summary of all questionnaires can be found in Table 1. It has to be noted that the Depression and Anxiety in Youth Scale (DAYS) [87], the Multiscore Depression Inventory for Children (MDI-C) [88], the Reynolds Adolescent Depression Scale (RADS) [89], and the Eppendorf Schizophrenia Inventory (ESI) [90] were neither publicly available nor purchasable. Hence, these questionnaires could not be included in the analysis. Further, we did not include the Child Behavior Checklist and Youth Self Report [91] as these are not specifically designed to assess the risk for psychosis.

For each diagnostic domain (adult depression, childhood depression, CHR-P, mania, OCD, and sleep disorder) we identified the respective core symptoms based on the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5) [92]. For CHR-P symptoms, we extracted the core symptoms from the Structured Interview for Psychosis-Risk Syndromes (SIPS) [8]. Thereby, symptom features were listed as individual symptoms (eg, from the DSM-5 item “Feelings of worthlessness or excessive or inappropriate guilt,” the symptoms “Feelings of worthlessness” and “Feelings of guilt” were extracted). Each questionnaire item was assigned to one of the core symptoms by 3 of 10 clinical experts independently (Figure 1A). All experts are working in the field of mental health as clinicians, psychotherapists, and researchers. Questionnaires were assigned to experts according to their field of expertise. Each element of a questionnaire to which a participant or observer must respond was considered as an item. Thereby, subitems were consolidated into one item. Items that were not related to symptoms but to assessment quality (eg, “These answers represent my honest feelings”) were removed from the analysis. If an item assessed multiple core symptoms, symptoms were combined and categorized with a more general symptom (eg, “There is no change from my usual appetite” can refer to both “decreased appetite” and “increased appetite,” which were combined as “changes in appetite”). In case an item could not be assigned to a core symptom from the DSM-5 or SIPS, an additional symptom was added to the list of core symptoms (eg, “I feel very bored” refers to “boredom”).

For each diagnostic domain, 3 clinical experts were independently provided with a list of questionnaire items and were instructed to assign each item to one of the core symptoms. Questionnaire items were assigned to core symptoms based on the highest agreement among expert ratings. In case no clear consensus was reached, the raters discussed and agreed upon the most appropriate categorization. Based on this, all items of questionnaires of the same diagnostic domain were clustered.

To cluster items by underlying symptoms using LLMs, we used 2 approaches. First, sentence embeddings of questionnaire items were derived from the pretrained sentence Bidirectional Encoder Representations from Transformers (sBERT) model “all-mpnet-base-v2” [14] (Figure 1B). The open-access sBERT model, trained on 160 gigabytes of English text corpora, was selected based on its superior performance in sentence encodings across 14 benchmark tasks and its recognition as one of the best validated models [93]. The resulting sentence embeddings of a fixed dimensionality of 768 were clustered using k-means clustering. The number of clusters was set to the same number as in the expert-based analysis. Second, OpenAI’s third-generation Generative Pretrained Transformer (GPT) model “GPT-3.5-Turbo-0125” [94] was used to assign each questionnaire item to a corresponding symptom (Figure 1C). Its wide availability, computational efficiency, and high performance at comparatively low cost made it a suitable choice for this study [95]. Each item was presented to the model alongside the list of core symptoms, and the model was prompted to assign the item to the most appropriate symptom. Based on the model’s responses, items of each diagnostic domain were grouped according to their assigned symptom.

Similarity of clustering within each diagnostic domain, among the 3 experts and between the experts’ highest-agreement cluster (hereafter referred to as expert-based clustering) and the two LLM-based clustering approaches, was quantified using the Adjusted Rand Index (ARI), a widely used metric for evaluating agreement between clustering solutions [96,97]. The ARI ranges from -0.5 to 1.0, where negative values indicate clustering discordance lower than chance, values near zero suggest random clustering, and values approaching one reflect near-perfect agreement. In the absence of established guidelines for interpreting the strength of the ARI, we applied the classification used for correlation coefficients: very weak (0.00‐0.19), weak (0.20‐0.39), moderate (0.40‐0.59), strong (0.60‐0.79), and very strong (0.80‐1.00). First, we evaluated the similarity in ratings for each pair of raters and computed the mean similarity within each diagnostic domain. Second, the similarity between expert-based and both the sBERT- and GPT-based clustering solutions was evaluated across the diagnostic domains using the ARI. Given that both self- and observer-rated questionnaires were included in the analysis and the rating types of questionnaires exert a strong influence on how items are phrased, we additionally examined the similarity between the expert-based clusterings and both LLM-based clusterings separately for rating types. As a control of the sBERT-based approach, sentence embedding vectors were randomly permuted, and the analysis was repeated (1000 repetitions) as described before.

In a second step, the content overlap of questionnaires within each diagnostic domain was calculated based on the method introduced by Fried [6] but using the GPT-based clustering approach. Specifically, for each diagnostic domain, the Jaccard index was calculated for each pair of questionnaires to determine their overlap of content. The Jaccard index is calculated by dividing the number of shared items between 2 clusters by the total number of items present in both clusters. The resulting values range from 0, indicating no overlap, to 1, representing complete overlap. Similar to Fried [6], we defined the strength of the Jaccard index as follows: very weak (0.00‐0.19), weak (0.20‐0.39), moderate (0.40‐0.59), strong (0.60‐0.79), and very strong (0.80‐1.0).

This study did not involve human participants, medical records, patient information of any kind, or secondary data analyses. Thus, the study did not meet the criteria for a review by an institutional review board, and no ethical approval was required.

A total of 23 symptoms of adult depression, 30 symptoms of childhood depression, 40 symptoms of CHR, 29 symptoms of mania, 27 symptoms of OCD, and 45 symptoms of sleep disorder were identified in the questionnaires based on core criteria from the DSM-5 or SIPS (Table S1 inMultimedia Appendix 1). A more detailed graphical representation of the number of items assigned to each cluster by the 3 clustering approaches can be found in Figures S1-6 in Multimedia Appendix 1. It should be noted that defining the expert-based clustering solution based on the highest agreement among experts and limiting the analysis to self-rating (SR) or observer-rating (OR) questionnaires reduced the number of identified symptoms in some of the diagnostic domains (Table 2).

Between the 3 raters, we observed a very strong similarity for adult depression questionnaires (mean ARI: 0.819), a strong similarity for childhood depression questionnaires (mean ARI: 0.616), and CHR-P questionnaires (mean ARI: 0.654), and a moderate similarity for mania questionnaires (mean ARI: 0.597), OCD questionnaires (mean ARI: 0.401), and sleep disorders questionnaires (mean ARI: 0.527) (Table 3, Figure 2). Similarities between individual raters can be found in Figures S1-7 in Multimedia Appendix 1.

Similarity between expert-based and sBERT-based clusterings varied across the diagnostic domains. The ARI ranged from 0.188 (for childhood depression questionnaires) to 0.371 (for adult depression questionnaires), indicating a very weak to weak similarity (Table 3). Including solely SR items, the mean ARI ranged between a very weak (0.143 for mania questionnaires) and weak similarity (0.366 for adult depression). For OR questionnaires, we observed a similarity lower than chance (ARI: −0.023) for childhood depression questionnaires. Across the other domains, we observed ARIs in the range of a weak (0.286 for OCD questionnaires) to moderate similarity (0.589 for CHR-P questionnaires). Irrespective of diagnostic domains and rating types, the mean ARI exceeded the 95% CI indicating agreement above chance level, with the exception of the childhood depression OR questionnaires.

Additionally, we observed a varying similarity between expert-based and GPT-based clustering solutions. We observed a very weak (0.195 for sleep disorder questionnaires) to strong (0.694 for adult depression questionnaires) similarity across the different diagnostic domains (Table 3). When focusing on SR items, the ARI varied from 0.195 (for sleep disorder questionnaires) to 0.651 (for adult depression questionnaires), indicating again a very weak to strong similarity. Including solely OR items, we observed a weak (0.378 for CHR-P questionnaires) to strong (for adult depression questionnaires) similarity.

Using the method introduced by Fried [6] but adapted to the GPT-based clustering approach to assess the content overlap of questionnaires, we observed a weak content overlap for CHR-P questionnaires (mean Jaccard index: 0.344) and a moderate content overlap for adult depression questionnaires (mean Jaccard index: 0.574), childhood depression questionnaires (mean Jaccard index: 0.443), mania questionnaires (mean Jaccard index: 0.419), OCD questionnaires (mean Jaccard index: 0.457), and sleep disorder questionnaires (mean Jaccard index: 0.461). An overview of the observed content overlap within each diagnostic domain is presented in Figure 3. Interactive sunburst plots showing a more detailed overview of the content overlap of the questionnaires in each diagnostic domain can be found on GitHub [98].

As similarity between the expert-based and GPT-based clustering was highest for adult depression questionnaires, we focused on the content overlap of these questionnaires in greater detail. The number of symptoms and the average Jaccard index per questionnaire can be found in Table 4.

A total of 20 symptoms were found in questionnaire items of adult depression questionnaires using GPT-based clustering. Of these, 15/20 symptoms (75%) were found in the CES-D. Around 8/20 symptoms (40%) were identified in the Montgomery-Åsberg Depression Rating Scale (MADRS). Six symptoms were featured across all questionnaires (Figure 3). One symptom was exclusively represented in the BDI-II, while another item appeared only in the Hamilton Rating Scale for Depression (HDRS). Across all adult depression questionnaires, the mean content overlap was 0.574. For a detailed overview of the Jaccard indices, see Table 4 and Figure 4A. The content overlap of Inventory of Depressive Symptoms (IDS) and Quick Inventory of Depressive Symptoms (QIDS) was highest, whereas the lowest overlap was found between IDS and HDRS. Details of symptoms found in questionnaires can be found in Figure 4B.

To the best of our knowledge, this is the first study to use LLMs to assess the content overlap of mental health questionnaires. First, we aimed to compare expert-based clustering of questionnaire items to two observer-independent automatic approaches using LLMs: (1) clustering sentence embeddings generated by sBERT using k-means, and (2) prompting the questionnaire items directly to the GPT model. Across the different diagnostic domains, we found a moderate to very strong agreement between expert raters. Although similarity between expert-based and sBERT-based clustering was above chance, GPT-based clustering was in general more aligned with expert-based clustering except for sleep disorder questionnaires. In a second step, we calculated the content overlap of questionnaires within each diagnostic domain based on the GPT clustering. We observed weak (CHR-P questionnaires) to moderate (adult depression, childhood depression, mania, OCD, and sleep disorder questionnaires) content overlap.

Generally, our findings demonstrate that LLMs such as sBERT and GPT can be effectively used to cluster questionnaire items based on content. Both models have previously been used to assess the validity of questionnaires and surveys [24,25,29]. Using a fine-tuned sBERT model to generate semantic embeddings of the International Personality Item Pool questionnaires, Wulff and Mata [29] demonstrated that these embeddings can predict a questionnaire’s empirical internal structure, convergent and divergent validity, and detect its structural fidelity. Similarly, the fine-tuned sBERT model SurveyBot3000 was able to infer correlations between questionnaire items and the intercorrelation of questionnaires of the American Psychological Association (APA) PsycTests corpus [24]. Further, Huang et al [25] used a GPT model to generate semantic embeddings of two gratitude questionnaires. Similarity between each questionnaire pair was then clustered to infer semantic similar items and identify redundancy within questionnaires. Our study extends these applications by highlighting LLMs’ utility in assessing content overlap in mental health questionnaires. Notably, we found that directly prompting items to GPT is particularly effective for generating meaningful content-based groupings of questionnaire items. Similarly, Petukhova [16] observed that GPT models outperform static embedding models like sBERT in clustering and semantic understanding tasks. Specifically, when combined with prompt engineering, GPT models offer advantages in generative reasoning over the fixed representations generated by sBERT [23]. Compared to methods that require generating and clustering embeddings from static models like sBERT, GPT prompting is not only more flexible but also easier to implement, as it does not require additional preprocessing or fine-tuning. Thus, GPT prompting appears to be a promising tool for rapidly obtaining an overview of the content overlap in mental health questionnaires. This application will likely further improve with future GPT models, as newer versions show increased ability to detect psychological constructs [99].

Compared to previous studies, we observed a higher overlap between the questionnaires with the exception of sleep disorder questionnaires [6-11]. This is likely due to differences in the number of identified symptoms. Compared to previous studies, we identified fewer symptoms across all diagnostic domains, which may have resulted in a less fine-grained analysis and, consequently, an increased apparent content overlap. However, using more nuanced symptoms was not feasible, as some items lack specificity; for example, “I sleep a lot less than usual” can reflect both initial and middle insomnia. Nevertheless, we observed a substantial heterogeneity between questionnaires across all diagnostic domains, which strengthens previous observations of content divergence between questionnaires. Such heterogeneity of content is not inherently problematic. On the contrary, variability across questionnaires can be beneficial; it allows clinicians and researchers to capture complementary information, adapt to different contexts, and contribute to scientific advancement [100,101]. However, to fully leverage this variability, it is essential to understand the distinct strengths and applications of each questionnaire. This study demonstrates a method for facilitating the identification of content overlap, thereby making this process more accessible.

Focusing on the content overlap of adult depression questionnaires in greater detail revealed that the overlap of questionnaires is moderate (0.574), indicating that the results from one questionnaire may only partially generalize to others. In line with the observations of Fried [6], the content of the QIDS demonstrated the highest mean similarity with other questionnaires, but we observed the lowest mean similarity for the HDRS, not the CES-D. Related to this, we identified idiosyncratic items in the HDRS (and BDI-II) but not in the CES-D. Several studies have reported that the HDRS and BDI-II capture idiosyncratic symptoms [4,6,101]. In line with Fried [6], our GPT-based clustering identified only items related to feelings of punishment in the BDI-II, whereas items related to hypochondriasis appeared exclusively in the HDRS. However, additional idiosyncratic symptoms were identified in other questionnaires, and these differ from those reported by Shafer [4]. These differences may again be attributed to a less fine-grained symptom structure. Overall, our findings demonstrate that GPT-based content overlap analysis of questionnaires is not only feasible but also fast and computationally efficient, making it a promising approach for large-scale questionnaire comparisons. In the future, this approach will facilitate an extended investigation of questionnaire content overlap by incorporating additional instruments and enable broadening the analysis to questionnaires assessing other mental health disorders, such as anxiety disorders or posttraumatic stress disorder.

Several limitations have to be addressed. First, the number of items and symptoms was not consistent across the diagnostic domains. The discrepancy in the number of identified clusters may have influenced the results. However, reducing the number of clusters, and thus the number of identified symptoms, would decrease the specificity of symptoms and thus lower the accuracy of the content overlap analysis [6]. Further, the semantic representation of an LLM is influenced by its training data [25]. Both LLMs used in this study were trained on publicly available texts (eg, web pages, books, or Wikipedia) but not clinical datasets (eg, diagnostic manuals). This might have influenced the model’s representation of questionnaire items. However, both LLMs were shown to contain reliable general psychiatric knowledge and were able to classify mental health conditions without fine-tuning [102-104]. In future work, it would be valuable to compare the clustering performance of pretrained and fine-tuned models. Moreover, while the present study as well as several previous publications indicate that semantic embeddings can capture some aspects of medical terms and psychological concepts [26,102,105], it needs to be acknowledged that such LLM-based representations are most likely far from complete and can potentially be biased due to a number of reasons [106]. Thus, although LLMs can facilitate efficient and low-effort creation of questionnaire content, human expertise remains essential for reviewing and interpreting their outputs. Finally, only English-language questionnaires were included in this study. Whether LLMs can effectively cluster questionnaire items in other languages remains to be tested.

In summary, our study demonstrates the feasibility of using LLMs to assess the content overlap in mental health questionnaires. In particular, prompting GPT models provides a novel and objective approach for evaluating similarity across questionnaires. It is important to note that human expertise remains essential for reviewing and interpreting the outputs of LLM. Although our findings differed somewhat from previous content overlap analyses, these differences were not substantial. Future content analysis could benefit from LLMs fine-tuned on a corpus of psychological text data. Nonetheless, this study demonstrates a novel application of LLMs in the field of mental health research.