Clinical trials are crucial for developing new therapies, but they require significant resources and can introduce delays in drug development, leading to increased costs [1,2]. Complex and restrictive eligibility criteria hinder patient enrollment, impacting target goals, timelines, and ultimately patient well-being [3-5]. This issue is particularly notable in cancer trials with poor recruitment and high failure rates [6-8] because >80% of the trials fail to meet their initial target accruals and timelines [6,9]. In addition, overly restrictive eligibility criteria limit the representation of the broader patient population, reducing real-world applicability and treatment impact [10-13]. Nonetheless, the practice of trials reusing complicated eligibility criteria without a clear rationale is a common one [14], despite the minimal impact on trial outcomes [15]. Liu et al [15] demonstrated that broadening eligibility criteria using a data-driven approach can benefit initially excluded patients. A comprehensive and standardized eligibility criteria knowledge base that is compatible with real-world data can address these challenges. Such a knowledge base optimizes trial protocol design, improves patient enrollment, enhances the reliability and applicability of evidence synthesis, and fosters the efficient development of new therapies. Furthermore, it enables opportunities such as generating synthetic control arms (SCAs) for single-arm clinical trials using electronic health records (EHRs) [16-18].

The importance of semantically representing eligibility criteria interoperable with EHRs has been highlighted in multiple studies [19-21]. Converting free-text eligibility criteria to computable formats poses challenges, addressed by a range of natural language processing (NLP) techniques and transformer models [22-26]. An NLP interface, Criteria2Query, enables computable queries for eligible cohort identification using EHRs [27]. This tool supports clinical trial knowledge base development, enhancing EHR interoperability and scalability for efficient eligibility criteria knowledge engineering [28]. Manually annotated data sets such as “Chia, a large annotated corpus of clinical trial eligibility criteria” [29] and the “Leaf Clinical Trials corpus, the largest and most comprehensive human-annotated corpus of publicly available clinical trials eligibility criteria” [30] have significantly enhanced NLP model training and the development of effective query structures. Despite significant progress in bridging the gap between eligibility criteria and EHRs, limitations persist in accurately representing the granularities of eligibility criteria and real-time eligible patient number checks [20,31,32]. Using varying hierarchical levels of medical concepts, whether as hypernyms or hyponyms, presents one of the challenges when aligning eligibility criteria with EHRs; for instance, numerous trial eligibility criteria use hypernyms, which encompass a group of related medical concepts, such as cardiovascular disease. Conversely, the patient problem list within the EHR specifies particular medical conditions or diseases (hyponyms), such as myocardial infarction. Establishing a standardized eligibility criteria knowledge base by transforming ambiguous hypernym concepts into computable hyponyms can enhance optimizing trial protocol design and identifying eligible patients through seamless integration with EHR data.

In this study, we aim to create a standardized eligibility criteria knowledge base that seamlessly integrates with EHRs. By using deep learning–based NLP techniques, hypernym concepts in eligibility criteria will be converted to their EHR-compatible hyponyms along with their corresponding values. In addition, the prototype user interface will be developed as a pilot study, enabling the data-driven optimization of clinical trial protocols and the identification of eligible patients through the integration of the eligibility criteria knowledge base and EHRs.

We obtained the data from ClinicalTrials.gov, specifically industry-sponsored phase 2 or 3 interventional clinical trials initiated between January 2013 and May 2020. A total of 3281 trials were identified: 817 (24.9%) for non–small cell lung cancer (NSCLC), 649 (19.78%) for prostate cancer (PCa), 1057 (32.22%) for breast cancer (BCa), 447 (13.62%) for multiple myeloma (MM), 160 (4.88%) for ulcerative colitis (UC), and 151 (4.6%) for Crohn disease (CD).

For the development of the prototype interface, we selected a subgroup of patients (n=2775) diagnosed with NSCLC from a previously curated cohort of patients with lung cancer. This cohort was established using the data from Mount Sinai-Sema4 Health System data [33], and patient information was deidentified for the purposes of this study.

Our NLP pipeline consists of 3 modules: ontology construction and manual annotation, model training and pipeline evaluation, and application.

The standardized knowledge base was constructed in an EntityGroup-AttributeName-Value format, involving 2 key steps: attribute normalization and transforming hypernyms to hyponyms with corresponding values.

We developed a prototype interface using the R programming language (R Foundation for Statistical Computing) and the Shiny package to enhance trial protocol design optimization. The interface allows users to simulate the number of eligible patients based on specific criteria, including a combination of criteria such as histology, stages, laboratory test values, performance scores, prior line of therapy, and comorbidities. For this pilot study, a subset of patients with NSCLC (n=2775) was selected and deidentified. To ensure consistency and accuracy, we standardized the sample entities found in both the eligibility criteria knowledge base and EHRs using concept codes such as the International Classification of Diseases; Logical Observation Identifiers, Names, and Codes (LOINC); and normalized medical prescription codes. In addition, we converted the patients’ absolute laboratory test values to either the upper limit of normal (ULN) or the lower limit of normal based on the provided normal ranges for each specific test. Tables 2 and 3 and Textbox 1 present some examples of normalized concepts and their codes.

The interface uses a rule-based algorithm to match patients’ EHR data with the criteria. The comprehensive rules for matching EHR data with criteria have been described in our previous studies [37]; for instance, we defined the following rules to map each laboratory test in EHRs to 1 corresponding LOINC code:

We associated medication classes with their respective medications; for instance, we extended the annotation “post-menopausal not older than 60 years and taking LHRH [luteinizing hormone–releasing hormone] agonist” to include “post-menopausal not older than 60 years and taking goserelin, leuprolide, or other LHRH agonists.” To achieve this, we used both our in-house knowledge bases and standard resources, such as the National Comprehensive Cancer Network’s Clinical Practice Guidelines in Oncology.

Users can specify different criteria and combinations, such as different laboratory test values with specific conditions such as no brain metastasis to determine the number of qualified patients. The algorithm matches each patient’s EHR data with the selected criteria and calculates the number of matched patients for each criterion. The performance of the interface was evaluated by comparing it to the manual patient selection process conducted by experienced clinical domain experts.

This study was confirmed and approved by the Program for the Protection of Human Subjects at the Mount Sinai School of Medicine (IRB-17-01245)

Our analysis of cancer clinical trials revealed that hormone therapy was the most frequently applied modality (1470/2970, 47.37%), primarily in BCa and PCa trials, followed by targeted therapy (753/2970, 25.35%) and immunotherapy (691/2970, 23.26%). Chemotherapy alone was used in 3.8% (113/2970) of the clinical trials. We developed an eligibility criteria ontology applicable to all cancer trials by manually analyzing 425 eligibility criteria (Figure 1). Entities were categorized into 10 primary groups (inside the blue dotted box) and 9 modifier groups based on semantic types and relations. Entities falling outside the blue dotted box were classified as other observation. The inclusion criteria mainly involved entities in the demographic, diagnosis, laboratory test, and vital groups, while the exclusion criteria commonly included entities in the comorbidity, procedure, and other medication groups. Entities in the biomarker, prior therapy, and disease status groups appeared in both the inclusion and exclusion criteria. Relationships originated from the primary groups and terminated in the modifier groups, except for the has outcome relationship, which started and ended in the primary group (Figure 1). To assess the applicability of the cancer eligibility criteria ontology in a different disease context, we conducted a manual analysis of 60 trials related to UC and CD. For reference, the computable formats of the manually annotated 485 trials can be found in Multimedia Appendices 4-8.

To evaluate the quality of our NLP pipeline, we computed precision, recall, and F₁-score measures. For the primary group entities, the average scores were 0.91 (precision), 0.79 (recall), and 0.83 (F₁-score). Table 4 presents the range of precision, recall, and F₁-score values of 17 primary group entities.

The integrated named entity recognition and relation model extracted 9090 NSCLC, 7427 PCa, 10,217 BCa, 6803 MM, 1565 CD, and 1586 UC entities along with their attribute relations. After normalization and manual curation processes, the eligibility criteria knowledge base for each disease type was established in the EntityGroup-AttributeName-Value format (Multimedia Appendices 9-14). The number of unique EntityGroup-AttributeName-Value combinations varied across disease types, with 494 from 817 NSCLC trials, 471 from 649 PCa trials, 525 from 1057 BCa trials, 389 from 447 MM trials, 231 from 160 UC trials, and 230 from 151 CD trials. Notably, UC and CD trials had a smaller number of unique EntityGroup-AttributeName-Value combinations compared to cancer trials, indicating the presence of more complicated eligibility criteria in cancer trials.

Figure 2 and Table 5 show the distribution of EntityGroup-AttributeName-Value combinations in each primary group from different diseases and provide examples. The laboratory test, prior therapy, and comorbidity groups exhibited a high number of EntityGroup-AttributeName-Value combinations, followed by the biomarker and other medication groups. Variations were observed between solid cancers and hematologic cancers, with higher numbers of EntityGroup-AttributeName-Value combinations in solid cancer types for prior therapy and biomarker, while laboratory test and comorbidity were comparable. The diagnosis group exhibited varying numbers of EntityGroup-AttributeName-Value combinations across all 4 cancer types (BCa, MM, NSCLC, and PCa). EntityGroup-AttributeName-Value in the biomarker, diagnosis, and prior therapy groups were specified per indication, while shared EntityGroup-AttributeName-Value were found in other primary groups.

Our study investigated the impact of various criteria on the number of eligible patients. We developed a prototype interface that uses real-world patient information. Using a subset of deidentified cohorts of patients with NSCLC (n=2775), we deployed an eligibility criteria knowledge base that we had constructed in the interface. Figure 5A displays the selected criteria list, Figure 5B shows the corresponding patient number, and Figure 5C illustrates the distribution of patient numbers in each group.

Sequentially incorporating criteria such as nonsquamous histology and stages III and IV criteria, we identified 2166 (78.05%) and 426 (15.35%) eligible patients, respectively, from the total pool of 2775 patients with NSCLC. The inclusion of AST and ALT ≤2.5x ULN criteria yielded 363 (13.08%) eligible patients from the pool of 2775 patients. Limiting AST and ALT to ≤1.0x ULN resulted in a decreased number of eligible patients (315/2775, 11.35%; Figure 5D). In addition, we explored the influence of Eastern Cooperative Oncology Group (ECOG) performance status as an additional criterion. With histology, stage, and ALT/AST laboratory test values (<2.5x ULN) as fixed criteria, by introducing ECOG scores of 0 to 2 or 0 to 1, we identified 194 (6.99%) and 151 (5.44%) eligible patients, respectively, from the pool of 2775 patients (Figure 5E).

Patient-matching performance was evaluated using precision, recall, and F₁-score performance metrics across specific clinical attributes. The average F₁-score, computed across 10 attributes from 8 domains (other primary malignancy, congestive heart failure, squamous NSCLC, organ/tissue transplantation, platelets, programmed death-1 antibody therapy, programmed cell death protein-1 or programmed cell death program-ligand 1 positive, stage groups, prior LOT [line of therapy], and ECOG), was 0.94 (range 0.82-1.00 [37]).

The challenge of achieving a high success rate in clinical trials is an ongoing issue [38,39]. Our study demonstrates the feasibility of a data-driven approach to optimize trial protocols and efficiently identify eligible patients by constructing a comprehensive, EHR-interoperable eligibility criteria knowledge base and integrating EHR data. To accomplish this, we analyzed 3281 clinical trials using our customized deep learning NLP model. We extracted all entities with their attributes and converted the hypernym concepts used in eligibility criteria to EHR-compatible hyponyms along with their corresponding values. We also evaluated the feasibility of optimizing the trial protocol design on the interface we developed. This interface offers an efficient and effective approach for assessing the number of eligible patients across various combinations of eligibility criteria such as different laboratory test values as well as combinations that account for vital signs.

We developed an eligibility criteria–specific ontology by manually scrutinizing 425 eligibility criteria to be used as a reference for manual annotation during NLP model training. Accurately identifying intricate semantic relationships among entities within eligibility criteria is crucial for constructing an appropriate ontology for precise information extraction, including temporal, arithmetic values, Boolean values, and negation modifiers [31]. Our customized NLP pipeline based on the eligibility criteria–specific ontology that we created enabled us to efficiently extract all pertinent attributes across different modalities and diseases, allowing for a more accurate definition of the trial population. To determine the applicability of our ontology generated using cancer clinical trials to other disease domains, we compared the concepts and relations in clinical trials of inflammatory bowel diseases. We observed very similar trends, suggesting that our eligibility criteria–specific ontology can be extended to other types of disease trials.

Moreover, the corpus of 485 manually annotated and standardized trials in a computable format can be used in eligible patient identification in EHRs.

Liu et al [28] conducted a thorough analysis of 352,100 clinical trials across various disease domains and constructed a knowledge base of clinical trial eligibility criteria. Their comprehensive knowledge base and user-friendly interface showcased the potential of advanced NLP techniques in enhancing eligibility criteria analysis and retrieval. Fang et al [40] also adopted a data-driven approach to optimizing clinical trial eligibility criteria in the context of Alzheimer disease and pancreatic cancer domains. Building upon these efforts, our study aimed to further narrow the gap between eligibility criteria and EHRs in multicancer domains, specifically in representing the granularities of eligibility criteria for identifying eligible patients and optimizing protocol designs. This was achieved by transforming hypernyms in the criteria into EHR-compatible hyponyms. We found that most of the primary groups include umbrella terms such as prior therapy (eg, proper prior therapy for actionable mutations) and biomarker (eg, EGFR inhibitor–resistant mutations). Our study also addressed the challenge of standardizing ambiguous clinical concepts in eligibility criteria for EHR interoperability and patient matching. To overcome this challenge, we converted hypernyms to the Entity-Attribute-Value format using prevailing values across different cancer types and modality therapies. We believe that our EHR-interoperable standardized eligibility criteria knowledge base and interface, integrating real-world EHR data, have the potential to improve the automatic screening system. This improvement has the potential to significantly reduce manual extraction efforts. Moreover, specific, computable criteria reduce ambiguity in patient identification and enable the inclusion of a broader range of patients who may qualify for the trial but could be excluded when using more general terms. This can increase patient trial enrollment, ultimately improving the overall success rate of trials. Notably, patients who were given the option to participate in a trial by their physicians demonstrated a significantly higher participation rate of 55% [41] compared to the current average of 5% to 8% among patients with cancer [42,43]. The implementation of our hypernym/hyponym semantic terminology model can likewise improve the effectiveness of information retrieval from EHRs and other clinical databases in the context of real-world evidence studies.

Certain criteria such as histology, stage, previous treatment, or biomarker are difficult to modify, while others such as vital signs or laboratory test values can be adjusted during the protocol design [15]. Our study revealed the impact of modifying laboratory test values while keeping other criteria constant, resulting in fluctuations in the number of eligible patients. Our findings, which demonstrate both the number of trials for different laboratory test value ranges and eligible patient numbers, offer insights for optimizing future protocol design and refining patient selection criteria. Seeking future collaboration with clinicians to conduct a direct comparison between the patient identification results by clinical domain experts and those generated by our prototype holds promise for a more comprehensive and informative evaluation of the prototype’s performance and its potential to enhance patient identification for clinical trials. Furthermore, a careful examination of the cases identified by the prototype can provide an understanding of the nature of false positives and false negatives. This will provide insights into how the prototype may differ in its patient identification results compared to manual extraction. Our eligibility criteria knowledge base can also be leveraged for generating SCAs using EHRs. SCAs, derived from real-world evidence, are regarded as substitutes for experimental control arms in trials [16-18]. The integration of SCAs into single-arm trial data or replacing traditional control arms with SCAs can alleviate the burden of target accrual in trials with low eligible patient numbers, such as rare disease or oncology trials with specific biomarkers. The Food and Drug Administration’s approval of the palbociclib inhibitor for male patients with metastatic BCa based on real-world evidence demonstrates the potential and relevance of SCAs in improving trial design and outcomes [44].

Our study has several limitations to consider. First, we focused on a limited scope, analyzing only 4 different cancer types and exploring extendibility in the context of inflammatory bowel diseases. Future studies should encompass a wider range of cancer types and disease domains for a more comprehensive analysis. Second, while most attributes were well defined, some umbrella terms lacked clear examples in other cancer types, potentially affecting result accuracy. Further manual annotation using knowledge bases could enhance the precision of the attribute tables. Third, our data set may be biased because we solely included industry-sponsored trials, potentially limiting the generalizability of our findings. In addition, the NLP training and test data sets in this study can display similarities owing to the shared attributes among different cancer trials, which heightens concerns regarding potential overfitting. Fourth, we did not address entity logic, and establishing the logic between entities would enhance cohort definition accuracy. Fifth and last, our interface feasibility testing was limited to small cohorts of patients with NSCLC, and the generalizability of our findings to other populations or disease conditions may vary. Furthermore, we did not perform a quantitative evaluation of the accuracy of matched patients although domain experts checked whether the patient information matched the eligibility criteria manually. While our model serves as a valuable illustration of how NLP can contribute to the design of trials across different diseases, we fully acknowledge the indispensable role of clinicians and biomedical researchers in ensuring the integrity of trial criteria. Clinical trials vary in their objectives, encompassing assessments of treatment end points, effectiveness, and other specific goals. The process is far more nuanced than merely adjusting laboratory test values because such modifications can have a substantial impact on the pool of eligible patients. Therefore, a comprehensive approach, considering both the clinical and biomedical aspects, is imperative for robust trial design.

Our study using an EHR-executable eligibility criteria knowledge base and real-world patient information provides valuable insights into the influence of different criteria on the number of eligible patients during the protocol design. The findings highlight the potential of using a data-driven approach that incorporates NLP and EHRs in clinical research.